A method and device for feature fusion extraction of video

Abstract

The present invention provides a method and device for video feature fusion extraction. The method comprises: dividing a feature map composed of multiple consecutive frames of the video into two groups according to the channel dimension; performing feature extraction on the two groups of feature maps using a channel excitation method in the spatial dimension and the temporal dimension, respectively, to obtain a spatial feature map and a temporal feature map; and fusing the spatial feature map and the temporal feature map to extract a feature map containing both spatial and temporal features of the video. The present invention solves the problem that existing feature extraction methods for videos ignore the interaction between the spatial and temporal dimensions, resulting in poor recognition results and low efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a feature fusion extraction method and device for video. Background Art

[0002] From experimental research to practical deployment, efficient algorithmic inference on platforms such as cloud servers or edge devices is a top priority in network model design. For example, YouTube users upload millions of videos daily for recommendation. Surveillance cameras in hospitals, shopping malls, and public transportation record massive amounts of content daily to ensure stable public order and travel safety. All of these scenarios require efficient and accurate video understanding.

[0003] Temporal modeling is crucial for video understanding methods. For example, models without temporal modeling capabilities can lead to confusion in recognition. Since the introduction of two-stream networks, deep learning methods have gradually become the standard for video understanding. 3D CNNs are currently the mainstream approach for video recognition. They link spatial and temporal features by extending the temporal dimension, but their heavy computational burden makes them difficult to deploy on edge devices. The success of Transformer models in modeling long-range interactions makes them a promising tool for solving video-related tasks. However, they lack inductive biases and scale quadratically with increasing input length. These limitations are further exacerbated when dealing with the high dimensionality of videos, leading to difficulties in model convergence. Several variants of 2D CNN methods have attracted our attention. TSM, one of the most effective models, breaks away from traditional 2D convolutional methods by independently processing the temporal dimension by shifting feature channels forward and backward. This allows for modeling both temporal and spatial information with fewer network parameters, enabling both offline and online video understanding. However, while TMS models model temporal information through partial channel shifting, they ignore the interaction between spatial and temporal dimensions, leaving room for improvement in their feature extraction capabilities. Summary of the Invention

[0004] In view of the above analysis, the present invention aims to provide a feature fusion extraction method and device for video, which is used to solve the problem that the feature extraction method for video in the existing technology ignores the interaction between spatial and temporal dimensions, and the extracted features ignore the time dimension information, resulting in poor recognition effect and low efficiency using the feature.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] In one aspect, the present invention provides a feature fusion extraction method for a video, comprising:

[0007] The feature map composed of the continuous multi-frame images of the video is divided into two groups X according to the channel dimensions and X te ;

[0008] The two sets of feature maps are extracted using the channel excitation method in the spatial dimension and the temporal dimension respectively to obtain spatial feature maps and temporal feature maps;

[0009] The spatial feature map and the temporal feature map are subjected to feature fusion to extract a feature map of the video containing spatial features and temporal features.

[0010] Furthermore, the channel excitation method is used to extract features in the spatial dimension to obtain a spatial feature map, including:

[0011] For a set of feature maps X s Perform average pooling in the spatial dimension, while keeping the temporal dimension unchanged, to obtain the primary spatial feature map;

[0012] One-dimensional convolution is used to model the spatial features of the primary spatial feature map to obtain a spatial feature map corresponding to the video.

[0013] Furthermore, the channel excitation method is used to extract features in the time series dimension to obtain a time series feature map, including:

[0014] For another set of feature maps X te Perform average pooling in the temporal dimension, keeping the spatial dimension unchanged, to obtain the primary temporal feature map;

[0015] The primary temporal feature graph is modeled using two-dimensional convolution to obtain a temporal feature graph corresponding to the video.

[0016] Furthermore, the adopting one-dimensional convolution to model the spatial features of the primary spatial feature map includes:

[0017] Perform 1D convolution on the primary spatial feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer and G is 8;

[0018] Perform 1D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G;

[0019] The feature map after channel recovery is batch normalized and activated by the activation function;

[0020] The feature weight obtained after activation is superimposed on the primary spatial feature map to obtain the spatial feature map corresponding to the video.

[0021] Furthermore, a set of feature maps X is given by the following formula s Perform average pooling to obtain the primary spatial feature map:

[0022]

[0023] in, is the primary spatial feature map obtained after global average pooling, F SGAP (X s ) represents a set of feature maps X s Perform spatial global average pooling, H×W is the height and width of the input feature map, x s (:,:,h,w) means that the temporal dimension remains unchanged and a set of feature maps Xs are average pooled in the spatial dimension.

[0024] Furthermore, the primary spatial feature map is modeled by the following formula:

[0025]

[0026] in, Indicates modeling of the primary spatial feature map, Conv1D C / 2G→C / 2Gr Indicates dimensionality reduction of the primary spatial feature map from C / 2G→C / 2Gr, Conv1D C / 2Gr→C / 2G Indicates the dimensionality recovery of the feature map from C / 2Gr→C / 2G, where r is the dimensionality reduction ratio of the convolutional layer; Indicates an overlay operation.

[0027] Furthermore, another set of feature maps X is given by the following formula te Perform average pooling to obtain the primary temporal feature map:

[0028]

[0029] Among them, is the primary temporal feature map obtained after global temporal average pooling, F TGAP (X te ) represents the temporal global average pooling of the primary temporal feature map, T is the number of frames of the input feature map, x te (:,t,:,:) indicates average pooling of the time series dimension, while other dimensions remain unchanged.

[0030] Furthermore, the feature fusion of the spatial feature map and the temporal feature map includes:

[0031] The temporal feature map and the spatial feature map are merged by staggering the channels; and the merged feature map is shuffled by channels;

[0032] The feature map after channel shuffling is extracted through the average pooling operation, and then a residual connection is performed with the merged feature map, and after activation through the activation function, a feature map corresponding to the video containing spatial features and temporal features is obtained.

[0033] Furthermore, feature fusion is performed using the following formula:

[0034] I out =ReLU(X ST +Pool(Shuffle(X ST )));

[0035] Among them, I out Represents the feature map output after feature fusion, X ST represents the sampled feature map after the attention excitation and merging of time-space separation, Shuffle(*) represents the ST Perform channel shuffling, Pool(*) means to perform average pooling and resampling on the feature map after channel shuffling, and compare it with X in a residual way. ST After connection, Relu activation function is used.

[0036] On the other hand, a computer device is also provided, comprising at least one processor and at least one memory communicatively connected to the processor; the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the aforementioned feature fusion extraction method for video.

[0037] Beneficial effects of this technical solution:

[0038] The present invention adopts a fusion method of separable temporal attention features and spatial attention features, extracts temporal features and spatial features respectively through SGAP and TGAP branches, and ensures the correlation of spatiotemporal features through channel excitation and hybrid recombination operations; maintains high efficiency while improving the performance of temporal feature extraction and spatial feature extraction; through a large number of experiments conducted on two benchmark datasets, it is shown that the feature fusion extraction method of the present invention outperforms existing action recognition networks in both accuracy and visual effects.

[0039] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description or be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0041] Figure 1 is a flow chart of a feature fusion extraction method for a video according to an embodiment of the present invention;

[0042] Figure 2 is a flow chart of temporal feature extraction and spatial feature extraction according to an embodiment of the present invention;

[0043] Figure 3 This is a flow chart of the feature fusion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the implementation cases of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0045] One embodiment of the present invention provides a feature fusion extraction method for video, such as Figure 1 As shown, the method includes:

[0046] Step S1: Divide the feature map composed of the continuous multi-frame images of the video into two groups X according to the channel dimension s and X te ; The feature map can be divided into two groups using the principle of half-half distribution;

[0047] Step S2: extract features from the two sets of feature maps using a channel excitation method in the spatial dimension and the temporal dimension, respectively, to obtain corresponding spatial feature maps and temporal feature maps;

[0048] Step S3: Fusing the spatial feature map and the temporal feature map to extract a feature map of the video containing spatial features and temporal features.

[0049] Specifically, the present invention proposes an effective feature extraction method for separable temporal and spatial attention features of videos; the method is based on the TSM model, such as Figure 2 As shown in the figure, feature maps are first grouped by channel dimension. Then, the dependencies between the two feature groups in the spatial and temporal channel dimensions are calculated separately. Finally, all features are cross-fused using a channel shuffle operation. This approach improves the feature representation after temporal and spatial channel transfer with less computation and improves the performance of the TSM model.

[0050] This embodiment is implemented through the following steps:

[0051] In step S1, the feature map composed of the continuous multi-frame images of the video is divided into two groups X according to the channel dimension s and X te ;

[0052] Specifically, given a feature map composed of multiple frames of images as input, in this embodiment, a feature map composed of 8 frames of images is used. First, the feature map is divided into G groups according to the channel dimension. Preferably, G can be 8, 10 or 12 groups, etc., according to actual needs. In this embodiment, G is 8; and each group of feature maps is further divided into two groups X s and X te , which are used for spatial feature extraction and temporal feature extraction respectively, and are expressed as:

[0053]

[0054]

[0055] Where T is the number of image frames, which is 8 in this embodiment, and C is the number of image channels x s n and x te n Represents the feature map X s and X te The nth frame in .

[0056] In step S2, feature extraction is performed on the two sets of feature maps using a channel excitation method in the spatial dimension and the temporal dimension, respectively, to obtain a spatial feature map and a temporal feature map;

[0057] Specifically, this embodiment uses the SGAP module and the TGAP module to perform spatial feature extraction and temporal feature extraction respectively, wherein the SGAP module includes time compression and pooling operations, and the TGAP module includes space compression and pooling operations;

[0058] More specifically, the input feature map is split into two groups X s and X te , each group has c channels, satisfying 2cG=CT; the average pooling method is used through the SGAP module to perform the first group of feature maps X s Feature extraction is performed on the spatial dimension to obtain a spatial feature map; the second set of feature maps X is pooled using the average pooling method through the TGAP module te Feature extraction is performed on the time dimension to obtain a time series feature graph.

[0059] Among them, the channel excitation method is used to extract features in the spatial dimension to obtain a spatial feature map, including: a set of feature maps X s Average pooling is performed in the spatial dimension, while the temporal dimension remains unchanged, to obtain a primary spatial feature map; one-dimensional convolution is used to model the spatial features of the primary spatial feature map to obtain a spatial feature map corresponding to the video.

[0060] The spatial features of the primary spatial feature map are modeled using one-dimensional convolution, including: performing 1D convolution on the primary spatial feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer and G is the grouping hyperparameter. In the experiment, 8 is taken as the number of feature groups; performing a 1D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G; batch normalizing the feature map after channel restoration and activating it through an activation function; and superimposing the feature weights obtained after activation on the primary spatial feature map to obtain the spatial feature map corresponding to the video.

[0061] Preferably, a set of feature maps X is given by the following formula s Perform average pooling to obtain the primary spatial feature map:

[0062]

[0063] in, is the primary spatial feature map obtained after global average pooling, F SGAP (X s ) represents a set of feature maps X s Perform spatial global average pooling, H×W is the height and width of the input feature map, x s (:,:,h,w) means that the temporal dimension remains unchanged and a set of feature maps Xs are average pooled in the spatial dimension.

[0064] The primary spatial feature map is modeled by the following formula:

[0065]

[0066] in, Indicates modeling of the primary spatial feature map, Conv1D C / 2G→C / 2Gr Indicates dimensionality reduction of the primary spatial feature map from C / 2G→C / 2Gr, Conv1D C / 2Gr→C / 2G Indicates the dimensionality recovery of the feature map from C / 2Gr→C / 2G, where r is the dimensionality reduction ratio of the convolutional layer; Indicates an overlay operation.

[0067] Furthermore, the channel excitation method is used to extract features in the time series dimension to obtain a time series feature map, including: te Perform average pooling in the temporal dimension, keeping the spatial dimension unchanged, to obtain a primary temporal feature map; use two-dimensional convolution to model the temporal features of the primary temporal feature map to obtain a temporal feature map corresponding to the video;

[0068] Among them, the spatial features of the primary temporal feature map are modeled using two-dimensional convolution, including: performing 2D convolution on the primary temporal feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer; performing a 2D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G; the feature map after channel restoration is batch normalized and activated by an activation function; and superimposing the feature weights obtained after activation on the primary temporal feature map to obtain the temporal feature map corresponding to the video.

[0069] Preferably, another set of feature maps X is calculated by the following formula: te Perform average pooling to obtain the primary temporal feature map:

[0070]

[0071] in, is the primary temporal feature map obtained after global temporal average pooling, F TGAP (X te ) represents the temporal global average pooling of the primary temporal feature map, T is the number of frames of the input feature map, x te (:,t,:,:) indicates average pooling of the time series dimension, while other dimensions remain unchanged.

[0072] The primary time series characteristic graph is modeled by the following formula:

[0073]

[0074] in, Indicates modeling of primary temporal feature maps, Conv2D C / 2G→C / 2Gr Indicates that the feature is reduced from C / 2G→C / 2Gr by 2D convolution, Conv2D C / 2Gr→C / 2G Indicates that the feature is restored from C / 2Gr→C / 2G through 2D convolution. Indicates superimposing the extracted feature weights with the primary temporal feature map.

[0075] In this embodiment, spatial features and temporal features are modeled respectively through 1D convolution and 2D convolution, and superimposed with the initial feature vector, so as to realize linear combination (information integration) of each pixel on different channels, and retain the original planar structure of the original image. Through channel adjustment, the functions of dimensionality increase and dimensionality reduction are completed, the parameters of model training can be controlled, and the effective features of the video can be obtained more specifically, and useless features can be filtered out. This incentive mechanism focuses on improving the ability to integrate the spatial information and temporal information of the model, and adopts channel grouping to learn spatial and temporal information in parallel, and filters out effective spatiotemporal semantics through an adaptive fusion mechanism. After passing through the convolution layer and being activated, the expression ability of the model is effectively improved, and the effectiveness of feature extraction is improved.

[0076] In step S3, the spatial feature map and the temporal feature map are subjected to feature fusion to extract a feature map of the video containing spatial features and temporal features.

[0077] Specifically, such as Figure 3 As shown, this embodiment adopts a feature fusion module to merge the temporal feature map and the spatial feature map by staggering the channels; and shuffles the merged feature map by channels; and extracts features from the feature map after the channel shuffle by an average pooling operation, and performs a residual connection with the merged feature map, and activates it through an activation function to obtain a feature map corresponding to the video containing spatial features and temporal features.

[0078] More specifically, feature fusion is performed using the following formula:

[0079] I out =ReLU(X ST +Pool(Shuffle(X ST )));

[0080] Among them, I out Represents the feature map output after feature fusion, X ST represents the sampled feature map after the attention excitation and merging of time-space separation, Shuffle(*) represents the ST Perform channel shuffling, Pool(*) means to perform average pooling and resampling on the feature map after channel shuffling, and compare it with X in a residual way. ST After connection, Relu activation function is used.

[0081] As a specific implementation, this example uses ResNet50 as the backbone network and packages the SGAP module, TGAP module, and feature fusion module of the present invention into a spatiotemporal separation attention excitation module for action recognition in a video dataset to verify the effectiveness of the feature fusion extraction method of the present invention. UCF101 and HMDB51 are selected as training datasets, and random horizontal flipping and 90°, 180°, and 270° rotations are used to expand the training data. To use the image data type when training the model, RGB video frames are extracted from the dataset and saved as images, stored in folders, and named according to their type. This example divides all video frame data into training and test sets in an 8:2 ratio.

[0082] In this embodiment, a spatiotemporal separation attention incentive module is added to the four backbones to improve the network's action recognition ability. When evaluating the performance of the action recognition network, the evaluation indicator used is precision. Precision refers to the proportion of events correctly predicted by the network among all prediction samples, reflecting the accuracy of the action recognition network. TP represents the number of correct predictions, FP represents the number of incorrect predictions, and TP+FP represents the total number of predictions. Precision (accuracy) represents the proportion of the number of correct predictions by the model to the total number of predictions.

[0083] During the experiment, the feature fusion extraction method was implemented on the PyTorch platform and conducted using Nvidia Tesla V100s graphics cards. The model was trained using the mini-batch stochastic gradient descent algorithm and the Adam optimization algorithm. The batch size was set to 12, the initial learning rate was 0.0001, the learning rate decay period was 10, the decay factor was 0.1, the dropout was set to 0.8, and the number of model iterations was 25. Research has shown that using pre-training is an effective method for initializing convolutional neural networks when there are insufficient training samples in the target dataset. To avoid overfitting, a pre-trained model from the Kinetics dataset was used to initialize the network weights.

[0084] This implementation case verifies the effectiveness of the feature fusion extraction method proposed in this invention through ablation experiments;

[0085] We first analyzed the impact of the placement of the spatiotemporal separation attention excitation module within the backbone network on network performance. Furthermore, to validate the module's effectiveness, we compared the impact of varying numbers of excitation blocks on network performance in the stage 3 branch.

[0086] Table 1 shows the results of the first ablation experiment, in which three spatiotemporal separation attention activation modules (SSAs) were added to stages 1, 2, 3, and 4, respectively. The experimental results show that the effect of the spatiotemporal attention activation block improves significantly as the network deepens. While maintaining the same basic settings, adding the spatiotemporal attention activation block to stage 4 effectively improves the network's temporal extraction capabilities. As shown in Table 1, using the spatiotemporal attention activation block improves network performance by 0.5% on the UCF101 dataset and by 1.4% on the HMDB51 dataset. This demonstrates that the module helps integrate spatial and temporal features, demonstrating its effectiveness.

[0087] Table 1. The impact of spatiotemporal attention block position on network performance

[0088]

[0089]

[0090] Table 2 shows the results of the second ablation experiment, in which the number of spatiotemporal attention blocks was set to 0, 1, 3, and 6, respectively. As the number of blocks increased, performance on the UCF101 and HMDB51 datasets gradually improved, but the number of parameters and computation also increased. This shows that the spatiotemporal attention blocks proposed in this paper can effectively improve network performance while adding only a small amount of parameters and computation.

[0091] Table 2 Effect of the number of spatiotemporal attention excitation blocks on network performance

[0092] Structure SSA UCF101 HMDB51 #Parameter FLOPs Baseline ×0 94.6％ 73.5％ 24.33M 58.42G Stage 3 ×1 94.7％ 73.8％ 24.33M 58.42G Stage 3 ×3 95.0％ 74.3％ 24.53M 58.42G Stage 3 ×6 95.2％ 74.9％ 24.83M 58.43G

[0093] The network with the spatiotemporal separation attention excitation module is compared with seven state-of-the-art lightweight SISR methods at *2, *3, and *4 scales, including IDT, DRTAP, TWO-STREAM, C3D, P3D, R(2+1)D, TSN, and TSM. As shown in Table 3, SSA outperforms other state-of-the-art lightweight methods on all benchmarks in terms of 2D CNN and 3D CNN metrics.

[0094] Table 3 Comparison of the effects of various methods on two datasets

[0095]

[0096]

[0097] In summary, the feature fusion extraction method for video provided by the embodiments of the present invention adopts a separable spatiotemporal attention feature fusion method to improve the performance of temporal feature extraction while maintaining high efficiency. The SGAP and TGAP branches of the present invention can extract temporal and spatial features respectively, and ensure the correlation of spatiotemporal features through channel excitation and hybrid recombination operations; the method focuses on improving the fusion ability of the spatial information and temporal information of the model, adopts channel grouping to learn spatial and temporal information in parallel, and filters out effective spatiotemporal semantics through an adaptive fusion mechanism; a large number of experiments on two benchmark datasets show that the present invention is superior to other advanced action recognition networks in terms of accuracy and visual effects.

[0098] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0099] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A feature fusion extraction method for video, characterized in that: include: The feature map composed of the continuous multi-frame images of the video is divided into two groups X according to the channel dimension s and X te ; The two sets of feature maps are extracted using the channel excitation method in the spatial dimension and the temporal dimension to obtain spatial feature maps and temporal feature maps; including: using the SGAP module to extract the feature maps X s Perform average pooling in the spatial dimension, keep the temporal dimension unchanged, and obtain a primary spatial feature map. Use one-dimensional convolution to model the spatial features of the primary spatial feature map to obtain the spatial feature map corresponding to the video. A set of feature maps X is calculated by the following formula: s Perform average pooling to obtain the primary spatial feature map: in, is the primary spatial feature map obtained after global average pooling, F SGAP (X s ) represents a set of feature maps X s Perform spatial global average pooling, H×W is the height and width of the input feature map, x s (:,:,h,w) means that the temporal dimension remains unchanged, and a set of feature maps X is s Perform average pooling; The primary spatial feature map is modeled by the following formula: in, Indicates modeling of the primary spatial feature map, Conv1D C / 2G→C / 2Gr Indicates dimensionality reduction of the primary spatial feature map from C / 2G→C / 2Gr, Conv1D C / 2Gr→C / 2G Indicates the dimensionality recovery of the feature map from C / 2Gr→C / 2G, where C is the number of image channels, G is the grouping hyperparameter, and r is the dimensionality reduction ratio of the convolutional layer; Indicates superposition operation; Through the TGAP module, another set of feature maps X te Perform average pooling on the temporal dimension, keep the spatial dimension unchanged, and obtain a primary temporal feature map. Use two-dimensional convolution to model the temporal features of the primary temporal feature map to obtain the temporal feature map corresponding to the video. Use the following formula to calculate another set of feature maps X te Perform average pooling to obtain the primary temporal feature map: in, is the primary temporal feature map obtained after global temporal average pooling, F TGAP (X te ) represents the temporal global average pooling of the primary temporal feature map, T is the number of frames of the input feature map, x te (:,t,:,:) means average pooling is performed on the time series dimension, while other dimensions remain unchanged; The primary time series characteristic graph is modeled by the following formula: in, Indicates modeling of primary temporal feature maps, Conv2D C / 2G→C / 2Gr Indicates that the feature is reduced from C / 2G→C / 2Gr by 2D convolution, Conv2D C / 2Gr→C / 2G Indicates that the feature is restored from C / 2Gr to C / 2G through 2D convolution, where C is the number of image channels, G is the grouping hyperparameter, and r is the dimensionality reduction ratio of the convolution layer. Indicates superimposing the extracted feature weights with the primary temporal feature map; The spatial feature map and the temporal feature map are subjected to feature fusion to extract a feature map containing spatial features and temporal features of the video, including: merging the temporal feature map and the spatial feature map by staggering channels through a feature fusion module; shuffling the merged feature map through channels; extracting features from the shuffled feature map through an average pooling operation, performing a residual connection with the merged feature map, and activating the feature map through an activation function to obtain a feature map containing spatial features and temporal features corresponding to the video; The SGAP module, TGAP module and feature fusion module are packaged into a spatiotemporal separation attention excitation module, which is set in stage 4 of the resnet50 backbone network to improve the network's temporal extraction capability.

2. The feature fusion extraction method for video according to claim 1, characterized in that: The adopting one-dimensional convolution to model the spatial features of the primary spatial feature map includes: Perform 1D convolution on the primary spatial feature map to reduce the number of channels to C / 2Gr, where r is the dimensionality reduction ratio of the convolution layer and G is the grouping hyperparameter; Perform 1D convolution operation on the feature map after channel dimensionality reduction to restore the number of channels to C / 2G; The feature map after channel recovery is batch normalized and activated by the activation function; The feature weight obtained after activation is superimposed on the primary spatial feature map to obtain the spatial feature map corresponding to the video.

3. The feature fusion extraction method for video according to claim 1, characterized in that: Feature fusion is performed using the following formula: I out =ReLU(X ST +Pool(Shuffle(X ST ))); Among them, I out Represents the feature map output after feature fusion, X ST represents the sampled feature map after the attention excitation and merging of time-space separation, Shuffle(*) represents the ST Perform channel shuffling, Pool(*) means to perform average pooling and resampling on the feature map after channel shuffling, and compare it with X in a residual way. ST After connection, Relu activation function is used.

4. A computer device, characterized in that: It includes at least one processor and at least one memory in communication with the processor; the memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the feature fusion extraction method for video according to any one of claims 1 to 3.

Images