Video summarization method based on multi-dimensional features and fine-grained hierarchical modeling

Abstract

The application provides a video summarization method based on multi-dimensional features and fine-grained hierarchical modeling, and relates to the technical field of video processing. In practical application, the video summarization technology can facilitate large-scale video retrieval and browsing. The method comprises the following steps: firstly, frame extraction is performed on an input video to obtain a frame sequence, and a multi-dimensional feature extraction network composed of a 2D network and a 3D network is used to extract multi-dimensional features; then, hierarchical temporal modeling is performed to complete the modeling process of the temporal dependence of the entire video sequence; finally, a regression network is used to obtain the importance score of each frame and generate a video summary. The application further explores the influence of the spatiotemporal features extracted by 3D feature extractors with different spatiotemporal complexities on the video summary result. The application shows excellent performance on the video summary datasets SumMe and TVSum. Whether from the application scene or the performance index, the application has strong practical value.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a video summarization method based on multi-dimensional features and fine-grained hierarchical modeling.

[0002] Research Background

[0003] In recent years, with the widespread adoption of video sharing platforms such as YouTube, TikTok, and Facebook, and the proliferation of video capture devices such as handheld drones, surveillance cameras, and smartphones, video data has exploded. It's reported that over 500 hours of video are uploaded to YouTube every minute. Such a massive amount of video data is beyond human processing. So how can people quickly and accurately find content of interest from this endless collection of videos? The answer lies in video summarization technology. Video summarization is a technique that automatically extracts representative segments from original long videos to shorten the video length while preserving the original video content.

[0004] Early research on video summarization focused on using handcrafted heuristics to identify representative and diverse frames. However, these methods were gradually phased out because handcrafted features could not encode deeper semantic information in videos. With the rise of deep learning, models based on recurrent neural networks (RNNs) and long short-term memory (LSTM) networks were proposed to learn high-level semantic feature representations of data and to capture temporal relationships between video frame sequences. However, RNNs are difficult to parallelize using GPUs, and the gating mechanism of LSM networks leads to severe attenuation of historical information from previous frames, making these methods very challenging for handling videos with complex and diverse scenes. The emergence of attention mechanisms has alleviated these problems, but current attention-based video summarization methods either use an encoder-decoder network structure (where the encoder encodes the content information of the input video sequence, and the decoder learns short-term and long-term dependencies between sequences through the encoded features to generate the final summary) or directly model the temporal relationships of the entire video sequence using a single attention mechanism. However, these methods have a problem: the modeling of temporal relationships is two-stage and too coarse, leading to the loss of some temporal relationships. Summary of the Invention

[0005] To alleviate the aforementioned problems, this invention provides a hierarchical modeling approach from fine-grained to coarse-grained to model the temporal relationships of the entire video. From a practical implementation perspective, this framework comprises three main parts: a multidimensional feature extractor for dynamic and static feature extraction, a dynamic temporal modeling module that progressively expands the receptive field of the video sequence, and a regression network that generates importance scores for each frame. Furthermore, this invention explores the impact of three dynamic feature extractors—P3D, I3D, and X3D—with different spatiotemporal complexities and using different pre-trained datasets on video summarization results, identifying spatiotemporal features more suitable for video summarization tasks. The technical solution proposed in this invention is as follows:

[0006] A video summarization method based on multi-dimensional features and fine-grained hierarchical modeling is proposed. First, a multi-dimensional feature extractor is used to extract fine-grained static image features representing visual content and fine-grained dynamic video features representing temporal information. Second, dynamic temporal modeling operations are used to progressively explore temporal relationships to improve video summarization performance. Finally, a regression network is used to obtain the importance score of each frame. Furthermore, this method explores the impact of different types of dynamic features on the summarization results to identify spatiotemporal features more suitable for this domain. Specific steps include:

[0007] 1) Extract frames from the input video at different frame extraction rates to obtain the corresponding static video frame sequences;

[0008] 2) The GoogLeNet neural network model is used to extract features from static video frame sequences to obtain fine-grained static image features that reflect low-level visual content. Simultaneously, a 3D convolutional neural network is used, with each set of 16 static video frames as input, to extract fine-grained dynamic video feature vectors that characterize temporal information. Specifically, 3D feature extractors Pseudo-3D, Inflated 3D, and Expand 3D with different spatiotemporal complexities and pre-trained datasets are used to extract P3D, I3D, and X3D video feature vectors covering fine-grained temporal relationships, collectively expressed as...

[0009] 3) The fine-grained static image features and fine-grained dynamic video features are used as inputs to the image temporal information modeling branch and the video temporal information modeling branch, respectively. These are used to aggregate the features within each frame or shot from fine-grained to coarse-grained into coarse-grained features that can efficiently represent semantic information. The video temporal information modeling branch includes:

[0010] One-dimensional sequence convolution: The one-dimensional sequence convolution converts three consecutive fine-grained dynamic video feature vectors into one. As a set of inputs, they are fused into a medium-granularity video feature vector that reduces visual redundancy between consecutive fine-grained features. This is equivalent to modeling the temporal correlation of consecutive n×3 frames and generating medium-granularity temporal features;

[0011] Hollow attention: Hollow attention, based on the ordinary attention mechanism, overlays a binary hollow mask matrix B onto a pairwise attention matrix A, such that the elements at position 1 in both A and B are negative infinity, thus masking A to obtain A′. This addresses the problem of drastically increased computational cost when the receptive field of a video segment is expanded to the entire video sequence; medium-granularity video feature vectors. The hollow attention mechanism can save computing resources while modeling the temporal relationship of the entire video sequence, thereby obtaining coarse-grained video features that can reflect the temporal relationship dependency of the entire video context.

[0012] 4) The coarse-grained features output from the image temporal information modeling branch and the video temporal information modeling branch are added and fused to obtain a fused temporal feature representation;

[0013] 5) The fusion-level temporal feature representation generates frame-level importance scores through three fully connected layers;

[0014] 6) In each iteration, the mean squared error between the frame-level importance score and the ground-truth is used as the training loss. This loss is then used to calculate the gradient through backpropagation and update the learnable parameters in the model.

[0015] 7) During the reasoning process, importance scores are used to select key segments of the video and form a video summary.

[0016] Specifically, the frame rate mentioned in step 1) is 2 frames per second for GoogLeNet; for P3D, I3D and X3D, the frame rate is equal to their respective frame rates.

[0017] Specifically, the image temporal information modeling branch and the video temporal information modeling branch mentioned in step 3) have the same processing procedure; for the image branch, in order to prevent the loss of original information while generating high-level semantic feature information, fine-grained static image features are connected to coarse-grained image feature representation through residuals.

[0018] This invention achieves the following technological innovations:

[0019] First, dynamic video features with temporal relationships extracted by 3D convolution are introduced on the basis of image features, which makes up for the lack of correlation and temporal sequence of static image features extracted by 2D convolution.

[0020] Second, a hierarchical temporal modeling method that gradually expands the receptive field of video sequences is proposed, thereby completing the modeling of the entire video temporal relationship from fine-grained to coarse-grained.

[0021] Third, by exploring the impact of three different types of dynamic video feature extractors on video summarization tasks, we found spatiotemporal features that are more suitable for video summarization tasks. Attached Figure Description

[0022] Figure 1 This is a framework diagram of the present invention.

[0023] Figure 2 This is a schematic diagram of the void attention proposed in this invention. Detailed Implementation

[0024] The method of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Reference Figure 1 and Figure 2 This invention provides a video summarization method based on multi-dimensional features and fine-grained hierarchical modeling, the method comprising the following steps:

[0026] S1: Extract frames from the input video at a rate of 2 frames per second to obtain {x1,x2,…,x...} T Similarly, according to the frame rate, we get {s1,s2,…,s}. L}, where T and L represent the number of downsampled frames and the total number of frames, respectively;

[0027] S2: GoogLeNet is used as the backbone network for static image feature extraction. {x1,x2,…,x} T The fine-grained static image feature vectors are extracted by sequentially feeding the data into the GoogLeNet network after removing the last three layers. The fine-grained static image features of each video with respect to GoogLeNet are represented by F. ig ∈R T×C Where C = 1024 represents the feature dimension of each frame. Meanwhile, {s1,s2,…,s…} L The n consecutive video frames in the dataset are used as inputs to P3D, I3D, and X3D respectively to obtain the corresponding fine-grained dynamic video feature vectors. The coarse-grained dynamic video feature representation for each video can be obtained using F... vd ∈R T′×C′ The description is as follows: T′ represents the number of segments in which a video with a total number of frames L is uniformly divided into n consecutive frames, and C′ represents the feature dimension of each video segment output by the 3D network, which is 2048 for P3D and X3D and 1024 for I3D.

[0028] S3:F ig and F vdThe processing is carried out in two branches: image temporal information modeling branch and video temporal information modeling branch.

[0029] One-dimensional sequence convolution: In the video temporal information modeling branch, one-dimensional sequence convolution is performed along the dimension of the video sequence, which combines three adjacent fine-grained dynamic video feature vectors that encompass short-term temporal correlations. Fuse into a single medium-granularity video feature vector

[0030] Hollow attention: the mid-granularity video temporal feature representation The pairwise attention matrix A∈R is computed as input to the empty attention function. T×T For the i-th and j-th segments, i,j∈{1,2,…,T′}, the corresponding medium-granularity video feature vectors are: as well as Therefore, a certain term A in the pairwise attention matrix ij The calculation process can be specifically represented as follows:

[0031]

[0032] Where q (q = T) is a constant, and W represents two linear mappings. Q ∈B C′×C′ and W K ∈R C′×C′ These are the parameters that the model needs to learn during the training process.

[0033] Next, the hole mask matrix B∈R T×T This is done by covering matrix A with elements at positions 1 in both A and B such that these elements are negative infinity, thus masking A to obtain A′. Then, the masked pairwise attention weights A′ are applied. ij After being transformed into corresponding normalized weights α by the softmax function, ij The calculation process is shown in the following formula:

[0034]

[0035] Finally, normalize the weights α. ij Following the random deactivation layer, the output vector α′ of the random deactivation layer j With the learnable linear mapping layer W Q ∈R C′×C′ The output matrix is ​​then multiplied to obtain attention-weighted coarse-grained video features. The calculation process is shown in the following formula:

[0036]

[0037] S4: For the image branch, fine-grained static image features are residually concatenated to coarse-grained image features. Finally, the output M of the image branch is... ig and the output M of the video timing branch vd The summation and fusion yields the final fused temporal feature representation.

[0038] S5: The fusion-level temporal feature representation first undergoes a random deactivation layer, followed by a normalization layer, then three fully connected layers. The first two fully connected layers are followed by a ReLU activation function, then a random deactivation layer is applied after the ReLU, followed by a normalization layer, and finally a Sigmoid layer is applied after the last neural network layer to generate frame-level importance scores. Where y t ∈R and 0≤y t ≤1.

[0039] S6: During the training phase, this invention calculates the mean squared error between the frame-level importance score and the ground-truth as the training loss. This loss is then used to calculate the gradient through backpropagation and update the learnable parameters in the model.

[0040] S7: During the inference process, the video is first segmented into different segments using the KTS algorithm. Then, the frame-level importance score within each segment is averaged to generate a segment-level importance score. Finally, the Knapsack algorithm is used to maximize the total summary score without exceeding 15% of the original video length, thus obtaining the final summary result. The mathematical representation of this stage is shown in the following formula:

[0041]

[0042] Where P represents the total number of segments, k p ∈{0,1} indicates whether to select this segment; m p This represents the score for that segment, n. p This indicates the length of the segment, and L represents the total number of video frames.

[0043] S8: To verify the accuracy and robustness of this invention, it is compared with existing algorithms on the video summarization datasets TVSum, SumMe, YouTube, and OVP. The F-score is used to evaluate the model performance in three settings: standard (C), augmented (A), and transfer (T). The results are shown in Table 1.

[0044] Table 1

[0045]

[0046] In the standard settings, the method of this invention outperforms the state-of-the-art model results by 2.4% and 1.8% on two datasets, respectively. In the enhanced settings, it also achieves the best results, exceeding the second-best results by 0.3% and 2.7%, respectively. To further verify the generalization ability of this invention, experiments were conducted in the transfer settings, and the experiments show that this invention outperforms most methods.

[0047] S9: To identify spatiotemporal features more suitable for video summarization tasks, this invention explores the impact of three dynamic feature extractors—P3D, I3D, and X3D—with different spatiotemporal complexities and using different pre-training datasets, on video summarization results. Table 2 shows the computational complexity, pre-training dataset, and number of parameters for different 3D networks. Table 3 shows the results data using different input feature representations in this invention.

[0048] Table 2

[0049]

[0050] Table 3

[0051]

[0052] Table 2 shows that X3D achieves a balance between computational complexity and parameter count, making it a lightweight feature extractor. Table 3 leads to the following conclusions: First, simple 2D features cannot meet the needs of video summarization; second, the input combining X3D and 2D features achieves optimal results, proving that video summarization tasks do not require overly complex spatiotemporal features, and lightweight 3D features are sufficient.

Claims

1. A video summarization method based on multi-dimensional features and fine-grained hierarchical modeling. The method first uses a multi-dimensional feature extractor to extract fine-grained static image features representing visual content and fine-grained dynamic video features representing temporal information. Second, it gradually explores temporal relationships through dynamic temporal modeling operations to improve the performance of video summarization. Finally, it obtains the importance score of each frame through a regression network. Furthermore, this method explores the impact of different types of dynamic features on the summarization results to identify spatiotemporal features more suitable for video summarization. Specific steps include: 1) Extract frames from the input video at different frame extraction rates to obtain the corresponding static video frame sequences; 2) The GoogLeNet neural network model is used to extract features from static video frame sequences to obtain fine-grained static image features that reflect low-level visual content. Simultaneously, a 3D convolutional neural network is used, with each set of 16 static video frames as input, to extract fine-grained dynamic video feature vectors that characterize temporal information. Specifically, 3D feature extractors Pseudo-3D, Inflated 3D, and Expand 3D, with different spatiotemporal complexities and pre-trained datasets, are used to extract P3D, I3D, and X3D video feature vectors, respectively, covering fine-grained temporal relationships. These are uniformly expressed as... ; 3) The fine-grained static image features and fine-grained dynamic video features are used as inputs to the image temporal information modeling branch and the video temporal information modeling branch, respectively. These are used to aggregate the features within each frame or shot from fine-grained to coarse-grained into coarse-grained features that can efficiently represent semantic information. The video temporal information modeling branch includes: One-dimensional sequence convolution: The one-dimensional sequence convolution combines three consecutive fine-grained dynamic video features As a set of inputs, they are fused into a medium-granularity video feature that reduces visual redundancy between consecutive fine-grained features. This is equivalent to continuous The temporal correlation of frames is modeled and medium-granularity temporal features are generated; Hollow attention: Hollow attention is based on the ordinary attention mechanism, using a binary hole mask matrix. Overlaid on pairwise attention matrix A Above, making A Zhongyu B The element with a position of 1 in the middle is negative infinity, thus achieving the goal of... A The mask is obtained To address the issue of drastically increased computational costs when the receptive field of a video segment is expanded to the entire video sequence; medium-granularity video features The hollow attention mechanism can save computing resources while modeling the temporal relationship of the entire video sequence, thereby obtaining coarse-grained video features that can reflect the temporal relationship dependency of the entire video context. 4) The coarse-grained features output from the image temporal information modeling branch and the video temporal information modeling branch are added and fused to obtain a fused temporal feature representation; 5) The fusion-level temporal feature representation generates frame-level importance scores through three fully connected layers; 6) In each iteration, the mean squared error between the frame-level importance score and the ground-truth is used as the training loss. This loss is then used to calculate the gradient through backpropagation and update the learnable parameters in the model. 7) During the reasoning process, importance scores are used to select key segments of the video and form a video summary.

2. The video summarization method based on multi-dimensional features and fine-grained hierarchical modeling according to claim 1, characterized in that: The frame rate mentioned in step 1) is 2 frames per second for GoogLeNet; for P3D, I3D and X3D, the frame rate is equal to their respective frame rates.

3. The video summarization method based on multi-dimensional features and fine-grained hierarchical modeling according to claim 1, characterized in that: The image temporal information modeling branch and the video temporal information modeling branch mentioned in step 3) have the same processing procedure. For the image branch, in order to prevent the loss of original information while generating high-level semantic feature information, fine-grained static image features are connected to coarse-grained image feature representation through residuals.

Images