Multi-scale spatiotemporal modeling video summary generation method and device

Abstract

The multi-scale spatiotemporal modeling video summary generation method and device provided by the present invention include a feature extraction module, a multi-scale spatiotemporal modeling module, an importance classifier, and a summary generation module; the multi-scale spatiotemporal modeling module includes a multi-scale aggregator, a cascaded time modeling module, and a parallel space modeling module that are freely arranged in sequence. The present invention comprehensively captures the local details and global structure of the video by constructing a multi-scale spatiotemporal modeling module including a multi-scale aggregator, a cascaded time modeling module, and a parallel space modeling module; and the order of the multi-scale aggregator, the cascaded time modeling module, and the parallel space modeling module can be freely arranged, so that the adaptability of the model to different video features is improved, so that the model has low computational complexity while maintaining high performance. The present invention has the advantages of low cost, simple structure, good performance, and strong adaptability.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method and device for generating a multi-scale spatiotemporal modeling video summary. Background Art

[0002] With the rapid development of internet technology and the widespread adoption of digital devices, the production and dissemination of video content has exploded. From short videos shared on social media to the vast library of film and television resources available on online video platforms, video has become a primary form of information acquisition and entertainment consumption. This growth trend places higher demands on the management and utilization of video content, prompting the emergence of video summarization technology. The exponential growth of online video content has led to information overload for users. Video summarization technology can help users quickly understand the core content of a video, thereby improving video browsing efficiency.

[0003] There is currently a lot of research on video summarization technology. Convolutional neural networks (CNNs) play a key role in video understanding, laying the foundation for video content analysis due to their excellent image feature extraction capabilities. Recurrent neural networks (RNNs) and their variants (such as long short-term memory networks (LSTMs)) have gradually become mainstream methods in the field of video summarization due to their ability to process sequential data and capture temporal dependencies. These networks can effectively exploit inter-frame correlations to perform frame-level saliency estimation. Transformers can integrate local and global temporal patterns to effectively model the complete video context while maintaining spatial saliency characteristics. This modeling approach gives video summarization systems significantly improved content analysis and summary synthesis capabilities.

[0004] Among these methods, those based on RNNs and LSTMs mainly focus on local temporal correlations, while ignoring long-distance dependencies and intra-frame visual saliency cues consistent with human attention patterns; the Transformer-based methods are limited by the Transformer's inherent quadratic computational complexity, which poses challenges to sequence-to-sequence applications including video summarization. Summary of the Invention

[0005] In view of the defects of the existing technology, the present invention provides a method and device for generating video summaries based on multi-scale spatiotemporal modeling.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] On the one hand, the present invention provides a multi-scale spatiotemporal modeling video summary generation device, comprising a feature extraction module, a multi-scale spatiotemporal modeling module, an importance classifier, and a summary generation module;

[0008] The feature extraction module is used to divide the input video into non-overlapping segments and extract features of each video segment using a convolutional neural network;

[0009] The multi-scale spatiotemporal modeling module includes a multi-scale aggregator arranged in a free sequence, a cascaded time modeling module, and a parallel space modeling module; the multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of input features, and adds segment category tags before the input features, and applies a feature fusion strategy to aggregate multi-scale features to obtain fused features; the cascaded time modeling module is composed of multiple cascaded time Mamba blocks, and the input features are refined by residual learning in the multiple cascaded time Mamba blocks to fuse short-term prompts and long-term event dependencies, and output refined features; the parallel space modeling module is composed of multiple parallel space Mamba blocks with shared parameters, each space Mamba block is used to process the features of a single frame, and the input features are converted into frame-level features by a linear layer and then input into the space Mamba block, which is processed in parallel in the space Mamba block through residual learning to extract spatial features, and the spatial features extracted by each space Mamba block are aggregated to form a feature representation of the video content;

[0010] The importance classifier outputs an importance score based on the output features of the multi-scale spatiotemporal modeling module and optimizes the importance score in combination with a loss function;

[0011] The summary generation module is used to solve the constrained optimization problem and construct a summary based on the optimized importance scores.

[0012] Furthermore, the fused features are obtained according to the following formula:

[0013] ;

[0014] in, It is a low-scale feature; is the basic scale feature; It is a high-scale feature; is the fused feature; is the weight of low-scale features; is the weight of the basic scale feature; is the weight of high-scale features; is a learnable linear frame projection, L Represents low scale, B Indicates basic scale, H Indicates high scale.

[0015] Furthermore, the input features are refined according to the following formula:

[0016] ;

[0017] in, For the l The input features of the temporal Mamba block, ; For the l Time Mamba block; For the l -1 Input features for the temporal Mamba block.

[0018] Furthermore, the feature representation of the video content is formed according to the following formula:

[0019] ;

[0020] in, is the feature representation of the video content; Parallel space Mamba blocks for shared parameters; For the t Frame-level features, .

[0021] Furthermore, the importance score is output according to the following steps:

[0022] Perform layer normalization on the output features of the multi-scale spatiotemporal modeling module;

[0023] By having The fully connected layer with activation function extracts higher-level semantic patterns of the features after layer normalization;

[0024] use The function normalizes the higher-level semantic patterns of the features into importance scores and outputs them.

[0025] Furthermore, the loss function is calculated according to the following formula:

[0026] ;

[0027] in, is the loss function; is the cross entropy loss function; is the mean square error loss function; is a tunable hyperparameter.

[0028] Furthermore, the cross entropy loss function is calculated according to the following formula:

[0029] ;

[0030] in, t is the index value of the frame; c is the result label of the classification; For the tThe true label of the frame; The frame importance classifier is t The prediction result of the frame.

[0031] Furthermore, the mean square error loss function is calculated according to the following formula:

[0032] ;

[0033] in, is the importance score of the prediction; is the true normalized importance score.

[0034] Furthermore, the summary is constructed according to the following steps:

[0035] The mean importance score is calculated by taking the weighted average of the optimized importance scores of each video clip. ;

[0036] Solve constrained optimization problems to construct a summary:

[0037] ;

[0038] in, i is the index value of the video segment; M is the number of video clips; For the i The number of frames in a video clip; is the video clip summary indicator, =1, indicating the i Video clips were selected for the video summary. =0 indicates the i Video clips are not important and are not selected; The total length of the original video.

[0039] In another aspect, the present invention provides a method for generating a video summary using multi-scale spatiotemporal modeling, comprising the following steps:

[0040] Build the multi-scale spatiotemporal modeling module mentioned above;

[0041] Divide the input video into non-overlapping segments and use convolutional neural networks to extract features of each video segment;

[0042] The multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of the input features, while adding segment category tags before the input features and applying a feature fusion strategy to aggregate the multi-scale features to obtain fused features. The cascaded temporal modeling module consists of multiple cascaded temporal Mamba blocks. The input features are refined through residual learning in multiple cascaded temporal Mamba blocks to fuse short-term cues and long-term event dependencies, and the refined features are output. The parallel spatial modeling module consists of multiple parallel spatial Mamba blocks with shared parameters. Each spatial Mamba block is used to process the features of a single frame. The input features are converted into frame-level features through a linear layer and then input into the spatial Mamba block. The spatial features are processed in parallel in the spatial Mamba blocks through residual learning to extract spatial features. The spatial features extracted by each spatial Mamba block are aggregated to form a feature representation of the video content.

[0043] According to the output features of the multi-scale spatiotemporal modeling module, the importance score is output and optimized in combination with the loss function;

[0044] Based on the optimized importance scores, a summary is constructed by solving the constrained optimization problem.

[0045] Compared with the prior art, the beneficial technical effects of the present invention are:

[0046] The present invention provides a multi-scale spatiotemporal modeling video summary generation method and device. By constructing a multi-scale spatiotemporal modeling module including a multi-scale aggregator, a cascaded temporal modeling module, and a parallel spatial modeling module, the method comprehensively captures the local details and global structure of the video. The multi-scale aggregator effectively aggregates multi-scale features through a feature fusion strategy to generate fused features, thereby capturing information at different granularity levels in the video content. The multi-scale aggregator also adds segment category tags before the input features, enabling the model to aggregate and summarize temporal information across frames in the video segments. The cascaded temporal modeling module captures temporal dependencies between frames through a temporal Mamba block, enhancing the model's ability to capture long-range temporal dependencies. The parallel spatial modeling module captures structural features within frames through a parallel spatial Mamba block, improving the model's ability to extract visual saliency cues within frames. Furthermore, its parallel processing capability can efficiently model spatial relationships and patterns within frames, significantly improving computational efficiency. Furthermore, the order of the multi-scale aggregator, the cascaded temporal modeling module, and the parallel spatial modeling module in the multi-scale spatiotemporal modeling module can be freely arranged, thereby improving the model's adaptability to different video features and enabling the model to have low computational complexity while maintaining high performance.

[0047] The multi-scale spatiotemporal modeling video summary generation method and device of the present invention have the advantages of low cost, simple structure, good performance and strong adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of a multi-scale spatiotemporal modeling video summary generation device provided by an embodiment;

[0050] Figure 2 A schematic diagram of a multi-scale spatiotemporal modeling module combination provided by an embodiment, wherein: Figure 2 (a) is a schematic diagram of the combination of multi-scale aggregator-cascaded time modeling module-parallel spatial modeling module. Figure 2 (b) Schematic diagram of the parallel spatial modeling module-multi-scale aggregator-cascaded temporal modeling module-concat combination. Figure 2 (c) Schematic diagram of the parallel spatial modeling module-multi-scale aggregator-cascaded temporal modeling module-pool combination;

[0051] Figure 3 A schematic diagram comparing the F1-score, number of floating-point operations, and number of parameters of different algorithms provided in one embodiment on the TVSum dataset;

[0052] Figure 4 A schematic diagram of qualitative results of different video summarization algorithms provided in an embodiment. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] Reference Figure 1 ,An embodiment provides a multi-scale spatiotemporal modeling video summary generation device, including a feature extraction module, a multi-scale spatiotemporal modeling module, an importance classifier, and a summary generation module;

[0055] The feature extraction module is used to divide the input video into non-overlapping segments and extract features of each video segment using a convolutional neural network;

[0056] The multi-scale spatiotemporal modeling module includes a multi-scale aggregator arranged in a free sequence, a cascaded time modeling module, and a parallel space modeling module; the multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of input features, and adds segment category tags before the input features, and applies a feature fusion strategy to aggregate multi-scale features to obtain fused features; the cascaded time modeling module is composed of multiple cascaded time Mamba blocks, and the input features are refined by residual learning in the multiple cascaded time Mamba blocks to fuse short-term prompts and long-term event dependencies, and output refined features; the parallel space modeling module is composed of multiple parallel space Mamba blocks with shared parameters, each space Mamba block is used to process the features of a single frame, and the input features are converted into frame-level features by a linear layer and then input into the space Mamba block, which is processed in parallel in the space Mamba block through residual learning to extract spatial features, and the spatial features extracted by each space Mamba block are aggregated to form a feature representation of the video content;

[0057] The importance classifier outputs an importance score based on the output features of the multi-scale spatiotemporal modeling module and optimizes the importance score in combination with a loss function;

[0058] The summary generation module is used to solve the constrained optimization problem and construct a summary based on the optimized importance scores.

[0059] The multi-scale aggregator is used to integrate video features of different granularity scales to ensure a more comprehensive understanding of the video sequence. The multi-scale aggregator uses deconvolution and pooling operations to obtain input features. Multi-scale features of the generated low-scale features , basic scale characteristics , high-scale features The input features It can be expressed as ,in, T is the number of frames in the video clip, and the spatial size of the input feature is , i is the number of input features, H is the height, W Furthermore, the segment category tag is added before the input features, so that the model can aggregate and summarize the temporal information across frames in the video segment.

[0060] In a preferred embodiment, the fused features are obtained according to the following formula:

[0061] ;

[0062] in, is the fused feature; is the weight of low-scale features; is the weight of the basic scale feature; is the weight of high-scale features; is a learnable linear frame projection, L Indicates low scale, B Indicates basic scale, H Indicates high scale.

[0063] The fused features of the multi-scale aggregator ,in, D is the number of channels.

[0064] The cascaded temporal modeling module consists of multiple cascaded temporal Mamba blocks. The bidirectional nature of the temporal Mamba blocks enables the model to integrate information from past and future frames, ensuring that the generated video summary is contextually relevant and temporally coherent. The cascaded temporal modeling module effectively improves the quality of video summaries by capturing key information from the input video features. Each temporal Mamba block selectively compresses information through its parallel scanning algorithm, enabling the model to process long sequence data without significantly increasing computational overhead. By stacking multiple temporal Mamba blocks, the model's ability to understand the semantics of video content from both forward and reverse perspectives is enhanced.

[0065] In a preferred embodiment, the input features are refined according to the following formula:

[0066] ;

[0067] in, For the l The input features of the temporal Mamba blocks, , ; For the l Time Mamba block; For the l -1 Input features for the temporal Mamba block.

[0068] The output features of the cascaded temporal modeling module are recorded as , which incorporates short-term cues and long-term event dependencies.

[0069] The parallel spatial modeling module consists of multiple parallel spatial Mamba blocks with shared parameters, each of which processes the features of a single frame. By focusing on a single frame, the parallel spatial modeling module is able to capture the complex spatial details and contextual information that are crucial for accurately summarizing visual content, ensuring that the generated summary is spatially coherent and contextually relevant. At the same time, the parallel nature of multiple spatial Mamba blocks allows for the simultaneous processing of multiple frames, significantly improving computational efficiency.

[0070] In a preferred embodiment, the feature representation of the video content is formed according to the following formula:

[0071] ;

[0072] in, is the feature representation of the video content; Parallel space Mamba blocks for shared parameters; For the t Frame-level features, .

[0073] Reference Figure 2 ,In one embodiment, three different combinations of multi-scale spatiotemporal modeling modules are presented.

[0074] Figure 2 (a) shows that the multi-scale spatiotemporal modeling module is a combination of a multi-scale aggregator, a cascaded temporal modeling module, and a parallel spatial modeling module; the multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of the input features, and then T The feature embeddings of the frames are merged into dimensions , forming the input of the cascaded temporal modeling module; adding a segment category tag to learn the global importance of video segments, forming the dimension , the cascaded temporal modeling module utilizes the temporal Mamba block to capture T The temporal correlation between frames, and the output dimension is The features of; Finally, the parallel spatial modeling module refines the intra-frame feature extraction, and the output dimension is characteristics.

[0075] Figure 2 (b) shows that the multi-scale spatiotemporal modeling module is a combination of parallel spatial modeling module-multi-scale aggregator-cascaded temporal modeling module-concat; after the parallel spatial modeling module extracts the spatial features within each frame, the multi-scale aggregator uses deconvolution and pooling operations to obtain the multi-scale features of the input features; then the multi-scale features are spliced ​​in the spatial dimension to form a dimension of The feature is then reduced to ,Finally, the temporal information is learned using the cascaded temporal modeling module.

[0076] Figure 2 (c) shows that the multi-scale spatiotemporal modeling module is a combination of parallel spatial modeling module-multi-scale aggregator-cascaded temporal modeling module-pool; after the parallel spatial modeling module extracts the spatial features within each frame, the multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of the input features, forming a dimension of The characteristics of; then use the cascade time modeling module to capture TThe temporal correlation between frames; then the output of each scale is average pooled and concatenated in the channel dimension to form a dimension of feature representation.

[0077] The outputs of the three multi-scale spatiotemporal modeling modules are input into the importance classifier to obtain the importance scores at the frame level.

[0078] In a preferred embodiment, the importance score is output according to the following steps:

[0079] Perform layer normalization on the output features of the multi-scale spatiotemporal modeling module;

[0080] By having The fully connected layer with activation function extracts higher-level semantic patterns of the features after layer normalization;

[0081] use The function normalizes the higher-level semantic patterns of the features into importance scores and outputs them.

[0082] In a preferred embodiment, the loss function is calculated according to the following formula:

[0083] ;

[0084] in, is the loss function; is the cross entropy loss function, which is used to measure the difference between the predicted category probability and the true label; is the mean squared error loss function, which is used to penalize deviations in continuous score predictions; is an adjustable hyperparameter used to control the loss balance.

[0085] The cross entropy loss function is calculated according to the following formula:

[0086] ;

[0087] in, t is the index value of the frame; c is the result label of the classification; For the t The true label of the frame; The frame importance classifier is t The prediction result of the frame.

[0088] The mean square error loss function is calculated according to the following formula:

[0089] ;

[0090] in, is the importance score of the prediction; is the true normalized importance score.

[0091] The summary is constructed according to the following steps:

[0092] The mean importance score is calculated by taking the weighted average of the optimized importance scores of each video clip. ;

[0093] Solve constrained optimization problems to construct a summary:

[0094] ;

[0095] in, i is the index value of the video segment; M is the number of video clips; For the i The number of frames in a video clip; is the video clip summary indicator, =1, indicating the i Video clips were selected for the video summary. =0 indicates the i Video clips are not important and are not selected; The total length of the original video.

[0096] In one embodiment, a method for generating a video summary using multi-scale spatiotemporal modeling is provided, comprising the following steps:

[0097] Building the multi-scale spatiotemporal modeling module described in any of the above embodiments;

[0098] Divide the input video into non-overlapping segments and use convolutional neural networks to extract features of each video segment;

[0099] The multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of the input features, while adding segment category tags before the input features and applying a feature fusion strategy to aggregate the multi-scale features to obtain fused features. The cascaded temporal modeling module consists of multiple cascaded temporal Mamba blocks. The input features are refined through residual learning in multiple cascaded temporal Mamba blocks to fuse short-term cues and long-term event dependencies, and the refined features are output. The parallel spatial modeling module consists of multiple parallel spatial Mamba blocks with shared parameters. Each spatial Mamba block is used to process the features of a single frame. The input features are converted into frame-level features through a linear layer and then input into the spatial Mamba block. The spatial features are processed in parallel in the spatial Mamba blocks through residual learning to extract spatial features. The spatial features extracted by each spatial Mamba block are aggregated to form a feature representation of the video content.

[0100] According to the output features of the multi-scale spatiotemporal modeling module, the importance score is output and optimized in combination with the loss function;

[0101] Based on the optimized importance scores, a summary is constructed by solving the constrained optimization problem.

[0102] In one embodiment, to verify the effectiveness of the present invention, a comparison is made with the existing method on the SumMe and TVSum datasets. The solution adopted in this embodiment is Solution A. Figure 2 As shown in (a), the evaluation indicators are F1-score, Kendall and Spearman.

[0103] The F1-score comparison results are shown in Table 1.

[0104] Table 1 Comparison results of different algorithms on F1-score

[0105]

[0106] As can be seen, compared to existing methods, our method surpasses existing methods in evaluation results on both the SumMe and TVSum datasets. Our method is able to more effectively learn temporal correlations between frames and spatial attention within frames to classify frame importance. Our method effectively provides top-tier video summarization results on both datasets.

[0107] Kendall ) (Kendall) and Spearman ( The comparison results of the (Spearman) correlation coefficient are shown in Table 2.

[0108] Table 2 Different algorithms in Kendall ( ) and Spearman ( ) on the comparison results

[0109]

[0110] As can be seen from Table 2, the present invention achieves the best and second best performance, indicating that there is a stronger consistency between the rankings provided by human annotations and model-predicted frame importance scores. The present invention is superior to Kendall ( ) and Spearman ( ) indicators demonstrates the high effectiveness of the present invention in learning complex spatiotemporal manipulation and spatial information in video frames.

[0111] Reference Figure 3 Table 3 compares the performance and complexity of various methods on the TVSum dataset. In the table, Scheme A, Scheme B, and Scheme C correspond to Figure 2(a) Figure 2 (b) Figure 2 (c) As shown in the combination. It can be seen that in the present invention, Figure 2 The setup shown in (a) achieves the best performance in terms of F1-score, Kendall and Spearman coefficients while maintaining a relatively small model size and low computational complexity compared to other methods. Figure 2 The setup shown in (c) achieves an F1-score of 66.6% with only 25.20M parameters, demonstrating excellent computational efficiency. Compared to the Transformer approach, the parameter count is reduced by 72.2%. Furthermore, with comparable FLOPs, it outperforms the CNN architecture by 3.8% in F1-score. Compared to lightweight approaches such as RR-STG, its F1-score is 22.7% higher. In summary, our method achieves an optimal balance between computational efficiency and summary accuracy.

[0112] Table 3 Different combinations and different algorithms in Kendall ( ) and Spearman ( ) on the comparison results

[0113]

[0114] Reference Figure 4 In order to intuitively compare the video summarization results, the qualitative results of the present invention are compared with those of the spatiotemporal visual transformer, the anchor-based flexible detection summarization network, and the anchor-free flexible detection summarization network. Figure 4 It can be seen that in the qualitative results of the 32nd video in the TVSum dataset, the method of the present invention shows higher similarity with the true label in terms of frame index and summary length, further illustrating the effectiveness of the present invention in maintaining consistency with the reference data.

[0115] Refer to Table 4, which shows Figure 2 The performance of the three combinations shown, Figure 2 The combination shown in (a) shows superior performance on both datasets, achieving an F1-score of 56.0% and a Kendall ( ) and Spearman's ( ), achieving an F1-score of 67.5% on TVSum; this shows that its combination effectively captures key spatial features while maintaining temporal coherence. Figure 2 (b) The temporal modeling capability of the combination shown in SumMe is compromised, as shown by 51.8% F1-score and a low correlation coefficient (Kendall (0.245) ) and Spearman's ( )); It maintains an F1-score of 66.7% on TVSum, showing reasonable spatial understanding ability. Figure 2 The combination shown in (c) achieves an F1-score of 48.2% on SumMe.

[0116] Table 4 Comparison results of different combinations on SumMe and TVSum

[0117]

[0118] In one embodiment, the impact of the cascaded temporal modeling module (CTMM) and the parallel spatial modeling module (PSMM) is evaluated through ablation experiments, and the results are shown in Table 5.

[0119] Table 5 Ablation experiments of CTMM and PSMM on TVSum dataset

[0120]

[0121] As can be seen from Table 5, when CTMM is disabled, the F1-score performance on the TVSum dataset decreases by 0.8%, Kendall ( ) and Spearman ( ) decreased by 0.049 and 0.062 respectively, indicating that the cascaded temporal modeling module plays a key role in establishing the temporal dependencies of non-adjacent frames. When the parallel spatial modeling module is removed, the F1-score decreases by 0.5%, and Kendall ( ) decreased by 0.067, Spearman ( ) decreased by 0.091, highlighting the ability of the parallel spatial modeling module to effectively extract discriminative spatial features through the multi-branch attention scheme.

[0122] Matters not covered by the present invention are known technologies.

[0123] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are all within the scope of protection of the present application.

[0125] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-scale spatiotemporal modeling video summary generation device, characterized in that: It includes feature extraction module, multi-scale spatiotemporal modeling module, importance classifier, and summary generation module; The feature extraction module is used to divide the input video into non-overlapping segments and extract features of each video segment using a convolutional neural network; The multi-scale spatiotemporal modeling module includes a multi-scale aggregator arranged in a free sequence, a cascaded time modeling module, and a parallel space modeling module; the multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of input features, and adds segment category tags before the input features, and applies a feature fusion strategy to aggregate multi-scale features to obtain fused features; the cascaded time modeling module is composed of multiple cascaded time Mamba blocks, and the input features are refined by residual learning in the multiple cascaded time Mamba blocks to fuse short-term prompts and long-term event dependencies, and output refined features; the parallel space modeling module is composed of multiple parallel space Mamba blocks with shared parameters, each space Mamba block is used to process the features of a single frame, and the input features are converted into frame-level features by a linear layer and then input into the space Mamba block, which is processed in parallel in the space Mamba block through residual learning to extract spatial features, and the spatial features extracted by each space Mamba block are aggregated to form a feature representation of the video content; The importance classifier outputs an importance score based on the output features of the multi-scale spatiotemporal modeling module, and optimizes the importance score in combination with a loss function; The summary generation module is used to solve the constrained optimization problem and construct a summary based on the optimized importance scores.

2. The multi-scale spatiotemporal modeling video summary generation device according to claim 1, characterized in that: The fused features are obtained according to the following formula: in, It is a low-scale feature; is the basic scale feature; It is a high-scale feature; is the fused feature; is the weight of low-scale features; is the weight of the basic scale feature; is the weight of high-scale features; is a learnable linear frame projection, L Indicates low scale, B Indicates basic scale, H Indicates high scale.

3. The multi-scale spatiotemporal modeling video summary generation device according to claim 1, characterized in that: Refine the input features according to the following formula: in, For the l The input features of the temporal Mamba blocks, ; For the l Time Mamba block; For the l -1 Input features for the temporal Mamba block.

4. The multi-scale spatiotemporal modeling video summary generation device according to claim 1, characterized in that: The feature representation of the video content is formed according to the following formula: in, is the feature representation of the video content; Parallel space Mamba blocks for shared parameters; For the t Frame-level features, .

5. The multi-scale spatiotemporal modeling video summary generation device according to claim 1, characterized in that: The importance score is output according to the following steps: Perform layer normalization on the output features of the multi-scale spatiotemporal modeling module; By having The fully connected layer with activation function extracts higher-level semantic patterns of the features after layer normalization; use The function normalizes the higher-level semantic patterns of the features into importance scores and outputs them.

6. The multi-scale spatiotemporal modeling video summary generation device according to claim 1, characterized in that: The loss function is calculated according to the following formula: in, is the loss function; is the cross entropy loss function; is the mean square error loss function; is a tunable hyperparameter.

7. The multi-scale spatiotemporal modeling video summary generation device according to claim 6, characterized in that: The cross entropy loss function is calculated according to the following formula: in, t is the index value of the frame; c is the result label of the classification; For the t The true label of the frame; The frame importance classifier is t The prediction result of the frame.

8. The multi-scale spatiotemporal modeling video summary generation device according to claim 6, characterized in that: The mean square error loss function is calculated according to the following formula: in, is the importance score of the prediction; is the true normalized importance score.

9. The multi-scale spatiotemporal modeling video summary generation device according to claim 1, characterized in that: The summary is constructed according to the following steps: The mean importance score is calculated by taking the weighted average of the optimized importance scores of each video clip. ; Solve constrained optimization problems to construct a summary: in, i is the index value of the video segment; M is the number of video clips; For the i The number of frames in a video clip; is the video clip summary indicator, =1, indicating the i Video clips were selected for the video summary. =0 indicates the i Video clips are not important and are not selected; The total length of the original video.

10. A multi-scale spatiotemporal modeling video summary generation method, characterized in that: The following steps are involved: Building the multi-scale spatiotemporal modeling module as claimed in claim 1; Divide the input video into non-overlapping segments and use convolutional neural networks to extract features of each video segment; The multi-scale aggregator uses deconvolution and pooling operations to obtain multi-scale features of the input features, while adding segment category tags before the input features and applying a feature fusion strategy to aggregate the multi-scale features to obtain fused features. The cascaded temporal modeling module consists of multiple cascaded temporal Mamba blocks. The input features are refined through residual learning in multiple cascaded temporal Mamba blocks to fuse short-term cues and long-term event dependencies, and the refined features are output. The parallel spatial modeling module consists of multiple parallel spatial Mamba blocks with shared parameters. Each spatial Mamba block is used to process the features of a single frame. The input features are converted into frame-level features through a linear layer and then input into the spatial Mamba block. The spatial features are processed in parallel in the spatial Mamba blocks through residual learning to extract spatial features. The spatial features extracted by each spatial Mamba block are aggregated to form a feature representation of the video content. According to the output features of the multi-scale spatiotemporal modeling module, the importance score is output and optimized in combination with the loss function; Based on the optimized importance scores, a summary is constructed by solving the constrained optimization problem.

Images