A multi-scale interaction and semantic calibration video summarization method decoupling motion and appearance

Abstract

The application discloses a kind of multi-scale interaction and semantic calibration video abstract method of motion and appearance decoupling, belong to computer vision technical field.The method includes: respectively extracting multi-scale motion feature and appearance feature to input video frame sequence, obtain corresponding feature pyramid;Motion feature pyramid and appearance feature pyramid are input video abstract model, and initial frame level importance score sequence is predicted by model;Again, through bidirectional LSTM, the score of key action boundary is strengthened by capturing interframe timing dependence, and the output optimized frame level importance score sequence is reduced by diversity penalty Redundant frame;Finally, by applying time interval constraint optimization frame distribution, video abstract is generated by selecting key frame through greedy algorithm.The method effectively solves the problems of motion and appearance feature coupling, insufficient multi-scale representation, simple feature fusion mechanism and modal weight imbalance in existing video abstract methods, and the experimental results on SumMe and TVSum benchmark datasets prove its effectiveness.

Description

Technical Field

[0001] This invention relates to a multi-scale interactive and semantically calibrated video summarization method that decouples motion and appearance, belonging to the field of computer vision technology. Background Technology

[0002] With the rapid growth in the number of videos, the efficient processing of massive amounts of video content has become an urgent need, giving rise to video summarization technology. This technology aims to solve the problem of low information retrieval efficiency in traditional video browsing methods in fields such as video surveillance, content retrieval, and social media by automatically extracting key content from videos to generate compact video summaries. However, existing video summarization methods still have three major limitations that hinder breakthroughs in intelligent summarization generation technology.

[0003] First, the coupling of motion and appearance features makes it difficult to distinguish between static and dynamic content. Traditional methods often use a simple combination of static features such as color histograms and textures with motion features such as optical flow. Deep learning methods also often treat the two as coupled features, making it difficult for the model to separate the association between static background and dynamic target.

[0004] Second, both motion and appearance features suffer from insufficient multi-scale representation, making it difficult to simultaneously capture global and local information. Existing methods often employ a single-scale mechanism in the feature extraction stage, which fails to capture motion information at multiple scales or to characterize appearance features at multiple scales.

[0005] Third, the feature fusion mechanism is simple, making it difficult to adaptively balance modal importance. Neither fixed-weight fusion of manually designed features nor feature splicing strategies in deep learning can dynamically adjust the contribution of motion and appearance features according to the video content.

[0006] The root of these problems lies in the fact that existing technologies do not delve deeply enough into the spatiotemporal semantics of videos. Therefore, overcoming these limitations is the key to promoting the evolution of video summarization technology towards a more intelligent and efficient direction. Summary of the Invention

[0007] To address the problems existing in existing methods, the present invention aims to provide a multi-scale interactive and semantic calibration video summarization method that decouples motion and appearance, and aims to solve the problems of existing video summarization methods in three aspects: coupling of motion and appearance features, insufficient multi-scale representation of motion and appearance features, and simple feature fusion mechanism and modal weight imbalance.

[0008] This invention provides a multi-scale interaction and semantic calibration video summarization method decoupling motion and appearance, the method comprising the following steps:

[0009] S1: Read the input video frame sequence, extract multi-scale motion features and appearance features using a feature extractor, and output the corresponding motion feature pyramid and appearance feature pyramid. The specific structure of the feature extractor and the pyramid generation process include:

[0010] Motion Feature Pyramid Generation: For video frame sequences, using MS-RAFT + The deep optical flow estimation network acquires pixel-level optical flow information and forms a four-level motion feature pyramid with scales of 1 / 16, 1 / 8, 1 / 4, and 1 / 2.

[0011] Appearance feature pyramid generation: For video frame sequences, basic appearance features are extracted using a pre-trained ResNet50 network, and then multi-scale transformation and fusion are performed using a feature pyramid network (FPN) to generate a 4-level appearance feature pyramid that matches the motion feature pyramid; at the same time, global semantic vectors are extracted from the highest 1 / 2 scale to provide a semantic benchmark for subsequent fusion.

[0012] S2: Input the motion feature pyramid and appearance feature pyramid into the video summarization generation model, and predict the initial frame-level importance score sequence through the model. The video summarization generation model includes:

[0013] Cross-modal semantic decoupler: The cross-modal semantic decoupler takes the motion feature pyramid and appearance feature pyramid as input, and uses attention guidance and gating mechanisms to decouple motion features and appearance features of the same scale, and outputs motion feature pyramids and appearance feature pyramids with independent semantic spaces;

[0014] Dynamic multi-scale interaction module: The dynamic multi-scale interaction module takes the motion feature pyramid and appearance feature pyramid of the independent semantic space as input, first performs modal alignment and splicing on the cross-modal motion features and appearance features of the same scale; then achieves global and local information complementarity through cross-scale bidirectional paths; finally, generates dynamic weights based on the cosine similarity between each scale feature and the global semantic vector, and outputs multi-scale frame-level fusion features through weighted fusion.

[0015] Semantic Consistency Calibrator: The semantic consistency calibrator takes multi-scale frame-level fusion features and global semantic vectors as inputs and achieves feature alignment with global topics through a triple mechanism. First, it calculates the cosine similarity between the frame-level fusion features at each scale and the global semantic vector, and filters and weakens features that deviate from the topic. Second, it applies L2 distance constraints to the retained features. Finally, it dynamically adjusts the frame loss weights based on the optical flow gradient to enhance the consistency of temporal key positions, and finally outputs the calibrated multi-scale frame-level fusion features.

[0016] Score prediction: Based on the calibrated multi-scale frame-level fusion features, the correlation between the weighted cosine similarity and the global semantic vector is calculated, and the initial frame-level importance score is output by Sigmoid function mapping. The initial importance scores of all frames are arranged in chronological order to obtain the initial frame-level importance score sequence.

[0017] S3: The initial frame-level importance score sequence is captured by bidirectional LSTM to capture inter-frame temporal dependencies, the scores of key action boundaries are enhanced, and redundant frames are reduced by diversity penalty to output the optimized frame-level importance score sequence.

[0018] S4: Optimize the frame distribution by applying time interval constraints to the optimized frame-level importance score sequence, select key frames using a greedy algorithm, and generate a video summary.

[0019] Preferably, constructing the cross-modal semantic decoupler includes:

[0020] We construct a cross-modal interaction mechanism that combines attention guidance and gating. Through two branches—motion-guided appearance decoupling and appearance-constrained motion decoupling—we achieve independent decoupling and semantic separation of motion and appearance features at corresponding scales.

[0021] Motion-guided appearance decoupling: Attention weights are generated using motion features to highlight dynamic information related to motion in appearance features. The formula for calculating motion-guided appearance attention weights is as follows: Where: Softmax(⋅) represents the normalization function, F M F represents the characteristics of motion. A Indicates appearance characteristics, W M W represents the learnable weight matrix acting on motion features. A This represents the learnable weight matrix applied to appearance features, and (⋅)T denotes the transpose operation. Indicates the scaling factor;

[0022] Appearance-constrained motion decoupling: A gating mechanism is used to suppress redundant information in motion features affected by appearance. The weight calculation formula for the gating mechanism is as follows: ,in: This represents the Sigmoid function. This indicates the splicing of motion and appearance features along the channel dimension. Indicates the gating parameters;

[0023] Independent semantic space mapping: decoupling motion features F at each scale M With appearance feature F A Each feature is mapped to an independent semantic space to generate a motion feature pyramid and an appearance feature pyramid.

[0024] Preferably, constructing the dynamic multi-scale interaction module includes:

[0025] A mechanism for coordinating motion and appearance features at the same scale is constructed. The number of channels for motion features and appearance features is uniformly adjusted by 1×1 convolution. Then, the features are concatenated along the channel dimension to form features at the same scale that contain information on motion and appearance coordination. Finally, LayerNorm is applied to the concatenated features for normalization to make the fused features more stable.

[0026] Construct bidirectional paths to enable cross-scale feature interaction, including bottom-up semantic guidance paths and top-down detail enhancement paths;

[0027] Dynamic scale weight fusion: based on features at each scale and the global semantic vector The similarity is used to generate weights, which are then fused at multiple scales to obtain multi-scale frame-level fusion features. The dynamic scale weight calculation formula is as follows: Where: exp(⋅) represents the exponential function, Represents the cosine similarity function. Represents s-scale features. Represents the global semantic vector; the multi-scale fusion calculation formula is: ,in: The dynamic weights representing the s-scale features. This represents the s-scale feature.

[0028] Preferably, constructing the semantic consistency calibrator includes:

[0029] Construct a similarity screening mechanism and calculate the fusion features at each scale level. With global semantic vector The cosine similarity is used to filter out features with similarity below a threshold, reducing their weight in subsequent calculations; the cosine similarity calculation formula is: ,

[0030] in Represents the cosine similarity function. This represents the s-scale fusion feature. This is a global semantic vector;

[0031] Construct multi-scale L2 distance constraints and calculate fused features at each scale. and The L2 loss; the L2 distance constraint calculation formula is:

[0032] ,

[0033] Where: T represents the length of the feature sequence, t represents the loop variable, and Pooling(⋅) represents the global average pooling operation. This represents the s-scale fusion feature, where (t,:) indicates selecting all column elements in the t-th row. This is a global semantic vector;

[0034] The loss weights for each frame are dynamically adjusted based on the optical flow feature gradient. The formula for calculating the optical flow loss weights is: w t =Grad(F M (t)), Grad(⋅) represents the optical flow gradient calculation function, F M (t) represents the optical flow characteristics of frame t; after optimization by a triple mechanism, the output calibrated multi-scale frame-level fusion feature F is obtained. t .

[0035] Preferably, constructing the score prediction includes:

[0036] The calibrated multi-scale frame-level fusion feature F t The feature vectors are split into four independent feature vectors corresponding to different scales. Weights are assigned to each scale, and the cosine similarity of each scale is weighted and summed according to the preset weight ω to obtain the comprehensive relevance score of the t-th frame.

[0037] The overall relevance score is mapped to the [0,1] interval using the Sigmoid function to generate the final initial frame-level importance score. The initial importance scores of all frames are arranged in chronological order to obtain the initial frame-level importance score sequence.

[0038] Preferably, constructing an optimized frame-level importance score sequence includes:

[0039] A bidirectional LSTM is constructed to extract temporal features. Temporal enhancement features are obtained by concatenating the forward and backward hidden states. The mean cosine similarity between each frame and its adjacent frames is calculated as the temporal association strength. The temporal score gradient is calculated. If the gradient exceeds the threshold, the frame is identified as a key action frame, and its importance score is increased.

[0040] A diversity penalty is calculated by calculating the cosine similarity of features between any two calibrated frames. If the similarity of a frame t with other frames exceeds a threshold and the scores of other frames are higher, a penalty is applied to frame t. The temporal enhancement and diversity penalty are combined to obtain the final optimized frame-level importance score sequence.

[0041] Preferably, the generation of video summaries includes:

[0042] To filter the optimized frame-level importance score sequence, firstly, the frame with the highest score is added to set S; if the index interval between a subsequent candidate frame and an existing frame in set S is less than the minimum threshold, it is temporarily moved to the waiting set. This process is repeated until the candidate pool is empty or set S reaches the target size.

[0043] A greedy algorithm is used to select keyframes sequentially from the top of the candidate pool until the number reaches the length of the target summary. Finally, the discrete keyframe index set S is converted into continuous video segments to generate the final video summary.

[0044] Compared with the prior art, the present invention has at least the following beneficial effects:

[0045] 1. This invention provides a multi-scale interactive and semantically calibrated video summarization method that decouples motion and appearance. It extracts multi-scale motion and appearance features by constructing a feature pyramid, removes redundancy through cross-modal semantic decoupling, achieves bidirectional fusion through dynamic multi-scale interaction, and uses a semantic consistency calibrator to constrain the fused features with global semantic vectors. Furthermore, it combines bidirectional LSTM to enhance key action boundary scores and diversity penalties to reduce redundant frames, and then uses time interval constraints and a greedy algorithm to select frames, generating high-quality video summaries. This method solves the problems of motion and appearance feature coupling, insufficient multi-scale representation, simplistic feature fusion, and modal weight imbalance in existing technologies.

[0046] 2. This invention proposes a cross-modal semantic decoupler. By constructing a cross-modal interaction mechanism with attention guidance and gating, it can accurately separate motion and appearance information at corresponding scales and effectively remove redundant features contained in both.

[0047] 3. This invention designs a dynamic multi-scale interaction module, which realizes multi-scale feature interaction through a three-level process of same-scale modal collaboration, cross-scale bidirectional interaction, and dynamic weight fusion. This solves the problems of global and local information imbalance and the poor adaptability of traditional single-scale models to complex scenes.

[0048] 4. This invention designs a semantic consistency calibrator that achieves precise alignment between features and the global semantic vector through a triple mechanism of similarity filtering, L2 distance constraint, and dynamic optical flow adjustment. This effectively resolves semantic ambiguity in local features and ensures a high degree of consistency between the summary content and the video topic.

[0049] 5. Experiments on the SumMe and TVSum benchmark datasets show that, compared with the state-of-the-art methods, the proposed method significantly improves the performance of video summarization generation, effectively solving the problems of motion and appearance feature coupling, insufficient multi-scale representation, simple feature fusion mechanism, and modality weight imbalance in existing video summarization methods. Attached Figure Description

[0050] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0051] Figure 1The flowchart illustrates the implementation of a multi-scale interaction and semantic calibration video summarization method for decoupling motion and appearance, as provided by this invention.

[0052] Figure 2 This is an overall framework diagram of a multi-scale interaction and semantic calibration video summarization generation model that decouples motion and appearance, as provided in an embodiment of the present invention.

[0053] Figure 3 This is a structural diagram of the cross-modal semantic decoupler provided in an embodiment of the present invention;

[0054] Figure 4 This is a structural diagram of the dynamic multi-scale interaction module provided in an embodiment of the present invention;

[0055] Figure 5 This is a structural diagram of a semantic consistency calibrator provided in an embodiment of the present invention. Detailed Implementation

[0056] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0057] like Figure 1 The diagram shown is a flowchart illustrating the implementation of a multi-scale interaction and semantic calibration video summarization method decoupling motion and appearance, as provided by the present invention. This embodiment provides a multi-scale interaction and semantic calibration video summarization method decoupling motion and appearance, comprising the following steps:

[0058] S1: Read the input video frame sequence, extract multi-scale motion features and appearance features through the feature extractor, and output the motion feature pyramid and appearance feature pyramid.

[0059] S2: Input the motion feature pyramid and appearance feature pyramid into the video summarization generation model, and predict the initial frame-level importance score sequence through the model.

[0060] S3: The initial frame-level importance score sequence is captured by bidirectional LSTM to capture inter-frame temporal dependencies, the scores of key action boundaries are enhanced, and redundant frames are reduced by diversity penalty to output the optimized frame-level importance score sequence.

[0061] S4: The optimized frame-level importance score sequence is optimized by applying time interval constraints to optimize the frame distribution, and finally key frames are selected by a greedy algorithm to generate a video summary.

[0062] Example 1

[0063] This invention provides a preferred embodiment that executes step S1, reading the input video frame sequence, extracting multi-scale motion features and appearance features using a feature extractor, and outputting a motion feature pyramid and an appearance feature pyramid. The specific steps are as follows:

[0064] Video frame sequence reading and preprocessing: Frame sequence V is extracted from the original video at a sampling rate of 2 FPS. Then, each frame is uniformly scaled to 256×256 pixels, and the pixel values ​​are normalized. Random horizontal flipping is applied to enhance data diversity.

[0065] Motion Feature Pyramid Generation: The frame sequence is split into consecutive frame pairs in sequence, and each frame pair is input into MS-RAFT. + The deep optical flow estimation network obtains pixel-level optical flow information and acquires motion feature pyramids with four resolutions: 1 / 16, 1 / 8, 1 / 4, and 1 / 2.

[0066] Appearance feature pyramid generation: The frame sequence is input into the pre-trained ResNet50 network to obtain basic appearance features. Then, the feature pyramid network FFN generates 4-level scale appearance features. The four scale appearance features are then uniformly adjusted through 3×3 convolution to obtain an appearance feature pyramid that perfectly matches the motion feature dimension.

[0067] Example 2

[0068] The present invention provides a preferred embodiment for executing S2, which inputs the motion feature pyramid and the appearance feature pyramid into the video summarization generation model, and predicts the initial frame-level importance score sequence through the model.

[0069] like Figure 2 The diagram shown illustrates the overall framework of a multi-scale interaction and semantic calibration video summarization generation model decoupled from motion and appearance, as provided in this embodiment. This video summarization generation model comprises four parts: a cross-modal semantic decoupler, a dynamic multi-scale interaction module, a semantic consistency calibrator, and a score prediction module. The specific construction steps of each module will be described below:

[0070] S21 constructs a cross-modal semantic decoupler, the structure of which is shown in the figure below. Figure 3 As shown.

[0071] Input: Receive the motion feature pyramid and appearance feature pyramid from S1, and match them one-to-one according to resolution.

[0072] We construct a cross-modal interaction mechanism with attention guidance and gating to achieve independent decoupling and semantic separation of motion features and appearance features at corresponding scales.

[0073] Motion-guided appearance decoupling branch: This involves decoupling motion features F... M Through the learnable weight matrix W MConvert the appearance feature F into a query vector. A Through the weight matrix W A The data is converted into key vectors and value vectors. The similarity between the query vector of motion features and the key vector of appearance features is calculated, and the attention weight A is obtained by normalization using the Softmax function. MA Using attention weight A MA The motion-guided appearance features are obtained by weighted summation of the value vectors of the appearance features.

[0074] The formula for attention weights is: Where: Softmax(⋅) represents the normalization function, F M F represents the characteristics of motion. A Indicates appearance characteristics, W M W represents the learnable weight matrix acting on motion features. A This represents the learnable weight matrix applied to appearance features, and (⋅)T denotes the transpose operation. Indicates the scaling factor;

[0075] Appearance constraint motion decoupling branch: Decouple appearance feature F A With motion characteristics F M The concatenation is performed along the channel dimension, and the gate weights G are generated using learnable gate parameters Wg, bg, and a sigmoid function. AM， Using gating weight G AM The motion features are weighted element by element to obtain the motion features with appearance constraints. The gating weight calculation formula is as follows: ,in This represents the Sigmoid function. This indicates the splicing of motion and appearance features along the channel dimension. Indicates the gating parameters;

[0076] Independent semantic space mapping: decoupling motion features F at each scale M With appearance feature F A These are mapped to independent semantic spaces to generate motion feature pyramids and appearance feature pyramids; for motion feature F... M， 3D convolution is used to construct motion semantic mapping, and F A Mapping to the motion semantic space Z M The output motion feature pyramid is calculated using the following formula: Wherein: F M Indicating motion characteristics, C 3D (⋅) represents a 3D convolution operation, and ReLU(⋅) represents an activation function; for appearance features F A The static semantics of appearance features are extracted using a pre-trained ResNet50 network, and the F... AMapping to the appearance semantic space Z A Output the appearance feature pyramid; the calculation formula for the mapping is: , where F A Indicates appearance characteristics;

[0077] Output: The final output consists of a motion feature pyramid and an appearance feature pyramid with independent spatial semantics.

[0078] S22 constructs a dynamic multi-scale interactive module, the structure of which is shown in the diagram below. Figure 4 As shown.

[0079] Input: Receives the motion feature pyramid and appearance feature pyramid of independent spatial semantics from the cross-modal decoupler output.

[0080] Construct a mechanism for coordinating motion and appearance at the same scale, and incorporate motion features at the same scale through 1×1 convolution. Appearance features The number of channels was adjusted to 256, resulting in and The features, after being stitched together and unified in the channel dimension, form a homogeneous feature that includes information on motion and appearance coordination.

[0081] Constructing cross-scale bidirectional path interactions can be divided into bottom-up path construction and top-down path construction.

[0082] The bottom-up semantic guidance path involves performing bilinear interpolation upsampling on coarse-scale features to inject global semantics into fine-scale features. The upsampling operation is calculated using the following formula:

[0083] Where: BilinearUpsample(⋅) represents the bilinear interpolation upsampling operation, Represents coarse-scale primitive features. The upsampled features are represented by `scale`, which is the upsampling scaling factor. Then, the upsampled features and the original features are fused through residual connections. The formula for calculating the residual fused features is as follows: ,in: This represents the original features at the current fine scale. This represents the residual weights, and LayerNorm(⋅) is the normalization operation. This indicates that the features after upsampling are executed iteratively from the coarsest scale to a finer scale, following the process described above.

[0084] The top-down detail enhancement path involves applying average pooling downsampling to fine-scale features, then generating an attention guide map to adapt the fine-scale features to the coarse-scale features. The downsampling operation is calculated using the following formula: ,in: Indicates the features after downsampling. This represents the average pooling function. Represents the original features at a fine scale, where kernel is the pooling kernel size; the formula for calculating the attention-guided map is:

[0085] Where: Softmax(⋅) represents the normalization function, W a Indicates learnable weights, Indicates the features after downsampling. Represents coarse-scale primitive features. This indicates that the channel splicing operation, following the above process, starts from the finest scale and iterates layer by layer towards a coarser scale.

[0086] Dynamic scale weight fusion: based on features at each scale and the global semantic vector The similarity is used to generate weights, which are then fused at multiple scales to obtain multi-scale frame-level fusion features. The dynamic scale weight calculation formula is as follows: Where: exp(⋅) represents the exponential function, Represents the cosine similarity function. Represents s-scale features. Represents the global semantic vector; the multi-scale fusion calculation formula is: ,in: The dynamic weights representing the s-scale features. This represents the s-scale feature.

[0087] Output: Weighted summation of features at all scales, outputting multi-scale frame-level fusion features.

[0088] S23 constructs a semantic consistency calibrator, the structure of which is shown in the diagram below. Figure 5 As shown.

[0089] Input: Receive multi-scale frame-level fusion features output by the dynamic multi-scale interaction module.

[0090] Constructing a similarity filtering mechanism: Through four parallel 1×1 convolutional layers, the fused features at each scale are compared with C. global Dimension matching. Then calculate the feature at each scale level and C. global The cosine similarity, and the similarity screening formula is: ,in Represents the cosine similarity function. This represents the s-scale fusion feature. This is the global semantic vector.

[0091] Constructing multi-scale L2 distance constraints: For the filtered features, calculate the distance with C. global The L2 distance loss, the L2 distance constraint calculation formula is:

[0092] Where: T represents the length of the feature sequence, t represents the loop variable, and Pooling(⋅) represents the global average pooling operation. This represents the s-scale fusion feature, where (t,:) indicates selecting all column elements in the t-th row. This is the global semantic vector.

[0093] Dynamic adjustment of optical flow: Different weights are assigned to each frame based on the optical flow gradient, and the optical flow gradient Grad(F) is used for dynamic adjustment. M The higher the weight w is, the higher the weight w is. t The larger the value, the more complex the formula for calculating the optical flow loss weight is: w t =Grad(F M (t)), Grad(⋅) represents the optical flow gradient calculation function, F M (t) represents the optical flow characteristics of the t-th frame.

[0094] Output: Optimize the above loss through backpropagation, and output the calibrated multi-scale frame-level fusion feature F. t .

[0095] S24 Score Prediction

[0096] calibrated multi-scale frame-level fusion features It is split into 4 independent feature vectors according to scale, corresponding to 4 scale levels. Assign weights to each scale The cosine similarity at the four scales is weighted according to a preset weight. The weighted sum is used to obtain the overall relevance score of frame t. This overall relevance score is then mapped to the [0,1] interval using the Sigmoid function. The initial importance scores of all frames are arranged in chronological order to obtain the initial frame-level importance score sequence.

[0097] Output: Generates the initial frame-level importance score sequence.

[0098] Example 3:

[0099] The present invention provides a preferred embodiment for executing S3, which optimizes the frame-level importance score sequence by capturing inter-frame temporal dependencies through bidirectional LSTM, strengthening the scores of key action boundaries, and reducing redundant frames through diversity penalty.

[0100] Input: Initial frame-level importance score sequence.

[0101] To construct a bidirectional LSTM for temporal feature extraction, the initial frame-level importance score sequence is input into the bidirectional LSTM to capture the temporal dependencies between contexts, thus obtaining the forward and backward hidden states.

[0102] By concatenating the forward and backward hidden states, we obtain temporally enhanced features. Output a temporally enhanced frame-level feature sequence. For each frame t, calculate its temporal association strength with adjacent frames. The temporal association calculation formula is as follows: Where: sim(⋅) represents the cosine similarity function, This represents the feature vector corresponding to frame t. This represents the feature vector corresponding to the previous adjacent frame of frame t. This represents the feature vector corresponding to the next adjacent frame after the t-th frame;

[0103] Calculate the gradient of the time series score and the time series score difference. ,like Exceeding the threshold If frame t is considered to be at a critical position in the action change, its importance score is increased. The formula for calculating the increased importance score is as follows: ,in Let be the initial importance score for frame t. To adjust the coefficient, Indicates the time series score difference;

[0104] Perform diversity penalty calculation to calculate the semantic similarity between any two frames. ,in: Represents the cosine similarity function. , For frame-level features, for each frame t, if there exist other frames that satisfy... , Then a penalty is applied to frame t.

[0105] Output: Combining temporal boosting and diversity penalty, we obtain the optimized frame-level importance score sequence.

[0106] Example 4:

[0107] The present invention provides a preferred embodiment to execute S4, which optimizes the frame distribution by applying time interval constraints based on the optimized frame-level importance score sequence, and finally selects key frames through a greedy algorithm to generate a video summary.

[0108] Input: The optimized sequence of frame-level importance scores.

[0109] To filter the optimized frame-level importance score sequence, firstly, the frame with the highest score is added to set S; subsequent candidate frames whose index interval with an existing frame in set S is less than the minimum threshold are temporarily moved to the waiting set. This process is repeated until the candidate pool is empty or set S reaches the target size.

[0110] A greedy algorithm is used to select keyframes sequentially from the top of the candidate pool until the number reaches the length of the target summary.

[0111] Output: The frame indices in the keyframe set S are sorted in chronological order, converted into continuous video segments, and spliced ​​together to form the final video summary.

[0112] To verify the effectiveness of the above embodiments, the present invention was applied in practice. The F-score (%) was calculated and compared with existing state-of-the-art methods. Experiments were conducted on two benchmark datasets, SumMe and TVSum, to evaluate the performance of the present invention in terms of keyframe selection accuracy, summary information completeness, and temporal coherence. SumMe is a popular video summarization dataset containing 25 videos from everyday life, sports, and other scenarios, with each frame having multiple user-annotated importance scores. TVSum consists of 50 videos covering news, documentaries, sports, and other categories, providing multi-user-annotated key segments. Both datasets are suitable for testing the model's generalization ability across different video types.

[0113] This experiment selected several mainstream video summarization methods for comparison, including:

[0114] The experimental results for DR-DAN, SUM-GPA, VJMHT, MAMI, and MGAF-SUM under standard evaluation settings are shown in Table 1. The method of this invention achieved the best summarization performance on both datasets.

[0115] Table 1. Comparison of F-scores with advanced methods

[0116]

[0117] Experimental results on the SumMe and TVSum datasets show that the proposed method outperforms existing methods, mainly due to the following innovations: The method constructs a motion feature pyramid and an appearance feature pyramid, removes redundancy between motion and appearance features through cross-modal decoupling, achieves bidirectional fusion through multi-scale interaction, uses semantic consistency calibration to constrain features with global semantics, and combines temporal enhancement and redundancy penalty to solve the problems of coupling motion and appearance features, insufficient multi-scale representation, and simple feature fusion in existing technologies, thereby improving the quality of video summarization.

Claims

1. A multi-scale interaction and semantic calibration video summarization method decoupled motion from appearance, characterized in that, The method comprises the following steps: S1: reading an input video frame sequence, extracting multi-scale motion features and appearance features respectively through a feature extractor, and outputting corresponding motion feature pyramids and appearance feature pyramids, the specific composition of the feature extractor and the pyramid generation process comprising: Motion feature pyramid generation: for the video frame sequence, MS-RAFT is used + The depth optical flow estimation network obtains pixel-level optical flow information, and forms a 4-level motion feature pyramid containing 1 / 16, 1 / 8, 1 / 4 and 1 / 2 scales; Appearance feature pyramid generation: for the video frame sequence, a pre-trained ResNet50 network is used to extract basic appearance features, and a feature pyramid network FPN is used for multi-scale conversion and fusion to generate a 4-level appearance feature pyramid matched with the motion feature pyramid; at the same time, a global semantic vector is extracted from the highest 1 / 2 scale to provide a semantic reference for subsequent fusion; S2: inputting the motion feature pyramid and the appearance feature pyramid into a video summary generation model to predict an initial frame-level importance score sequence through the model, the video summary generation model comprising: Cross-modal semantic decoupler: the cross-modal semantic decoupler takes the motion feature pyramid and the appearance feature pyramid as input, adopts an attention guide and a gating mechanism to decouple the motion features and the appearance features at the same scale in independent semantic spaces, and outputs the motion feature pyramid and the appearance feature pyramid in the independent semantic spaces; Dynamic multi-scale interaction module: the dynamic multi-scale interaction module takes the motion feature pyramid and the appearance feature pyramid in the independent semantic spaces as input, first performs modal alignment and splicing on the motion features and the appearance features at the same scale and in the cross-modal mode, then realizes global and local information complementation through a cross-scale bidirectional path, finally generates dynamic weights based on the cosine similarity of the features at each scale and the global semantic vector, and outputs multi-scale frame-level fusion features through weighted fusion; Semantic consistency calibrator: the semantic consistency calibrator takes the multi-scale frame-level fusion features and the global semantic vector as input, realizes feature and global theme alignment through a triple mechanism, first calculates the cosine similarity of the multi-scale frame-level fusion features and the global semantic vector, filters and weakens the features deviating from the theme, then applies an L2 distance constraint to the retained features, and finally adjusts the frame loss weight dynamically according to the optical flow gradient to strengthen the consistency of the key positions in time sequence, and finally outputs the calibrated multi-scale frame-level fusion features; Score prediction: based on the calibrated multi-scale frame-level fusion features, the relevance with the global semantic vector is calculated through weighted cosine similarity, and the initial frame-level importance score is outputted through a Sigmoid function mapping, and the initial importance scores of all frames are arranged in time sequence, and the initial frame-level importance score sequence is obtained; S3: the initial frame-level importance score sequence is captured through a bidirectional LSTM to strengthen the scores of key action boundaries, and the diversity penalty is used to reduce redundant frames, and an optimized frame-level importance score sequence is outputted; S4: the optimized frame-level importance score sequence is optimized through time interval constraint, and finally key frames are selected through a greedy algorithm to generate a video summary.

2. A multi-scale interaction and semantic calibration video summarization method decoupled motion and appearance, as claimed in claim 1, wherein, The cross-modal semantic decoupler comprises: A cross-modal interaction mechanism of attention-guided and gated mechanism is constructed, motion-guided appearance decoupling and appearance-constrained motion decoupling are realized through two branches, and independent decoupling and semantic separation of motion and appearance features are realized in the corresponding scale; Motion-guided appearance decoupling: the motion features are used to generate attention weights, and the dynamic information related to the motion in the appearance features is highlighted, and the motion-guided appearance attention weight calculation formula is: ， where: Softmax(·) denotes a normalization function, F M denotes motion features, F A denotes appearance features, W M denotes a learnable weight matrix acting on motion features, W A denotes a learnable weight matrix acting on appearance features, (·) T denotes a transpose operation, denotes a scaling factor; Appearance-constrained motion decoupling: the redundant information in the motion features disturbed by the appearance is inhibited through the gating mechanism, and the gating mechanism weight calculation formula is: ， wherein: represents a Sigmoid function, represents motion and appearance features concatenation in channel dimension, represents a gating parameter; independent semantic space mapping: decoupling each scale of motion feature F M and appearance feature F A , respectively, to generate the motion feature pyramid and appearance feature pyramid in the independent semantic space.

3. The multiscale interactive and semantic calibration video summarization method decoupled from motion and appearance of claim 1, wherein, The dynamic multi-scale interaction module comprises: A same-scale motion and appearance feature collaborative mechanism is constructed, the channel numbers of the motion features and the appearance features are uniformly adjusted through 1*1 convolution, then the channel dimension is spliced to form same-scale features containing motion and appearance collaborative information, and finally, the spliced features are subjected to LayerNorm for normalization, so that the fused features are more stable; A bidirectional path is constructed to realize cross-scale feature interaction, including a semantic-guided bottom-up path and a detail-enhanced top-down path; Dynamic scale weight fusion: according to the similarity of each scale feature and the global semantic vector , generate weights, then perform multi-scale fusion, and finally obtain multi-scale frame-level fusion features; the dynamic scale weight calculation formula is: , wherein: represents the exponential function, denotes a cosine similarity function, denotes s scale feature, denotes a global semantic vector; a multi-scale fusion calculation formula is: wherein: denotes a dynamic weight of the s scale feature, denotes s scale feature.

4. The multiscale interactive and semantic calibration video summarization method decoupled motion from appearance as claimed in claim 1, wherein, The semantic consistency calibrator comprises: A similarity screening mechanism is constructed to calculate the scale fusion features at each level with the global semantic vector Cosine similarity is calculated, and features with a similarity lower than a threshold value are screened out, and their weights in subsequent calculations are reduced. The cosine similarity calculation formula is: wherein denotes a cosine similarity function, denotes the s-scale fusion feature, is a global semantic vector; A multi-scale L2 distance constraint is constructed to calculate the fusion features of each scale With L2 loss; the L2 distance constraint calculation formula is: ， wherein: T represents the length of the feature sequence, t represents the loop variable, Pooling(·) represents the global average pooling operation, represents the s scale fusion feature, (t, :) represents all column elements selected from the tth row, is a global semantic vector; According to the gradient of the optical flow feature, the loss weight of each frame is dynamically adjusted, and the optical flow loss weight calculation formula is: w t =Grad(F M (t)),Grad(⋅) represents an optical flow gradient calculation function, F M (t) represents the optical flow feature of the t-th frame;After optimization by the triple mechanism, the calibrated multi-scale frame-level fusion feature F t is output.

5. The multiscale interactive and semantic calibration video summarization method decoupled motion from appearance as claimed in claim 1, wherein, The score prediction comprises: The calibrated multi-scale frame-level fusion feature F t The four independent feature vectors correspond to different scales, and each scale is assigned a weight ω. The cosine similarity of each scale is weighted and summed according to the preset weight ω to obtain the comprehensive correlation score of the t-th frame. The comprehensive correlation score is mapped to the [0, 1] interval through a Sigmoid function to generate the final initial frame-level importance score, and the initial importance scores of all frames are arranged in time sequence to obtain an initial frame-level importance score sequence.

6. The multiscale interactive and semantic calibration video summarization method decoupled motion from appearance as claimed in claim 1, wherein, The optimized frame-level importance score sequence comprises: A bidirectional LSTM is constructed to extract time sequence features, and time sequence enhanced features are obtained by splicing forward and backward hidden states; the cosine similarity mean value of each frame and adjacent frame is calculated as the time sequence correlation strength, and the time sequence score gradient is calculated, if the threshold is exceeded, it is determined as a key action frame, and the importance score is improved; A diversity penalty calculation is performed, the cosine similarity of the calibrated features of any two frames is calculated, if the similarity of a frame t and other frames exceeds the threshold and the score of the other frames is higher, the frame t is punished; the time sequence enhancement and the diversity penalty are combined to obtain the optimized frame-level importance score sequence.

7. The multiscale interactive and semantically calibrated video summarization method decoupled from motion and appearance of claim 1, wherein, The video summary comprises: The optimized frame-level importance score sequence is screened, first, the frame with the highest score is selected into a set S; if the index interval between the subsequent candidate frame and the frame in the set S is less than a minimum threshold, the candidate frame is temporarily moved to a waiting set, and the operation is repeated until the candidate pool is empty or the set S reaches the target size; A greedy algorithm is used to select key frames from the top of the candidate pool, until the number reaches the length of the target summary, finally, the discrete key frame index set S is converted into a continuous video segment to generate the final video summary.

Images