A multi-scale temporal modeling video summary generation method fusing semantic enhancement and boundary perception

Abstract

The application discloses a kind of multi-scale time sequence modeling video abstract generation method of fusion semantic enhancement and boundary perception, belong to computer vision technical field.The method includes: the video frame sequence of input is encoded, and the corresponding time sequence feature representation is extracted;The time sequence feature extracted is input video abstract generation model, and the importance score is output by model;The importance score is time sequence smoothing and peak detection, and the candidate set of key frame is determined;Based on the candidate set of key frame, in combination with non-maximum suppression and segment merging strategy, the final video abstract segment is generated.The experimental results obtained on benchmark dataset SumMe and TVSum prove the advancement of the method.The video abstract generation method proposed in the application can effectively solve the problems of event boundary ambiguity and insufficient multi-scale time sequence understanding in existing video abstract generation methods.

Description

A Multi-Scale Temporal Modeling Video Summarization Generation Method Integrating Semantic Enhancement and Boundary Awareness Technical Field

[0001] This invention relates to a multi-scale temporal modeling video summarization method that integrates semantic enhancement and boundary awareness, belonging to the field of computer vision technology. Background Technology

[0002] Video summarization technology aims to automatically condense lengthy raw videos into concise, coherent, and information-rich summaries, a key technology for addressing the inefficiencies in storage, retrieval, and browsing caused by the explosive growth of video data. Current mainstream methods can be broadly categorized into two types: frame importance scoring-based methods and temporal segment detection-based methods. Frame importance scoring-based methods typically assign an importance score to each frame in a video frame sequence, then select high-scoring frames to form a summary. While these methods effectively identify key content, they often ignore the temporal dependencies between frames, resulting in temporally disjointed summary segments lacking contextual logic. Temporal segment detection-based methods attempt to directly detect meaningful complete segments from the video as candidate summaries, their performance largely depending on pre-defined temporal anchors or sliding windows. However, fixed-scale or sparse anchor designs struggle to adapt to the significant variations in the duration of actions and events in videos, easily leading to inaccurate segment boundary localization, especially insensitivity to short-duration actions or long-duration events. In recent years, deep learning techniques have made significant progress in both of these frameworks, particularly by introducing attention mechanisms to model long-range dependencies. Despite this, existing models still have significant shortcomings in three key areas: how to refine the temporal semantics of videos, how to effectively integrate multi-scale context to understand events of different durations, and how to accurately perceive and locate the start and end boundaries of actions or events. Specifically, semantic modeling often remains at the frame level, lacking explicit enhancement and utilization of local semantic differences; multi-scale information fusion methods are relatively simple and fail to adaptively strengthen key temporal regions; and the utilization of boundary information is insufficient, resulting in blurred boundaries in the generated summary fragments and poor alignment with real-world event boundaries. Therefore, a video summarization generation method that deeply integrates semantic understanding, multi-scale temporal analysis, and boundary awareness is needed to improve the completeness, coherence, and boundary accuracy of summaries. Summary of the Invention

[0003] To address the problems existing in existing methods, the present invention aims to provide a multi-scale temporal modeling video summarization generation method that integrates semantic enhancement and boundary awareness, thereby solving the problems of blurred event boundaries and insufficient multi-scale temporal understanding in existing video summarization generation methods.

[0004] This invention provides a multi-scale temporal modeling video summarization generation method that integrates semantic enhancement and boundary awareness, the method comprising the following steps:

[0005] S1: Perform feature encoding on the input video frame sequence and extract the corresponding temporal feature representation;

[0006] S2: Input the extracted temporal features into the video summarization generation model, and the model outputs an importance score. The video summarization generation model includes:

[0007] Semantic enhancement module: The semantic enhancement module takes the temporal features as input, calculates local semantic differences through depthwise separable convolution, and uses semantic weighting units to dynamically generate feature enhancement weights based on the local semantic differences. The temporal features are multiplied element-wise with the feature enhancement weights to obtain weighted features. The weighted features are then residually connected with the temporal features to obtain enhanced semantic features.

[0008] Multi-scale temporal attention module: The multi-scale temporal attention module takes the enhanced semantic features as input, extracts multi-scale contextual information of different time spans through parallel multi-scale convolutional layers, fuses the multi-scale contextual information to generate adaptive attention weights, and performs feature enhancement based on the adaptive attention weights to obtain weighted enhanced features;

[0009] Boundary Awareness Module: The boundary awareness module takes the weighted enhanced features as input, calculates the inter-frame feature differences and performs nonlinear transformation, and connects the transformed difference features with the residuals of the weighted enhanced features to obtain the boundary optimization features;

[0010] Importance prediction module: The importance prediction module takes the boundary optimization feature as input, performs frame-level importance evaluation, and obtains an importance score;

[0011] S3: Perform temporal smoothing and peak detection on the importance scores to determine the keyframe candidate set;

[0012] S4: Based on the keyframe candidate set, and combining nonmaximum suppression and segment merging strategies, generate the final video summary segment.

[0013] Preferably, the semantic enhancement module achieves feature enhancement through the following steps:

[0014] Depthwise separable convolutions are performed on temporal features with a kernel size of 3 and the number of groups equal to the feature dimension. The absolute difference between the convolution output and the temporal features is calculated to obtain the local semantic differences.

[0015] The local semantic differences are input into the semantic weight unit, which consists of a fully connected layer and a sigmoid activation function, and mapped to feature enhancement weights in the range of [0,1].

[0016] The temporal features are multiplied element-wise with the feature enhancement weights to obtain weighted features. Then, the weighted features are residually connected with the temporal features to obtain enhanced semantic features.

[0017] Preferably, the multi-scale temporal attention module includes:

[0018] Parallel multi-scale convolutional layers: Three parallel one-dimensional convolutional layers are used to process and enhance semantic features respectively, with convolutional kernel sizes of 3, 5 and 7, and corresponding padding of 1, 2 and 3 respectively, to extract temporal context information at different time spans;

[0019] Feature fusion layer: used to concatenate the outputs of the parallel multi-scale convolutional layers along the feature dimension, fuse them through a 1×1 convolutional layer and reduce the dimensionality to the enhanced semantic feature dimension to obtain fused features;

[0020] Attention generation layer: consists of a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence, used to generate adaptive attention weights in the range [0,1] based on the fused features;

[0021] Weighted residual unit: used to multiply the adaptive attention weights and fusion features element by element to obtain weighted fusion features, and then perform residual connection between the weighted fusion features and enhanced semantic features to output weighted enhanced features.

[0022] Preferably, the boundary perception module optimizes boundary perception in the following ways:

[0023] A one-dimensional convolution operation is performed on the weighted enhancement features, with a kernel size of 3 and padding of 1. The feature changes between adjacent frames are calculated, and the absolute value of the convolution output is taken to obtain the feature differences between frames.

[0024] The inter-frame feature differences are input into a transform network, which includes two fully connected layers with a ReLU activation function in between to perform a non-linear transformation on the inter-frame feature differences.

[0025] The transformed difference features are joined with the weighted enhancement features by residual connection to obtain the boundary optimization features.

[0026] Preferably, the importance prediction module performs the following operations:

[0027] The boundary optimization features are input into a feature transformation network for processing. The feature transformation network includes a fully connected layer, a ReLU activation function, Dropout regularization, and layer normalization connected in sequence.

[0028] The processed features are input into three prediction heads in parallel to predict importance score, position offset and centrality score respectively. The importance score and centrality score are activated by the sigmoid function and the position offset is activated by the exponential function.

[0029] The predicted importance score is calibrated by multiplying the centrality score element-wise, and the calibrated score is then normalized.

[0030] Preferably, the position offset is a binary vector, including a left offset and a right offset, which represent the distance from the current frame to the left boundary of the corresponding summary segment and the distance to the right boundary, respectively.

[0031] Preferably, the time-series smoothing and peak detection employ the following steps:

[0032] The importance scores are smoothed using a filtering process to suppress timing noise.

[0033] Local peaks in the smoothed importance scores are detected to obtain preliminary keyframe candidates;

[0034] A set of keyframe candidates is selected from the initial keyframe candidates by combining a preset threshold.

[0035] Preferably, the nonmaximum suppression and fragment merging strategy includes:

[0036] Calculate the bounding box of the corresponding candidate summary fragment based on the position offset of the keyframe candidate;

[0037] Non-maximum suppression is performed based on importance scores and bounding box overlap to remove redundant candidates;

[0038] Candidate summary fragments that are temporally adjacent and content-related are merged into consecutive fragments.

[0039] Preferably, the video summarization generation model is trained end-to-end using a multi-task loss function, which is a weighted sum of classification loss, regression loss, and centrality loss.

[0040] Among them, classification loss is used to supervise the prediction of importance score, regression loss is used to supervise the prediction of position offset, and centrality loss is used to supervise the prediction of centrality score.

[0041] Preferably, the regression loss adopts soft IoU loss, which calculates the intersection-union ratio based on the predicted offset and the true offset, and optimizes it through negative logarithm to improve the accuracy of bounding box regression.

[0042] Compared with the prior art, the present invention has at least the following beneficial effects:

[0043] This invention provides a multi-scale temporal modeling video summarization method that integrates semantic enhancement and boundary awareness. Through the cascaded design of a semantic enhancement module, a multi-scale temporal attention module, and a boundary awareness module, it achieves progressive temporal modeling from local semantic enhancement and multi-scale context fusion to boundary feature optimization, which significantly improves the semantic integrity, temporal coherence, and boundary localization accuracy of video summarization.

[0044] This invention proposes a semantic enhancement module that explicitly models and strengthens local semantic changes between adjacent frames by coordinating the processing of temporal features with depthwise separable convolutional layers and semantic weight units. This achieves adaptive focusing on key semantic information, thereby improving the model's accuracy in recognizing key video content.

[0045] This invention proposes a multi-scale temporal attention module, which enhances the modeling ability for both long and short-term events by extracting and fusing contextual information from different time spans in parallel and combining it with an adaptive attention mechanism for feature weighting. This solves the problem that a fixed-scale receptive field is difficult to adapt to the duration of events with varying durations.

[0046] This invention proposes a boundary awareness module that enhances the start and end boundary features of actions or events by explicitly calculating the differences in features between frames and performing nonlinear transformations, thereby achieving more accurate segment localization and solving the problems of blurred boundaries and inaccurate segment start and end points in existing methods.

[0047] Experiments on the SumMe and TVSum benchmark datasets show that, compared with state-of-the-art methods, the proposed method significantly improves the performance of video summarization, ensuring that the generated video summaries have temporal coherence, content integrity, and boundary accuracy. Attached Figure Description

[0048] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0049] Figure 1 is a flowchart of the implementation of a multi-scale temporal modeling video summarization generation method that integrates semantic enhancement and boundary awareness provided by the present invention;

[0050] Figure 2 is an overall framework diagram of a multi-scale temporal modeling video summarization generation method that integrates semantic enhancement and boundary awareness, provided by an embodiment of the present invention.

[0051] Figure 3 is a schematic diagram of the semantic enhancement module provided in an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of the multi-scale temporal attention module provided in an embodiment of the present invention;

[0053] Figure 5 is a schematic diagram of the boundary sensing module provided in an embodiment of the present invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0055] Figure 1 shows a flowchart of the implementation of a multi-scale temporal modeling video summarization generation method integrating semantic enhancement and boundary awareness provided by the present invention. This embodiment provides a multi-scale temporal modeling video summarization generation method integrating semantic enhancement and boundary awareness, including the following steps:

[0056] S1: Perform feature encoding on the input video frame sequence and extract the corresponding temporal feature representation;

[0057] S2: Input the extracted temporal features into the video summarization generation model, and the model outputs an importance score;

[0058] S3: Perform temporal smoothing and peak detection on the importance scores to determine the keyframe candidate set;

[0059] S4: Based on the keyframe candidate set, and combining nonmaximum suppression and segment merging strategies, generate the final video summary segment.

[0060] Example 1:

[0061] This invention provides a preferred embodiment that performs step S1, performing feature encoding on the input video frame sequence to extract the corresponding temporal feature representation. Specifically, it includes the following steps:

[0062] First, the original video is sampled uniformly at fixed intervals to obtain a video frame sequence containing N frames. This sampling strategy ensures coverage of the full duration of the video while controlling the size of the input data. Subsequently, a GoogleNet deep convolutional neural network pre-trained on the ImageNet dataset is used as a feature extractor. For each frame... Forward propagation is performed to extract the output of the global average pooling layer before the last fully connected layer of the network, resulting in a 1024-dimensional frame-level feature vector. This feature vector comprehensively represents the global visual semantic information of the frame. Next, the feature vectors of all frames are arranged sequentially along the time dimension to form an initial temporal feature matrix with a dimension of N×1024, where N is the total number of video frames. This matrix is ​​then normalized using the L2 norm. Finally, a linear projection layer maps the normalized feature dimension from 1024 to a uniform dimension D defined by the model, yielding the temporal feature matrix. , where T represents the temporal length of the video (total number of frames), and D is the dimension of the features of each frame, which serves as the input to the subsequent video summarization generation model.

[0063] Example 2:

[0064] The present invention provides a preferred embodiment for executing S2, in which the extracted temporal features are input into the video summarization generation model, and the model outputs an importance score.

[0065] Figure 2 shows the overall framework diagram of the network model in this embodiment. The network mainly consists of four parts: a semantic enhancement module, a multi-scale temporal attention module, a boundary awareness module, and an importance prediction module. The specific construction steps of each module will be described below:

[0066] S21. Construct the semantic enhancement module, as shown in Figure 3. The semantic enhancement module is used to enhance the local semantic information related to the summary in the temporal features. This module contains a depthwise separable convolutional layer and a semantic weight unit.

[0067] First, a depthwise separable convolution operation is performed on the input temporal features X. This convolution has a kernel size K=3, padding P=1, and the number of groups equals the input feature dimension D, meaning each feature channel is convolved independently. This convolution operation is denoted as... Its output dimension is still Calculate the element-wise absolute difference between the convolutional output and the original input temporal features X to obtain the local semantic differences:

[0068] ,

[0069] in This represents element-wise absolute value operation. This represents depthwise separable convolution operations. .

[0070] Secondly, local semantic differences Input semantic weight units. A semantic weight unit consists of a fully connected layer and a sigmoid activation function. Let the weights of the fully connected layer be... , bias is The output of the semantic weight unit, i.e. the feature enhancement weight G, is calculated as follows:

[0071] ,

[0072] To simplify the description, consider position-by-position (frame) operations, for the features of frame t. and their corresponding semantic differences ,have:

[0073] ,

[0074] in For the Sigmoid function, Its output range is (0,1), therefore The weights of all frames are concatenated to obtain the result. .

[0075] Finally, the feature enhancement weights G are multiplied element-wise with the original temporal features X, and then residually connected with the temporal features X to obtain the enhanced semantic features. :

[0076] ,

[0077] in This indicates element-wise multiplication.

[0078] S22, Construct a multi-scale temporal attention module, as shown in Figure 4. This module aims to capture and fuse contextual information at different time scales and focus on key temporal regions through an attention mechanism. The module consists of parallel multi-scale convolutional layers, feature fusion layers, and attention generation layers. Details are as follows:

[0079] (1) Parallel multi-scale convolutional layers: Three independent one-dimensional convolutional layers (Conv1D) are used in parallel to process and enhance semantic features. The kernel sizes of the three convolutional layers are set to [values ​​to be filled in]. To ensure that the output timing length remains unchanged, the padding is set to the corresponding values. Let these three convolution operations be denoted as... , , For input Three multi-scale feature maps were obtained:

[0080] ,

[0081] in , .

[0082] (2) Feature fusion layer: The three feature maps are concatenated along the feature dimension (second dimension) to obtain the concatenated feature. :

[0083] ,

[0084] Among the symbols This represents a concatenation operation along the feature dimension. Subsequently, a 1×1 convolutional layer (i.e., pointwise convolution) is used to... By performing fusion and dimensionality reduction, the feature is mapped back from 3D to the original D dimension, resulting in the fused feature. :

[0085] ,

[0086] in , This represents a one-dimensional convolution with a kernel size of 1.

[0087] (3) Attention generation layer: The attention generation layer is based on fused features Generate adaptive attention weights. This layer is specifically a small two-layer fully connected network, using the ReLU activation function in the middle and the Sigmoid function at the end to constrain the output. Interval. Suppose the first fully connected layer compresses the feature dimension from D to D / 4, with weights of... , bias is The second fully connected layer restores the dimension from D / 4 to D, with weights of... , bias is Then attention weights The calculation process is as follows:

[0088] ,

[0089] in , This is the Sigmoid function.

[0090] (4) Weighted residual unit: The generated attention weight A is combined with the fused features Element-wise multiplication is performed to achieve differentiated enhancement of different time points and different feature channels, resulting in a weighted fusion feature. This weighted fusion feature is then combined with the enhanced semantic feature. Perform residual connections to obtain the final weighted enhanced features. :

[0091] .

[0092] S23, Construct the boundary awareness module, as shown in Figure 5. The boundary awareness module aims to explicitly model the boundary information of actions or events in the video, and enhance the feature response of the boundary region by calculating the inter-frame feature changes.

[0093] (1) Inter-frame difference calculation: A one-dimensional convolutional layer (Conv1D) is used to calculate the feature gradient or change between adjacent frames. This convolutional layer has a kernel size of K=3, padding P=1, and no bias term (bias=False). The input weighted enhancement features are then processed. The input format of the convolutional layer is transposed to fit the input format of the convolutional layer (from T×D to D×T), and then transposed back after convolution. The absolute value of the convolutional output is taken to capture the intensity of the change without regard to the direction, thus obtaining the inter-frame feature differences. :

[0094] ,

[0095] in This represents the one-dimensional convolution operation.

[0096] (2) Differential Feature Transformation: Transform the inter-frame feature differences A transformation network is used to perform a nonlinear transformation to enhance its representation capability. The transformation network consists of two fully connected layers, with the ReLU activation function used in between. Let the parameters of the first layer be... The second layer parameters are The differences after transformation are then... The calculation is as follows:

[0097] .

[0098] Residual Enhancement Output: The transformed difference feature R is weighted and enhanced with the original input of the module. By performing residual connections, boundary information is injected into the weighted enhancement features, resulting in boundary-optimized features. :

[0099] .

[0100] S24, Construct the importance prediction module. Based on the optimized features, the importance prediction module performs three prediction tasks in parallel: frame-level importance classification, positional offset regression, and centrality estimation.

[0101] (1) Feature Transformation: First, a feature transformation network is used to transform the feature transformation network. The network is processed by including a fully connected layer, a ReLU activation function, Dropout regularization, and LayerNorm. Let the fully connected layer map the features from D dimensions to the hidden dimension H, with weights... , bias is The dropout probability of Dropout is p=0.5. Transformed features The calculation is as follows:

[0102] ,

[0103] LayerNorm normalizes along the feature dimension (the last dimension).

[0104] (2) Multi-task parallel prediction: Three independent fully connected layers are used as prediction heads to process H at the same time.

[0105] Classification head: Used to predict the importance probability score of each frame belonging to the video summary. The header consists of a fully connected layer. It consists of a sigmoid activation function and a weighted sigmoid function. , bias is ,but:

[0106] .

[0107] Regression head: Used to predict the positional offset for each frame. Each row is a two-dimensional vector. , representing the distance from the frame to the left boundary (left offset) and the distance to the right boundary (right offset) of its corresponding summary segment, respectively. This header consists of a fully connected layer. It is composed of an exponential function (Exp) activation, ensuring that the offset is positive. Let its weight be... , bias is ,but:

[0108] .

[0109] Centrality head: Used to predict the centrality score for each frame. This score characterizes how central a frame is within its corresponding summary segment; the closer a frame is to the segment center, the higher the score. Its structure is similar to the classification head, consisting of fully connected layers. It consists of sigmoid activation and sigmoid activation. Let its weights be... , bias is ,but:

[0110] .

[0111] (3) Prediction result calibration: In the inference stage, to further improve the reliability of importance scores, the importance scores predicted by the classification head are calibrated. The centrality score predicted by the centrality head Element-wise multiplication is performed to calibrate the scores. Then, the calibrated scores are normalized so that their maximum value is 1, facilitating subsequent thresholding or sorting. Let the calibrated importance score be... ,but:

[0112] ,

[0113] in For a very small positive number (e.g.) ), used to prevent division by zero.

[0114] The video summarization generation model is trained end-to-end using a multi-task loss function. During training, for a given training sample, the model simultaneously outputs importance score prediction, position offset prediction, and centrality score prediction. Multi-task loss function Classification loss Regression loss and centrality loss Weighted sum:

[0115] ,

[0116] in, and To balance the hyperparameters.

[0117] (1) Classification loss Used to supervise the prediction of importance scores, employing Focal Loss or cross-entropy loss to measure the difference between the predicted importance scores and the true labels.

[0118] (2) Regression loss : Used to supervise the prediction of positional offsets, employing soft IoU loss. For positive sample frames (i.e., frames belonging to the summary), based on the predicted offsets... Compared with the actual offset Calculate the Intersection over Union (IoU) ratio and optimize it using the negative logarithmic form:

[0119] ,

[0120] in The number of positive samples. The value is a very small positive number to prevent numerical instability. Soft IoU loss directly optimizes the overlap between the predicted bounding box and the ground truth bounding box, improving the accuracy of bounding box regression.

[0121] (3) Centrality loss : Used to supervise the prediction of centrality scores, employing binary cross-entropy loss to constrain the predicted centrality scores to be close to the true centrality.

[0122] Through end-to-end training, the model can collaboratively optimize three tasks, ultimately generating accurate importance scores and boundary localization.

[0123] Example 3:

[0124] This invention provides a preferred embodiment that performs step S3, performing temporal smoothing and peak detection on the importance score to determine a candidate set of keyframes. The specific steps are as follows:

[0125] Temporal smoothing: A one-dimensional Gaussian filter is applied to the importance score sequence. A sliding window of size 5 is used to perform a weighted average for each position and its two adjacent frames, effectively suppressing temporal noise caused by fluctuations in model predictions and making the changes in importance scores smoother and more continuous.

[0126] Local peak detection: Detect all local peak points on the smoothed importance score sequence. Specifically, for frame t, if its importance score is greater than the scores of the previous moment t-1 and the next moment t+1, and is greater than a preset absolute value threshold (e.g., 0.3), then the frame is marked as a preliminary keyframe candidate.

[0127] Threshold filtering: Initial candidates are filtered based on a preset threshold. The threshold can be set as a certain percentage (e.g., 1.2 times) of the average importance score of all frames, or determined by adjusting the validation set. Only candidates with scores exceeding the threshold are retained to form the final keyframe candidate set for subsequent segment generation.

[0128] This embodiment uses a three-step process of temporal smoothing, peak detection, and threshold filtering to reliably and robustly extract keyframe candidates representing key video content from the importance scores output by the model, laying the foundation for generating high-quality video summaries.

[0129] Example 4:

[0130] This invention provides a preferred embodiment that executes step S4, generating the final video summary segment based on the keyframe candidate set and combining non-maximum suppression and segment merging strategies. The specific steps are as follows:

[0131] Boundary box calculation: For each keyframe candidate, the corresponding candidate summary fragment bounding box is calculated using the corresponding position offset (including left and right offsets) predicted by the model. Specifically, for the candidate in frame t, its left boundary is t - left offset, and its right boundary is t + right offset + 1, thus obtaining the time interval [left boundary, right boundary] of the candidate fragment.

[0132] Non-maximum suppression: Candidate segments are sorted in descending order of importance score. Each candidate segment is processed sequentially. If its temporal overlap (IoU) with a selected segment exceeds a preset threshold (e.g., 0.5), the candidate is considered redundant and removed; otherwise, it is retained. This process ensures that only the most important candidate is retained within the overlapping region.

[0133] Fragment merging: After non-maximum suppression, the remaining candidate fragments are sorted in temporal order. Temporally adjacent and overlapping fragments or fragments with small intervals are merged to form longer continuous summary fragments. The specific merging condition is: if the interval between two fragments is less than a preset threshold (e.g., 10 frames) or they overlap, they are merged into a single continuous fragment.

[0134] Through the above steps, the present invention can generate compact, continuous and non-redundant final video summary segments from discrete keyframe candidates, effectively improving the coherence and completeness of the summary.

[0135] To verify the effectiveness of the proposed method, comparative experiments were conducted to objectively evaluate its performance. The experiments were performed on two widely used benchmark datasets in the video summarization field—SumMe and TVSum—using standardized F-scores (…). Using , % as the main evaluation index, the performance of the method of this invention is systematically compared with that of existing mainstream advanced methods.

[0136] The SumMe dataset contains 25 videos covering diverse scenarios, each with frame-level importance annotations from at least 15 annotators, suitable for evaluating the consistency between algorithm-generated summaries and human subjective judgment. The TVSum dataset contains 50 carefully selected videos from YouTube, covering 10 categories including news, documentaries, and travel. Each video is independently annotated by 20 annotators, suitable for testing the generalization performance of algorithms across different video types.

[0137] To ensure the statistical reliability of the evaluation results, the experiment adopted a 5-fold cross-validation scheme: each dataset was evenly divided into 5 mutually exclusive subsets, and one subset was used as the test set and the rest as the training set. Five training and testing cycles were performed in sequence. Finally, the average F score of the five experiments was used as the performance index of the model, which effectively reduced the bias that may be introduced by a single data partition.

[0138] Regarding the selection of baseline methods, the experiments covered representative methods in the current video summarization field, including deep learning-based methods such as LMHA, DR-DSN, CAAN, and VJMHT, as well as attention-based methods such as VASNet. Under the same experimental settings, as shown in Table 1, the proposed method significantly outperformed the listed baseline methods in F-scores on both the SumMe and TVSum datasets. This result verifies that the proposed method, through the collaborative modeling of semantic enhancement, multi-scale temporal attention, and boundary awareness, can more accurately locate key content and more rationally organize temporal structures, thereby effectively improving semantic integrity, temporal coherence, and boundary localization accuracy. This demonstrates the advanced nature and robustness of the proposed method in practical applications.

[0139] Table 1 Comparison with advanced methods Comparison results

[0140]

[0141] The superior experimental results achieved by the proposed method on the SumMe and TVSum datasets validate the effectiveness and advancement of the proposed scheme. This performance improvement is mainly attributed to the following innovative designs: First, the semantic enhancement module explicitly extracts and strengthens local semantic differences through the collaborative design of depthwise separable convolutions and semantic weight units, enabling the model to more sensitively capture key semantic features related to the summary content, effectively improving the accuracy of frame-level importance discrimination. Second, the multi-scale temporal attention module captures contextual information across different time spans through parallel multi-scale convolutions and adaptively generates attention weights based on fused features, achieving adaptive focusing on events of varying lengths and enhancing the model's ability to model complex temporal structures in videos. Furthermore, the boundary awareness module explicitly strengthens the start and end boundary features of actions or events by calculating inter-frame feature differences and performing nonlinear transformations, significantly improving the boundary localization accuracy of candidate summary segments, thereby enhancing the temporal integrity and boundary consistency of the generated summary. Compared to methods like VASNet that rely on global attention mechanisms, the multi-scale temporal attention mechanism of this invention can model multi-granular temporal dependencies more meticulously, effectively capturing the key context of long and short-term events. Compared to reinforcement learning-based methods like DR-DSN, this invention avoids complex reward design and sampling processes through an end-to-end training paradigm, improving training efficiency and stability. Compared to methods like CAAN and VJMHT that combine multimodal or hierarchical modeling, the semantic enhancement and boundary awareness modules of this invention achieve more refined semantic difference extraction and boundary feature optimization within a single visual modality. Furthermore, through a multi-task collaborative prediction and calibration mechanism using importance scores, positional offsets, and centrality scores, the joint performance of key content selection and segment localization is further optimized. Experiments show that this invention, through the cascading and synergy of the above modules, ensures processing efficiency while verifying its effectiveness and advancement in video content understanding and summarization tasks.

Claims

1. A method for generating video summaries using multi-scale temporal modeling that integrates semantic enhancement and boundary awareness, characterized in that, Includes the following steps: S1: Perform feature encoding on the input video frame sequence and extract the corresponding temporal feature representation; S2: Input the extracted temporal features into the video summarization generation model, and the model outputs an importance score. The video summarization generation model includes: a semantic enhancement module: the semantic enhancement module takes the temporal features as input, calculates local semantic differences through depthwise separable convolution, and dynamically generates feature enhancement weights based on the local semantic differences using semantic weight units. The temporal features are multiplied element-wise with the feature enhancement weights to obtain weighted features, and then the weighted features are residually concatenated with the temporal features to obtain enhanced semantic features; a multi-scale temporal attention module: the multi-scale temporal attention module takes the enhanced semantic features as input, and extracts different temporal features through parallel multi-scale convolutional layers. Multi-scale contextual information spanning a range is fused to generate adaptive attention weights, and feature enhancement is performed based on these adaptive attention weights to obtain weighted enhanced features; Boundary awareness module: The boundary awareness module takes the weighted enhanced features as input, calculates inter-frame feature differences and performs nonlinear transformation, and connects the transformed difference features with the residuals of the weighted enhanced features to obtain boundary optimized features; Importance prediction module: The importance prediction module takes the boundary optimized features as input and performs frame-level importance evaluation to obtain an importance score; S3: Temporal smoothing and peak detection are performed on the importance score to determine a keyframe candidate set; S4: Based on the keyframe candidate set, and combining nonmaximum suppression and segment merging strategies, generate the final video summary segment.

2. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 1, characterized in that, The semantic enhancement module achieves feature enhancement through the following steps: performing depthwise separable convolution on temporal features with a kernel size of 3 and a group number equal to the feature dimension; calculating the absolute difference between the convolution output and the temporal features to obtain local semantic differences; inputting the local semantic differences into a semantic weight unit composed of a fully connected layer and a sigmoid activation function, mapping it to feature enhancement weights in the range [0,1]; multiplying the temporal features element-wise with the feature enhancement weights to obtain weighted features; and then performing a residual connection between the weighted features and the temporal features to obtain enhanced semantic features.

3. The method according to claim 1, characterized in that, The multi-scale temporal attention module includes: a parallel multi-scale convolutional layer: three parallel one-dimensional convolutional layers process the enhanced semantic features respectively, with convolutional kernel sizes of 3, 5, and 7, and corresponding padding of 1, 2, and 3, to extract temporal context information of different time spans; a feature fusion layer: used to concatenate the outputs of the parallel multi-scale convolutional layers along the feature dimension, fuse them through a 1×1 convolutional layer, and reduce the dimensionality to the enhanced semantic feature dimension to obtain fused features; an attention generation layer: including a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence, used to generate adaptive attention weights in the range [0,1] based on the fused features; and a weighted residual unit: used to multiply the adaptive attention weights element-wise with the fused features to obtain weighted fused features, and then perform residual connection between the weighted fused features and the enhanced semantic features to output weighted enhanced features.

4. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 1, characterized in that, The boundary awareness module optimizes boundary awareness by performing a one-dimensional convolution operation on the weighted enhanced features, with a kernel size of 3 and padding of 1, calculating the feature changes between adjacent frames, and taking the absolute value of the convolution output to obtain the feature differences between frames. The inter-frame feature differences are input into a transform network, which includes two fully connected layers with a ReLU activation function in between to perform a nonlinear transformation on the inter-frame feature differences. The transformed difference features are then connected with the weighted enhancement features via residual connections to obtain boundary optimization features.

5. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 1, characterized in that, The importance prediction module performs the following operations: It inputs the boundary optimization features into a feature transformation network for processing. The feature transformation network includes a fully connected layer, a ReLU activation function, Dropout regularization, and layer normalization connected in sequence. The processed features are then input into three prediction heads in parallel to predict the importance score, position offset, and centrality score, respectively. The importance score and centrality score are activated by the Sigmoid function, and the position offset is activated by the exponential function. The predicted importance score is then calibrated by multiplying the centrality score element-wise, and the calibrated score is normalized.

6. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 5, characterized in that, The position offset is a binary vector, including a left offset and a right offset, which represent the distance from the current frame to the left boundary of the corresponding summary segment and the distance to the right boundary, respectively.

7. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 1, characterized in that, The temporal smoothing and peak detection process employs the following steps: smoothing the importance score to suppress temporal noise; detecting local peaks in the smoothed importance score to obtain preliminary keyframe candidates. A set of keyframe candidates is selected from the initial keyframe candidates by combining a preset threshold.

8. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 1, characterized in that, The non-maximum suppression and fragment merging strategy includes: calculating the bounding boxes of the corresponding candidate summary fragments based on the position offset of the keyframe candidates; performing non-maximum suppression based on the importance score and the overlap of the bounding boxes to remove redundant candidates; and merging temporally adjacent and content-related candidate summary fragments into continuous fragments.

9. The method for generating multi-scale temporal modeling video summaries that integrates semantic enhancement and boundary awareness as described in claim 1, characterized in that, The video summarization generation model is trained end-to-end using a multi-task loss function, which is a weighted sum of classification loss, regression loss, and centrality loss. The classification loss is used to predict the importance score, the regression loss is used to predict the position offset, and the centrality loss is used to predict the centrality score.

Images