A video summarization method based on dynamic multi-dimensional heterogeneous graph and attention aggregation

Abstract

The application discloses a video summary method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation, and belongs to the technical field of computer vision. The method comprises the following steps: reading an input video frame sequence, and extracting frame-level visual features through a feature extractor; based on the frame-level visual features, frame-level time sequence features are obtained through time sequence correlation modeling, and frame-level spatial features are obtained through spatial attribute analysis; then, the video frame-level visual features, time sequence features and spatial features are input into a video summary generation model, and frame-level importance scores are predicted through the model; subsequently, the obtained key frames are aggregated into shot units, and key shots are selected by adopting a dynamic programming algorithm according to a summary time length constraint; a loss function is constructed to train the model, the model parameters are optimized, the trained model is used to predict new videos, and finally, a video summary is generated. The method solves the problems of insufficient long-range correlation mining and missing cross-dimensional collaboration of existing video summary methods.

Description

Technical Field

[0001] This invention relates to a video summarization method based on dynamic multidimensional heterogeneous graphs and attention aggregation, belonging to the field of computer vision technology. Background Technology

[0002] The widespread adoption of internet technology and smart terminals has driven the exponential growth of video data. Video summarization, as a core task of intelligent video analysis, aims to automatically filter key content and condense the core information of the original video in a concise form, providing a key solution for the efficient retrieval, storage, and browsing of massive amounts of video. It has been widely used in many fields such as security monitoring, social media, and film and television production, and has both important theoretical significance and practical value.

[0003] It is worth noting that video data naturally possesses complex structural relationships across time sequences and long-range dependencies. However, current mainstream video summarization technologies still have significant shortcomings. Most methods are limited to surface-level modeling in a single dimension (such as focusing only on temporal continuity or visual similarity), failing to fully explore the deep correlations between temporal, visual, and spatial dimensions in videos. This results in poor core representativeness and logical coherence of the summaries. To address this, this invention leverages the powerful unstructured association modeling capabilities of graph structures to transform videos into structured representations. By constructing association graphs from three core dimensions—temporal, visual, and spatial—it deeply mines the long-range dependencies and cross-dimensional collaborative relationships between video frames, thereby improving the accuracy and completeness of general video summarization. Summary of the Invention

[0004] To address the problems existing in existing methods, the present invention aims to provide a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation, which aims to solve the core problems of insufficient long-range association mining and lack of cross-dimensional collaboration in existing video summarization methods.

[0005] This invention provides a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation, the method comprising the following:

[0006] S1: Read the input video frame sequence and extract frame-level visual features using a feature extractor;

[0007] S2: Based on frame-level visual features, frame-level temporal features are obtained through temporal correlation modeling, and frame-level spatial features are obtained through spatial attribute analysis;

[0008] S3: Input the video frame-level visual features, temporal features, and spatial features into the video summarization generation model, and predict the frame-level importance score through the model. The video summarization generation model includes:

[0009] Multi-dimensional heterogeneous graph module: The multi-dimensional heterogeneous graph module takes video frame-level visual features, temporal features and spatial features as input, and video frames as nodes to construct temporal logical edges, visual semantic edges and spatial structural edges respectively, and assigns weights to each type of edge to form a multi-dimensional heterogeneous graph; then, based on the weight information of each type of edge, it constructs the corresponding adjacency matrix to make the deep inter-frame association explicit.

[0010] The graph dynamic optimization module takes a multi-dimensional heterogeneous graph as input. First, it dynamically adjusts the structure and weight of the edges according to changes in video content, and updates the relationships between nodes. Second, it removes invalid edges and retains valid edges through a dual threshold mechanism. Finally, it adds relative position information between nodes to improve the modeling accuracy of dependencies between nodes. The final result is an optimized multi-dimensional heterogeneous graph.

[0011] Graph Feature Aggregation Module: The graph feature aggregation module takes the optimized multi-dimensional heterogeneous graph as input, focuses on the core related nodes through the graph attention mechanism, completes the deep fusion of cross-dimensional features and the efficient transmission of node context information, and finally obtains the aggregated node features.

[0012] Keyframe filtering module: The keyframe filtering module takes the aggregated node features as input, calculates the importance score of each video frame, sets a scoring threshold, and filters out keyframes with core semantics.

[0013] S4: The selected keyframes are aggregated into shot units, and key shots are selected using a dynamic programming algorithm based on the summary duration constraint.

[0014] S5: Build a loss function to train the model, optimize the model parameters, use the trained model to predict new videos, and finally generate video summaries.

[0015] Preferably, the visual features refer to the depth features that characterize the image content and semantic information of the video frame; the temporal features refer to the features that characterize the temporal sequence of the video frames and the temporal dependence between frames; and the spatial features refer to the features that characterize the position, size, and spatial distribution structure of the target within the frame.

[0016] Preferably, constructing the multi-dimensional heterogeneous graph module includes:

[0017] Using video frames as nodes, frame-level visual features f i Temporal features t i Spatial features s i Concatenate the nodes according to their dimensions to form node features v. i That is, v i =[f i ;t i ;s iThen, the node features of all frames are integrated into a node feature set V={v 1， v2,...,v T}, where v i Let i represent the feature vector of node i, and T represent the total number of video frames;

[0018] Construct temporal logical edges E based on node features. t Visual semantic edge E f Spatial structure edge E s The graph is then processed and weighted according to the degree of association between nodes, forming a multi-dimensional heterogeneous graph. Next, a temporal adjacency matrix A is generated based on the connection relationships and weight values ​​of each type of edge. t Visual adjacency matrix A f Spatial adjacency matrix A s , where the adjacency matrix element A t [i,j]、A f [i,j]、A s [i,j] represent the weights of the temporal logical edge, visual semantic edge, and spatial structure edge between node i and node j, respectively. If there is no edge, the corresponding element value is 0.

[0019] Preferably, the graph dynamic optimization module is constructed by:

[0020] An adaptive weight adjustment mechanism is adopted. By calculating the L2 norm of the feature difference between adjacent nodes, the degree of abrupt change in the video content corresponding to the node is quantified. If it is greater than the preset content abrupt change threshold, the edge weights around the abrupt node are adjusted. The edge weights are also dynamically fine-tuned in combination with the feature information of adjacent nodes to make the graph structure adapt to the changes in video content.

[0021] Set dual thresholds [T] low ,T high ], filter the weights w in the adjacency matrix <T low or w>T high The invalid associated edges are then removed, and the edge weights of nodes between the threshold values ​​are non-linearly enhanced to ensure that each node retains only valid associated edges, thus maintaining edge sparsity. The specific enhancement formula is as follows:

[0022] ,

[0023] Where: w' is the weight after enhancement; w is the weight before enhancement; k, b are learnable parameters; T low The lower limit threshold parameter for invalid associated edges; T high The upper limit threshold parameter for invalid associated edges;

[0024] Calculate the relative position value d between any two nodes i and j. ij and the relative position value d ijThe mapping is converted into an embedding vector, which is then processed by a multilayer perceptron (MLP) to obtain weight coefficients. These weight coefficients are multiplied by the corresponding elements of each type of adjacency matrix to update the weights of each type of adjacency matrix. The optimized adjacency matrices are then fused using element-wise addition to obtain the optimized adjacency matrix, and finally, the optimized multidimensional heterogeneous graph is obtained.

[0025] Preferably, constructing the graph feature aggregation module includes:

[0026] A linear transformation is performed on the optimized multi-dimensional heterogeneous graph node feature set V. Combined with the optimized adjacency matrix, the attention coefficients between nodes are calculated using a graph attention mechanism. The specific formula is as follows:

[0027] ,

[0028] in: Let be the attention coefficient of node i to node j; V is the learnable attention weight vector; W is the learnable linear transformation matrix; v i v j v k Wv represents the feature vectors of nodes i, j, and k, respectively. i Wv j Wv k v i v j v k The eigenvectors after linear transformation; `.` is the feature concatenation operator; `LeakyReLU(.)` is the non-linear activation function; `exp(.)` is the exponential function. This represents the summation and normalization of the exponential scores of all neighboring nodes k of node i; N i Let i be the set of neighboring nodes of node i;

[0029] Next, weighted aggregation is performed on the associated nodes based on the attention coefficients. The aggregated features of h attention heads are concatenated dimensionally, and the concatenated features are input into a 1×1 convolutional network for feature refinement, finally obtaining the aggregated node features v. agg .

[0030] Preferably, constructing the keyframes into a shot unit includes:

[0031] The keyframe sequence is subjected to boundary detection by a temporal segmentation algorithm. The similarity of visual features between frames is used as the condition for shot boundary determination. Continuously related keyframes are aggregated into shot units. Then, the keyframe features in each shot unit are averaged to obtain shot-level features. These features are input into a fully connected network to output shot importance scores and optimize the temporal coherence between shots, providing a basis for subsequent selection.

[0032] Preferably, constructing the loss function includes:

[0033] The loss function is a weighted hybrid loss function that combines regression loss and contrastive loss. Its core purpose is to balance the prediction accuracy of frame-level importance scores with the feature discrimination between keyframes and non-keyframes. The specific formula is as follows:

[0034] ,

[0035] Where: λ1 and λ2 are weighting parameters, and λ1 + λ2 = 1; L reg For regression loss; L con To compare the losses.

[0036] Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] This invention provides a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation. By extracting temporal, visual, and spatial features from the video, a multi-dimensional heterogeneous graph is constructed. The structure of the multi-dimensional heterogeneous graph is then dynamically weighted and cross-dimensional feature aggregation is performed. Frame importance scores are calculated, and key frames and key shots are selected, ultimately generating a video summary. This method effectively solves the problems of insufficient long-range association mining and lack of cross-dimensional collaboration in existing video summarization methods.

[0038] This invention designs a multi-dimensional heterogeneous graph module, which uses video frames as nodes to construct temporal logical edges, visual semantic edges, and spatial structural edges respectively, and assigns weights to each type of edge to form a multi-dimensional heterogeneous graph. Then, based on the weight information of each type of edge, a corresponding adjacency matrix is ​​constructed to make the deep inter-frame association explicit.

[0039] This invention designs a graph dynamic optimization module, which enhances the effectiveness and accuracy of core associations by dynamically adjusting edge structure and weights, filtering invalid associations through a dual threshold mechanism, and adding relative position information between nodes.

[0040] This invention designs a graph feature aggregation module, which completes cross-dimensional feature fusion and contextual information transmission based on graph attention mechanism, enhances the model's ability to mine long-range and deep associations, and improves the completeness and logic of the summary.

[0041] The invention was experimentally validated on two benchmark datasets, SumMe and TVSum. The experimental results show that the proposed method has a significant improvement in video summarization performance compared with the state-of-the-art methods. Attached Figure Description

[0042] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0043] Figure 1 This invention provides a flowchart for implementing a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation;

[0044] Figure 2 This invention provides an overall framework diagram for a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation.

[0045] Figure 3 This is a structural diagram of the multi-dimensional heterogeneous graph module provided in an embodiment of the present invention;

[0046] Figure 4 This is a structural diagram of the graph dynamic optimization module provided in an embodiment of the present invention;

[0047] Figure 5 This is a structural diagram of the graph feature aggregation module provided in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0049] like Figure 1 The diagram shown is a flowchart illustrating the implementation of a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation, provided by this invention. This embodiment provides a video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation, comprising the following steps:

[0050] S1: Read the input video frame sequence and extract frame-level visual features using a feature extractor.

[0051] S2: Based on frame-level visual features, frame-level temporal features are obtained through temporal correlation modeling, and frame-level spatial features are obtained through spatial attribute analysis.

[0052] S3: Input the video frame-level visual features, temporal features, and spatial features into the video summarization generation model, and predict the frame-level importance score through the model.

[0053] S4: The selected keyframes are aggregated into shot units, and key shots are selected using a dynamic programming algorithm based on the summary duration constraint.

[0054] S5: Build a loss function to train the model, optimize the model parameters, use the trained model to predict new videos, and finally generate video summaries.

[0055] Example 1:

[0056] This invention provides a preferred embodiment that executes step S1, reading the input video frame sequence and extracting frame-level visual features using a feature extractor. The specific steps are as follows:

[0057] Given an original video, presample it at a sampling rate of 2 FPS to obtain a video frame sequence V. Then, use the GoogLeNet network pre-trained on ImageNet as a feature extractor to extract visual features for each frame. Specifically, use the 1024-dimensional feature vector output from the pool5 layer of the GoogLeNet network as the feature representation of each frame, denoted as F={f1,f2,...,f...}. T}; where f i Let be the visual feature vector of the i-th frame, used to characterize the visual content within the frame.

[0058] Example 2:

[0059] This invention provides a preferred embodiment for executing S2, which, based on frame-level visual features, obtains frame-level temporal features through temporal correlation modeling, and obtains frame-level spatial features through spatial attribute analysis. The specific steps are as follows:

[0060] Extracting temporal features: First, for the visual features f of the i-th frame... i The cosine similarity between the forward similarity sequence and the visual features of the preceding k frames and the following k frames is calculated, where k∈[3,10]. Then, the forward similarity sequence and the backward similarity sequence are concatenated to obtain the bidirectional similarity sequence S. i The formula for calculating cosine similarity is:

[0061] ,

[0062] Where: sim(.) is the cosine similarity between the i-th frame and the j-th frame, f i f j These are the visual feature vectors of the i-th and j-th frames, respectively; For visual feature vector f i and visual feature vector f j The L2 norm of , where j is the k frames before or after the i-th frame, i.e. or i <j≤i+k。

[0063] Secondly, sinusoidal position coding is used to generate the temporal embedding vector pos. i The absolute time position of the captured frame is then stitched together with the S-frame. i With pos i The feature vector is obtained and input into a 3-layer fully connected network to output frame-level temporal features t. iThe temporal features of all frames are then integrated into a temporal feature sequence, denoted as T={t1,t2,...,t...}. T}, where t i Let T represent the temporal feature vector of the i-th frame, and T represent the total number of video frames.

[0064] Spatial feature extraction: First, a pre-trained Faster R-CNN model is used to detect objects in video frames, filtering out target objects with a confidence score greater than δ, and obtaining their bounding box coordinates (x, y, y). min ,y min ,x max ,y max ), where δ is the target detection confidence threshold parameter, δ∈[0.5,0.9], x min y min These represent the x and y coordinates of the top-left corner of the target bounding box, respectively. max y max These represent the x-coordinate and y-coordinate of the lower right corner of the target bounding box, respectively.

[0065] Secondly, these bounding box coordinates are normalized to the [0,1] interval, and the relative area of ​​each target is calculated. k and center coordinates (x) c,k ,y c,k This yields the normalized spatial attributes of the target, which are then compared with the visual features f. i The global average pooling features are concatenated and then input into a two-layer convolutional network to output frame-level spatial features s. i The spatial features of all frames are then integrated into a spatial feature sequence, denoted as S={s1,s2,...,s...} T}, where s i Let represent the spatial feature vector of the i-th frame, and T represent the total number of video frames.

[0066] Example 3:

[0067] The present invention provides a preferred embodiment for executing S3, which inputs video frame-level visual features, temporal features and spatial features into a video summarization generation model, and predicts frame-level importance scores through the model.

[0068] like Figure 2 The diagram shown is the overall framework of the video summarization generation model in this embodiment. This model comprises four parts: a multi-dimensional heterogeneous graph module, a graph dynamic optimization module, a graph feature aggregation module, and a keyframe filtering module. The specific construction steps of each module will be described below:

[0069] S31 constructs a multi-dimensional heterogeneous graph module, the structure of which is as follows: Figure 3 As shown.

[0070] First, using video frames as nodes, the visual features f of each frame are... i Temporal features t i Spatial features s i Concatenate the nodes according to their dimensions to form node features v. i =[f i ;t i ;s i Then, the node features of all frames are integrated into a node feature set V={v 1， v2,...,v T}, where v i Let represent the feature vector of the i-th node, and T represent the total number of video frames.

[0071] Secondly, three types of edge sets are constructed based on the node set, namely, temporal logical edges E. t Visual semantic edge E f and spatial structure edge E s Among them, the sequential logic edge E t For any two nodes i and j, the following condition is satisfied: When constructing bidirectional related edges, where This refers to the time-series edge interval parameter. For node pairs that satisfy this condition, the edge weights are further calculated using the following formula:

[0072] ,

[0073] Where: symbol t represents the weight of the temporal logical edge between node i and node j. i ,t j Let represent the temporal feature vectors of node i and node j. Represents the time series eigenvector t i With time series feature vector t j The L2 norm of , and sigmoid(). is the activation function.

[0074] Visual semantic edges include visual similarity edges and visual difference edges, where visual semantic edge E f In sim(f) i ,f j When α ≥ α, visual similarity edges are constructed, where α is the visual similarity threshold parameter, α ∈ [0.5, 0.85]. For node pairs satisfying this condition, the edge weight is calculated using the following formula: ,in Let sim(.) represent the weight of the visually similar edge between node i and node j, and let sim(.) represent the visual similarity between node i and node j; Simultaneously, in sim(f i ,f jWhen )≤β, construct visual difference edges, where β is the visual difference threshold parameter, β∈[0.1,0.4]; the weight calculation formula is: ,in sim(.) represents the weight of the visual difference edge between node i and node j, and sim(.) represents the visual similarity between node i and node j.

[0075] Spatial structure edge E s In sim(s) i ,s j The edge is constructed when ≥ γ, where γ is the spatial similarity threshold parameter, γ∈[0.5,0.8]; for node pairs satisfying this condition, the edge weight is calculated, and the edge weight calculation formula is: ,in Let be the weight of the spatial structure edge between node i and node j, and sim(.) represent the spatial similarity between node i and node j.

[0076] Finally, based on the three types of edge sets mentioned above, the temporal adjacency matrix A is generated. t Visual adjacency matrix A f Spatial adjacency matrix A s The adjacency matrix element A t [i,j]、A f [i,j]、A s [i,j] represent the weights of the temporal logical edge, visual semantic edge, and spatial structure edge between node i and node j, respectively. If there is no edge, the corresponding element value is 0.

[0077] The S32 construction graph dynamic optimization module has the following structure diagram: Figure 4 As shown.

[0078] First, an adaptive weight adjustment mechanism is used to calculate the degree of abrupt change in features between adjacent nodes. The calculation formula is as follows: ,in This indicates the degree of mutation between node i and node i-1. Let L2 norm represent the difference between the eigenvectors of node i and node i-1. Then adjust the weight of the relevant edge, where ε is the node feature mutation threshold parameter, ε∈[0.3,1.2].

[0079] Secondly, set a double threshold [T] low ,T high ], filter the weights w in the adjacency matrix <T low or w>T high Invalid associated edges are identified, and then the edge weights of nodes between the threshold values ​​are non-linearly enhanced to ensure that each node retains only valid associated edges, thus maintaining edge sparsity. The specific enhancement formula is as follows:

[0080] ,

[0081] Where: w' is the weight after enhancement; w is the weight before enhancement; k and b are learnable parameters, k∈[1.0,3.0], b∈[-0.2,0.2]; T low T is the lower limit threshold parameter for invalid associated edges. low ∈[0.05,0.25]; T high For the upper limit threshold parameter of invalid associated edges, T high ∈[0.75,0.95].

[0082] Finally, calculate the relative position value d between any two frames i and j. ij and the relative position value d ij The mapping is converted into an embedding vector, which is then processed by a multilayer perceptron (MLP) to obtain weight coefficients. These weight coefficients are multiplied by the corresponding elements of each type of adjacency matrix to update the weights of each type of adjacency matrix. The optimized adjacency matrices are then fused using element-wise addition to obtain the optimized adjacency matrix, and finally, the optimized multidimensional heterogeneous graph is obtained.

[0083] The S33 construction graph feature aggregation module has the following structure diagram: Figure 5 As shown.

[0084] First, using the optimized multi-dimensional heterogeneous graph node feature set V and the optimized adjacency matrix as input, feature fusion is achieved through a multi-head graph attention mechanism. First, the number of attention heads h is set, and the weight matrix W is initialized. h Bias b h and attention vector a h , where h represents the h-th attention head, h∈[4,16].

[0085] Secondly, a feature aggregation operation is performed on each attention head. First, a linear transformation is applied to the node feature set V. The specific calculation formula is: V h =V×W h +b h V h W represents the node feature matrix after the linear transformation of the h-th attention head, and V represents the node feature set matrix; h b represents the learnable weight matrix for the h-th attention head; h Let represent the learnable bias vector of the h-th attention head.

[0086] Next, calculate the attention coefficients for any two nodes i and j. The specific calculation formula is as follows:

[0087] ,

[0088] in: Let be the attention coefficient of node i to its neighbor node j; V is the learnable attention weight vector; W is the learnable linear transformation matrix; v i v j v k These are the feature vectors of nodes i, j, and k, respectively; Wv i Wv j Wv k v i v j v k The eigenvectors after linear transformation; `.` is the feature concatenation operator; `LeakyReLU(.)` is the non-linear activation function; `exp(.)` is the exponential function. This represents the summation and normalization of the exponential scores of all neighboring nodes k of node i; N i Let i be the set of neighboring nodes of node i.

[0089] Finally, the aggregated features of h attention heads are concatenated to obtain a concatenated feature containing multi-view correlation information. Finally, the splicing features will be combined. Inputting a 1×1 convolutional network refines features, filters redundant information, strengthens core features, and ultimately yields aggregated node features V that integrate temporal, visual, and spatial multi-dimensional information. agg The specific calculation formula is as follows:

[0090] ,

[0091] Where: V agg The features of the aggregated nodes, For splicing features, w conv Let b be the weight matrix of the convolution. conv Here is the bias vector, and ReLU(.) is the activation function.

[0092] S34 constructs the keyframe filtering module.

[0093] Aggregate node feature V agg Input a regression model, calculate the importance score for each frame, and output the frame-level importance score s. i ∈[0,1], the specific calculation formula is as follows:

[0094] ,

[0095] Where: s i Let i be the importance score of node i. For V agg The aggregated feature vector of node i, W 2， b2 represents the weight matrix and bias of the fully connected network, respectively, and ReLU(.) and sigmoid(.) are the activation functions.

[0096] Finally, a screening threshold θ is set based on the statistical characteristics of the scores, where θ∈[0.3,0.7], and frames with importance scores higher than the threshold θ are selected as keyframes.

[0097] Example 4:

[0098] This invention provides a preferred embodiment that executes step S4, aggregating the selected keyframes into shot units, and selecting key shots using a dynamic programming algorithm based on the summary duration constraint. The specific steps are as follows:

[0099] First, a temporal segmentation algorithm is used to determine the boundaries of the keyframe sequence. The cosine similarity of visual features between frames is used as the basis for shot switching. The cosine similarity threshold is set to 0.65. If the visual similarity between two adjacent keyframes is lower than this threshold, it is determined to be a shot boundary. Continuous keyframes that do not cross the shot boundary are formed into an independent shot unit.

[0100] Secondly, mean aggregation is performed on all keyframes within each shot unit to obtain shot-level features. These shot-level features are then input into a two-layer fully connected network and a sigmoid activation function to output shot importance scores.

[0101] Finally, under the constraint that the summary duration is 10%-15% of the original video duration, a dynamic programming algorithm is used to select the optimal combination of key shots.

[0102] Example 5:

[0103] This invention provides a preferred embodiment that executes step S5, constructs a loss function to train a model, optimizes the model parameters, uses the trained model to predict new videos, and finally generates video summaries. The specific steps are as follows:

[0104] First, a hybrid loss function combining regression loss and contrastive loss is adopted to balance the accuracy of frame importance prediction with the feature discrimination between key frames and non-key frames. The regression loss uses smoothed L1 loss to calculate the error between the frame importance score predicted by the model and the manually labeled true score, thereby improving prediction accuracy. The contrastive loss is used to widen the feature distance between key frames and non-key frames, enhancing the discrimination between the two types of frames and ensuring that the key frames selected by the model are more representative.

[0105] Secondly, model training and parameter optimization are performed. With the goal of minimizing the total loss, the gradient descent optimization algorithm is used to iteratively update all learnable parameters in the model. When the loss value tends to stabilize and the convergence condition is met, training is stopped, and the optimized model is obtained.

[0106] Finally, after the model is trained, it is used to predict new videos and ultimately generate video summaries.

[0107] To verify the effectiveness of the above embodiments, the present invention was applied in practice. The F-score (%) was calculated and compared with existing state-of-the-art methods. Experiments were conducted on two benchmark datasets, SumMe and TVSum, to evaluate the effectiveness of the present invention in modeling long-range dependencies and cross-dimensional collaborative relationships in videos. SumMe is a popular video summarization dataset containing 25 videos from everyday life, sports, and other scenarios, with multiple user-annotated importance scores for each frame. TVSum consists of 50 videos covering news, documentaries, sports, and other categories, providing key segments annotated by multiple users. Both datasets are suitable for testing the model's generalization ability across different video types.

[0108] This invention employs standard 5-fold cross-validation. All videos are uniformly divided into five subsets of equal size. Four subsets are selected as the training set and one subset as the test set in each round of cross-validation. After five rounds of cross-validation, the mean of the five experimental results is used as the final evaluation metric. This approach effectively mitigates the fluctuations in the results of random data partitioning, reduces its influence on experimental conclusions, and ensures the fairness and reliability of the comparison.

[0109] This experiment selected several mainstream video summarization methods for comparison, including DR-DAN, VASNet, VJMHT, MAMI, and MGAF-SUM. Under standard evaluation settings, the experimental results are shown in Table 1. The method of this invention achieved the best summarization performance on both datasets.

[0110] Table 1. Comparison with advanced methods Comparison results

[0111]

[0112] Experimental results show that this method significantly outperforms existing state-of-the-art methods on both the SumMe and TVSum datasets. This is mainly due to the following innovative designs: constructing a multi-dimensional heterogeneous graph to achieve collaborative modeling of multi-source information such as visual, temporal, and spatial information, comprehensively characterizing inter-frame relationships; proposing a graph dynamic optimization module to remove invalid association edges and retain valid associations; and proposing a graph feature aggregation module to enhance the modeling ability of long-range dependencies and deep semantic relationships in videos through a graph attention mechanism. This method effectively solves the problems of insufficient long-range association mining and lack of cross-dimensional collaboration in existing video summarization methods.

Claims

1. A video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation, characterized in that, Includes the following steps: S1: Read the input video frame sequence and extract frame-level visual features using a feature extractor; S2: Based on frame-level visual features, frame-level temporal features are obtained through temporal correlation modeling, and frame-level spatial features are obtained through spatial attribute analysis; S3: Input the video frame-level visual features, temporal features, and spatial features into the video summarization generation model, and predict the frame-level importance score through the model. The video summarization generation model includes: Multi-dimensional heterogeneous graph module: The multi-dimensional heterogeneous graph module takes video frame-level visual features, temporal features and spatial features as input, and video frames as nodes to construct temporal logical edges, visual semantic edges and spatial structural edges respectively, and assigns weights to each type of edge to form a multi-dimensional heterogeneous graph; then, based on the weight information of each type of edge, it constructs the corresponding adjacency matrix to make the deep inter-frame association explicit. The graph dynamic optimization module takes a multi-dimensional heterogeneous graph as input. First, it dynamically adjusts the structure and weight of the edges according to changes in video content, and updates the relationships between nodes. Second, it removes invalid edges and retains valid edges through a dual threshold mechanism. Finally, it adds relative position information between nodes to improve the modeling accuracy of dependencies between nodes. The final result is an optimized multi-dimensional heterogeneous graph. Graph Feature Aggregation Module: The graph feature aggregation module takes the optimized multi-dimensional heterogeneous graph as input, focuses on the core related nodes through the graph attention mechanism, completes the deep fusion of cross-dimensional features and the efficient transmission of node context information, and finally obtains the aggregated node features. Keyframe filtering module: The keyframe filtering module takes the aggregated node features as input, calculates the importance score of each video frame, sets a scoring threshold, and filters out keyframes with core semantics. S4: The selected keyframes are aggregated into shot units, and key shots are selected using a dynamic programming algorithm based on the summary duration constraint. S5: Build a loss function to train the model, optimize the model parameters, use the trained model to predict new videos, and finally generate video summaries.

2. The video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation as described in claim 1, characterized in that, The visual features refer to the depth features that characterize the image content and semantic information of video frames; the temporal features refer to the features that characterize the temporal sequence of video frames and the temporal dependence between frames; and the spatial features refer to the features that characterize the position, size, and spatial distribution structure of targets within a frame.

3. The video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation as described in claim 1, characterized in that, The multi-dimensional heterogeneous graph module includes: Using video frames as nodes, frame-level visual features f i Temporal features t i Spatial features s i Concatenate the nodes according to their dimensions to form node features v. i ,Right now Then, the node features of all frames are integrated into a node feature set. , where v i Let i represent the feature vector of node i, and T represent the total number of video frames; Temporal logical edges are constructed based on node features. Visual semantic edge Spatial structure edge The graph is then processed and weighted according to the degree of association between nodes, forming a multi-dimensional heterogeneous graph. Next, a temporal adjacency matrix is ​​generated based on the connection relationships and weight values ​​of each type of edge. Visual adjacency matrix Spatial adjacency matrix , where the adjacency matrix elements , , These represent the weights of the temporal logical edge, visual semantic edge, and spatial structural edge between node i and node j, respectively. If there is no edge, the corresponding element value is 0.

4. The video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation as described in claim 1, characterized in that, The graph dynamic optimization module includes: An adaptive weight adjustment mechanism is adopted. By calculating the L2 norm of the feature difference between adjacent nodes, the degree of video content mutation corresponding to the node is quantified. If it is greater than the preset content mutation threshold, the edge weights around the mutated node are adjusted. The edge weights are dynamically fine-tuned in combination with the feature information of adjacent nodes so that the graph structure can adapt to the changes in video content. Set dual thresholds Filter the weights in the adjacency matrix or The invalid associated edges are then removed, and the edge weights of nodes between the threshold values ​​are non-linearly enhanced to ensure that each node retains only valid associated edges, thus maintaining edge sparsity. The specific enhancement formula is as follows: ， in: The weights are: w = weights before enhancement; k and b are learnable parameters. The lower limit threshold parameter for invalid associated edges; The upper limit threshold parameter for invalid associated edges; Calculate the relative position values ​​of any two nodes i and j. and relative position value The mapping is converted into an embedding vector, which is then processed by a multilayer perceptron (MLP) to obtain weight coefficients. These weight coefficients are multiplied by the corresponding elements of each type of adjacency matrix to update the weights of each type of adjacency matrix. The optimized adjacency matrices are then fused using element-wise addition to obtain the optimized adjacency matrix, and finally, the optimized multidimensional heterogeneous graph is obtained.

5. The video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation as described in claim 1, characterized in that, The graph feature aggregation module includes: For the optimized multi-dimensional heterogeneous graph node feature set A linear transformation is performed, and combined with the optimized adjacency matrix, the attention coefficients between nodes are calculated using a graph attention mechanism. The specific formula is as follows: ， in: Let be the attention coefficient of node i to its neighbor node j; V is the learnable attention weight vector; W is the learnable linear transformation matrix; v i v j v k These are the feature vectors of nodes i, j, and k, respectively. v i v j v k The eigenvectors after linear transformation; For feature concatenation operators; It is a non-linear activation function; It is an exponential function; This represents the summation and normalization of the exponential scores of all neighboring nodes k of node i; N i Let i be the set of neighboring nodes of node i; Next, weighted aggregation is performed on the associated nodes based on the attention coefficients. The aggregated features of h attention heads are concatenated along their dimensions, and the concatenated features are input into a 1×1 convolutional network for feature refinement, ultimately yielding the aggregated node features. .

6. The video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation as described in claim 1, characterized in that, The keyframes are aggregated into a shot unit, including: The keyframe sequence is subjected to boundary detection by a temporal segmentation algorithm. The similarity of visual features between frames is used as the condition for shot boundary determination. Continuously related keyframes are aggregated into shot units. Then, the keyframe features in each shot unit are averaged to obtain shot-level features. These features are input into a fully connected network to output shot importance scores. At the same time, the temporal coherence between shots is optimized to provide a basis for subsequent selection.

7. The video summarization method based on dynamic multi-dimensional heterogeneous graphs and attention aggregation as described in claim 1, characterized in that, The loss function includes: The loss function is a weighted hybrid loss function that combines regression loss and contrastive loss. Its core purpose is to balance the prediction accuracy of frame-level importance scores with the feature discrimination between keyframes and non-keyframes. The specific formula is as follows: , in: For loss function, For weighted parameters, and ; For regression loss; To compare the losses.

Images