Multi-modal video abstract model and method based on graph learning and reconstruction interaction
Through the multimodal video summary model based on graph learning and reconstruction of interaction, the problem of low video storage, browsing and retrieval efficiency is solved, and efficient retrieval of video data and concise video summary generation is realized.
Patent Information
- Application Number
- CN202411596378.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2025-02-28
AI Technical Summary
With the continuous upload of massive video data, the efficiency of video storage, browsing and retrieval has become an urgent problem.
A multimodal video digest model based on graph learning and reconstruction interaction is proposed, including feature extraction module, modal intra-graph learning convolution module, inter-modular reconstruction interaction module and abstract generation module, through which video digest generation module is realized.
This model helps users quickly find the content they need in huge video data, achieving efficient retrieval, browsing and convenient storage of video data. The generated video summary is concise but comprehensively summarizes the video content.
Smart Images
Figure CN119540827A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a multimodal video summarization model and method based on graph learning and reconstruction interaction. Background Art
[0002] With the rise of short video platforms such as Tik Tok, Kuaishou, Bilibili, and YouTube, it has become easier to shoot videos, and more and more users are recording their lives through videos. This change has not only enriched people's lifestyles, but also allowed people to have more fun in the process of sharing their lives. However, with the continuous uploading of massive amounts of video data, the efficiency of video storage, browsing, and retrieval has become an urgent problem to be solved. In order to help users quickly find the content they need in huge amounts of video data and to achieve efficient retrieval, browsing, and convenient storage of video data, video summarization technology has emerged. Video summarization technology aims to generate a concise but comprehensive summary of the video content from key and information-rich clips in the video. Summary of the invention
[0003] In order to solve the above technical problems, the present invention provides a multimodal video summarization model and method based on graph learning and reconstruction interaction, which solves the problems of slow video storage, slow browsing and low retrieval efficiency.
[0004] The technical solution provided by the present invention is as follows:
[0005] A multimodal video summary model based on graph learning and reconstruction interaction includes a feature extraction module, an intra-modal graph learning convolution module, an inter-modal reconstruction interaction module and a summary generation module; the feature extraction module is connected to the intra-modal graph learning convolution module, the intra-modal graph learning convolution module is connected to the inter-modal reconstruction interaction module, and the inter-modal reconstruction interaction module is connected to the summary generation module.
[0006] Preferably, the feature extraction module specifically uses pre-trained GoogLeNet and RoBERTa to extract visual features of video frames and contextual semantic features of sentences.
[0007] Preferably, the intra-modal graph learning convolution module specifically adopts a graph learning convolutional network to dynamically extract the spatial features of the video.
[0008] Preferably, the inter-modal reconstruction interaction module is specifically a reconstruction interaction loss, including feature reconstruction loss and graph reconstruction loss.
[0009] Preferably, the summary generation module specifically obtains the prediction score of each frame through a multi-layer perceptron and a Softmax function, calculates the importance score of the shot using a KTS algorithm, and selects key shots in combination with a knapsack algorithm.
[0010] Preferably, the multimodal video summarization method based on graph learning and reconstruction interaction comprises the following steps:
[0011] Step S1: extract features;
[0012] Step S2: constructing the modality internal graph learning convolution;
[0013] Step S3: construct inter-modal reconstruction interaction;
[0014] Step S4: Generate summary.
[0015] Preferably, step S2 includes the following sub-steps:
[0016] Sub-step S21: define the problem;
[0017] Sub-step S22: constructing a graph learning structure;
[0018] Sub-step S23: Construct a graph learning convolutional network.
[0019] Preferably, step S4 includes the following sub-steps:
[0020] Sub-step S41: Select key shots;
[0021] Sub-step S42: Optimizing the loss function.
[0022] The technical effects of the multimodal video summarization model and method based on graph learning and reconstruction interaction of the present invention are as follows:
[0023] 1. The present invention helps users to quickly find the required content in the huge video data, and realizes the efficient retrieval, browsing and convenient storage of video data, and video summarization technology.
[0024] 2. Video summarization technology aims to generate a concise but comprehensive summary of the video content from key and information-rich clips in the video. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is the F1 value of the model of the present invention on the SumMe and TVSum datasets under different settings.
[0026] Figure 2 The F1 values of the different modalities of the present invention are set on the SumMe and TVSum datasets.
[0027] Figure 3 It is a comparison chart of the real summary and the predicted summary results of the present invention. DETAILED DESCRIPTION
[0028] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0029] The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0030] A multimodal video summary model based on graph learning and reconstruction interaction includes a feature extraction module, an intra-modal graph learning convolution module, an inter-modal reconstruction interaction module and a summary generation module; the feature extraction module is connected to the intra-modal graph learning convolution module, the intra-modal graph learning convolution module is connected to the inter-modal reconstruction interaction module, and the inter-modal reconstruction interaction module is connected to the summary generation module.
[0031] The feature extraction module of this implementation scheme specifically uses pre-trained GoogLeNet and RoBERTa to extract the visual features of the video frame and the contextual semantic features of the sentence.
[0032] The modal intra-graph learning convolution module of this implementation scheme specifically adopts a graph learning convolutional network to dynamically extract the spatial features of the video.
[0033] The inter-modal reconstruction interaction module of this implementation is specifically a reconstruction interaction loss, including feature reconstruction loss and graph reconstruction loss.
[0034] The summary generation module of this implementation scheme specifically obtains the prediction score of each frame through a multi-layer perceptron and a Softmax function, calculates the importance score of the shot using the KTS algorithm, and selects key shots in combination with the knapsack algorithm.
[0035] The multimodal video summarization method based on graph learning and reconstruction interaction of this embodiment includes the following steps:
[0036] Step S1: extract features;
[0037] Step S2: constructing the modality internal graph learning convolution;
[0038] Step S3: construct inter-modal reconstruction interaction;
[0039] Step S4: Generate summary.
[0040] Step S2 of this embodiment includes the following sub-steps:
[0041] Sub-step S21: define the problem;
[0042] Sub-step S22: constructing a graph learning structure;
[0043] Sub-step S23: Construct a graph learning convolutional network.
[0044] Step S4 of this embodiment includes the following sub-steps:
[0045] Sub-step S41: Select key shots;
[0046] Sub-step S42: Optimizing the loss function.
[0047] When this implementation plan is implemented,
[0048] 1 Overall model
[0049] Single-modal data has limitations in extracting video semantic information, and existing multimodal video summarization methods often ignore the specificity of a single modality. Although multimodal video summarization methods integrate information from different modalities, this process often ignores the specificity of each modality. Therefore, we propose a multimodal video summarization model based on graph learning and reconstruction interaction, which achieves modality fusion while retaining the specificity of a single modality. The model consists of four main modules: feature extraction module, intra-modality graph learning convolution module, inter-modality reconstruction interaction module, and summary generation module. In the feature extraction module, pre-trained GoogLeNet and RoBERTa are used to extract the visual features of video frames and the contextual semantic features of sentences. In the intra-modality graph learning convolution module, a graph learning convolutional network is used to dynamically extract the spatial features of the video. The reconstruction interaction loss is designed in the inter-modality reconstruction interaction module, including feature reconstruction loss and graph reconstruction loss. Finally, the prediction score of each frame is obtained by multi-layer perceptron and Softmax function, and the KTS algorithm is used to calculate the importance score of the shot, and the knapsack algorithm is combined to select key shots to generate summaries.
[0050] 1.1 Feature Extraction Module Given an input video V, which is a sequence of n consecutive frames, denoted as V = {f1, f2, ..., f n Given an input text T, consisting of m sentences, represented by T = {T1, T2, ..., T m}. Then, the pre-trained feature extraction models GoogLeNet and RoBERTa are used to extract the features of frames and sentences, respectively, where the feature dimension of the frame is C v ,C v =1024, the dimension of text features is defined as C v , C v =768. Feature table of the extracted video The text feature is represented as
[0051] 1.2 Intra-modal convolution module
[0052] 1.2.1 Problem Definition
[0053] Graph convolutional networks (GCNs) obtain high-level representations of nodes by aggregating the neighborhood information of nodes, thereby effectively extracting the spatial structural features of videos. However, traditional GCNs have a limitation when processing video data: they usually use a fixed graph structure and cannot dynamically update the association information between nodes. Although the graph structure is certain in some application scenarios, there is no natural graph structure in the video data itself. To this end, we need to construct a graph structure, and the constructed graph is sensitive to local noise and outliers. To overcome this limitation, we use a graph learning convolutional network (GLCN), which combines graph learning with graph convolution and can dynamically learn the optimal graph structure. First, define the video graph as G v =(V v , E v ), where V o Represents a set of nodes in the graph. We use the video frame features as nodes in the video graph. E v represents the set of edges, defined as A weight vector a = (a1, a2, ..., a n ) T ∈R n×1 Calculate the edge weights. Define the text graph as G t =(V t , E t ), V t represents the set of text graph nodes. We take text features as nodes of the text graph. E t represents the set of edges, defined as A weight vector b = (b1, b2, ..., b m ) T ∈R m×1 Calculate the edge weight.
[0054] 1.2.2 Graph Learning Structure After the graph is constructed, we obtain the adjacency matrix A of the video graph v and the adjacency matrix A of the text graph t . Given the weight vector a and the weight vector b. Usually, graph learning can be calculated using the following formula:
[0055]
[0056] Where ReLU(·) = max(0,·) represents the activation function, which is used to ensure the non-negativity of edge weights. The softmax(·) function normalizes the weight values to ensure that the graph can meet the following characteristics:
[0057]
[0058] The loss function is defined as:
[0059]
[0060] 1.2.3 Graph Learning Convolutional Networks
[0061] GLCN aims to optimize the graph representation of GCN by combining graph learning and graph convolution operations, thereby enhancing their overall performance. We apply GLCN to the encoding of video graphs and text graphs to fully capture the spatial structure and semantic information in videos. Figure 2 As shown in Figure 1, GLCN consists of graph learning layers and convolutional layers. The goal of the graph learning layer is to generate the optimal adaptive graph representation, while the convolutional layer uses these adaptive graphs to dynamically extract the spatial and structural features of the video. The layer-by-layer propagation formula of the convolutional layer is as follows:
[0062]
[0063] Where k represents the number of convolutional layers, k = 0, 1, ..., K-1. Parameter Matrix is a learnable weight matrix, They correspond to the output representations of the video graph and text graph after the kth convolution layer. Since the output features after graph learning still need to be used for subsequent tasks, no activation function is applied at this stage to preserve the integrity of the information.
[0064] 1.3 Inter-modal Reconstruction Interaction Module
[0065] Previous multimodal interaction methods often only focus on the fusion between modalities, while ignoring the specificity of a single modality. To solve this problem, we designed an inter-modal reconstruction interaction module. When fusing modalities, we use modal reconstruction to retain the specificity of a single modality. The specific analysis is as follows: the smaller the total loss of the model, the better the fusion of the two modalities. At the same time, the smaller the reconstruction loss, the closer the distance between the reconstructed features and the original features, and the reconstructed image and the original image, that is, the more similar, which can maintain the specificity of each modality to a certain extent. Next, we will introduce this module in detail.
[0066] First, modality fusion is performed to update the node representation of the video modality according to the multi-modal adjacent relationship. The specific calculation formula is as follows:
[0067] H v =S tv V v W tv (8)
[0068] H t =S vt V t W vt (9)
[0069] Among them, H v ∈R m×d represents the frame features obtained by graph learning convolution, H t ∈R n×d represents the sentence features extracted by graph learning convolution, S tv ∈R m×n represents the adjacency matrix composed of edges between sentences and frames, S vt ∈R n×m represents the adjacency matrix between frames and sentences, W tv ∈R m×n , W vt ∈R n×m is a learnable weight matrix.
[0070] Our reconstruction includes feature reconstruction and graph reconstruction. The encoded graph is processed by the reverse graph convolution operation to generate the reconstructed features, and the formula is as follows:
[0071] re_H v =S tv H v W′ tv (10)
[0072] re_H t =S vt H t W′ vt (11)
[0073] After obtaining the reconstructed features, they are compared with the original features to calculate the reconstruction loss. The model is optimized by minimizing the reconstruction loss. The graph reconstruction is obtained by multiplying the adjacency matrix formed by the edges between sentences and frames with the adjacency matrix formed by frames and sentences. The specific calculation formula is as follows:
[0074] re_S t =S tv ·S vt (12)
[0075] re_S v =S vt ·S tv (13)
[0076] The reconstruction loss includes feature reconstruction and graph structure reconstruction. The specific formula is as follows:
[0077]
[0078] Re_Loss=α*Feature_Re_Loss+β*Adj_Re_Loss (16)
[0079] 1.4 Key lens selection
[0080] In the training phase, we guide model training by predicting the criticality score of each frame. In the testing phase, we first use the trained model to predict the importance score of each frame, and then apply the KTS algorithm to calculate the score of each shot. Specifically, we take the average of the importance scores of all frames in the shot as the score of the current shot S k , k∈{1, 2, ..., num}, where num represents the total number of shots in the video. Next, the knapsack algorithm is used to select key shots to construct the final summary. To ensure the effectiveness of the summary, we ensure that the total duration of the selected shots does not exceed 15% of the total duration of the video. The selection formula for the video summary is as follows:
[0081]
[0082] Where num represents the total number of video shots, l k is the length of the kth lens, S k represents the importance score of the predicted k-th shot, ω k ∈{0,1} is a binary variable, when ω k When =1, it means that the kth lens is selected as the key lens, otherwise it is not selected.
[0083] 1.5 Optimization During model training, each sample input will generate a predicted value. The difference between the predicted value and the true value is called loss. The smaller the loss value, the better the performance of the model. The loss function is used to quantify the gap between the predicted value and the true value. We use the mean square error (MSE) as the loss function to calculate the error between the predicted value and the true value. Specifically, define y real is the manually annotated label, y pre is the score predicted by the model, and the loss function is calculated as follows:
[0084]
[0085] The final loss of the model is the sum of MSE loss, graph learning loss and reconstruction loss. The specific calculation formula is as follows:
[0086] Final Loss =Loss+GL Loss +Re_Loss (19)
[0087] 2 Experimental setup
[0088] The network based on graph learning and reconstructed interaction is implemented in the Pytorch framework. After a large number of experiments, the optimal learning rate of the model is set to 1e-3, and the decreasing rate of each round of learning rate is set to 1e-7. The experimental results show that under this parameter setting, the model performance is optimal. The Adam optimizer is used for model optimization, and the total number of training rounds is 500. In terms of the setting of feature dimensions, the visual feature dimension of the video is 1024, and the input dimension of the text feature is 768. After processing by the graph learning layer, the feature dimensions of both modalities are compressed to 512. After encoding by the graph convolution layer, the feature dimension is further reduced to 256, and after reconstructing the interaction, the feature dimension is 128. Finally, after processing by the multi-layer perceptron, the model outputs a prediction summary score with a dimension of 1, and normalizes it through the Softmax function to generate a summary.
[0089] 3 Ablation experiments
[0090] Compared with the basic model. The proposed video summarization network based on graph learning and reconstruction interaction designs graph learning loss and reconstruction interaction loss. We set up the following ablation experiments to demonstrate the effectiveness of each part.
[0091] 1) Graph Learning Loss. In the experiment, we set up two networks (GCN) and (GLCN). GCN stands for Graph Convolutional Network, and GLCN stands for Graph Learning Convolutional Network with Graph Learning Loss.
[0092] 2) Reconstruction interaction loss. The network Re(-) without reconstruction interaction loss and the network Re(Loss) with reconstruction interaction loss are set.
[0093] Table 1 F1 values of different settings on SumMe and TVSum datasets
[0094]
[0095] Table 1 shows the F1 values of the models on the SumMe and TVSum datasets under different settings. By comparing the GCN, Re(-) and GLCN, Re(-) methods, it can be concluded that the effect of using graph learning convolutional networks is significantly better than traditional graph convolutional networks, indicating that adaptive graph structures can generate higher quality summaries. Further comparison of the GCN, Re(-) and GCN, Re(Loss) methods shows that the F1 values of the GCN, Re(Loss) method on both datasets are higher than those of the GCN, Re(-), which verifies the effectiveness of the reconstruction interaction loss. In summary, the results in Table 1 show that the GLCN, Re(Loss) method achieves the highest F1 value, proving the effectiveness and superiority of our proposed model.
[0096] Table 2 F1 values of different modal settings on SumMe and TVSum datasets
[0097]
[0098] Table 2 shows the F1 values of different modal settings on the SumMe and TVSum datasets. Among them, the Unimodal method only uses the visual modality, while the Multimodal method combines the visual and textual modalities. From the results in the table, it can be seen that the F1 values of the Multimodal method on these two datasets are significantly higher than those of the Unimodal method. This result verifies the effectiveness of the modal fusion strategy we proposed.
[0099] 4 Qualitative results analysis
[0100] By visually comparing the model-predicted summary with the manually annotated summary, the effectiveness and accuracy of the model in generating high-quality videos is intuitively demonstrated. Taking the first video of the SumMe dataset and the 24th and 43rd videos of the TVSum dataset as examples, Figure 3 The visualization results are shown, where "Gound_Truth" represents the manually annotated summary, "Ours" represents the summary predicted by the model, and the horizontal axis represents the sequence number of the current shot. In the first video of the SumMe dataset, the scene is a plane landing at the airport. Figure 3 (a) shows that the model’s predicted summary is completely consistent with the manually annotated summary, which verifies the model’s effectiveness and accuracy in simple video scenes. The 24th video of TVSum is a collection of a series of difficult actions. Figure 3 (b) shows that the summary of the video consists of multiple short clips, and it is difficult to extract an accurate summary. However, the summary predicted by the model is still completely consistent with the manually annotated summary, demonstrating the robustness of the model in processing complex clips. The 43rd video in the TVSum dataset is an interview with a motorcycle rider, in which the interview clips and the car skills performance clips appear alternately. Figure 3 (c) shows that although the summary of the video consists of many short shot clips, the model's prediction results are mostly consistent with the manual annotations, with only a few clips showing differences. This proves that the proposed model is still effective when facing complex videos.
Claims
1. A multimodal video summarization model based on graph learning and reconstruction interaction, characterized by: It includes a feature extraction module, an intra-modal graph learning convolution module, an inter-modal reconstruction interaction module and a summary generation module; the feature extraction module is connected to the intra-modal graph learning convolution module, the intra-modal graph learning convolution module is connected to the inter-modal reconstruction interaction module, and the inter-modal reconstruction interaction module is connected to the summary generation module.
2. The multimodal video summarization based on graph learning and reconstruction interaction according to claim 1, characterized in that: The feature extraction module specifically uses pre-trained GoogLeNet and RoBERTa to extract visual features of video frames and contextual semantic features of sentences.
3. The multimodal video summarization model based on graph learning and reconstruction interaction according to claim 1, characterized in that: The intra-modal graph learning convolution module specifically adopts a graph learning convolutional network to dynamically extract the spatial features of the video.
4. The multimodal video summarization model based on graph learning and reconstruction interaction according to claim 1, characterized in that: The inter-modality reconstruction interaction module is specifically a reconstruction interaction loss, including feature reconstruction loss and graph reconstruction loss.
5. The multimodal video summarization model based on graph learning and reconstruction interaction according to claim 1, characterized in that: The summary generation module specifically obtains the prediction score of each frame through a multi-layer perceptron and a Softmax function, calculates the importance score of the shot using a KTS algorithm, and selects key shots in combination with a knapsack algorithm.
6. A multimodal video summarization method based on graph learning and reconstruction interaction, characterized in that: The following steps are involved: Step S1: extract features; Step S2: constructing the modality internal graph learning convolution; Step S3: construct inter-modal reconstruction interaction; Step S4: Generate summary.
7. The multimodal video summarization method based on graph learning and reconstruction interaction according to claim 6, characterized in that: The step S2 comprises the following sub-steps: Sub-step S21: define the problem; Sub-step S22: constructing a graph learning structure; Sub-step S23: Construct a graph learning convolutional network.
8. The multimodal video summarization method based on graph learning and reconstruction interaction according to claim 6, characterized in that: The step S4 comprises the following sub-steps: Sub-step S41: Select key shots; Sub-step S42: Optimizing the loss function.