A Multimodal Teaching Video Summarization Method Based on Heterogeneous Graphs

Abstract

This invention discloses a multimodal teaching video summarization method based on heterogeneous graphs, comprising: acquiring several video samples to form a training set; establishing a multimodal summarization generation model and training it using the training set, wherein the model performs the following operations: inputting video frame sequences and sentence sequences into a visual feature extraction model and a language model respectively, obtaining a set of visual feature vectors and a set of text feature vectors to form a multimodal feature representation; initializing an adjacency matrix; performing a Hadamard product on the intra-modal constraint matrix, inter-modal constraint matrix, and adjacency matrix to obtain an optimized heterogeneous graph; executing a two-stage fusion strategy; using the multimodal unified representation output by the trained multimodal summarization generation model to filter key video frame node sets and key sentence node sets, and retaining the corresponding connection relationships within the optimized heterogeneous graph as sub-adjacency matrices to obtain a multimodal summarization graph. This method can generate semantically consistent and content-rich teaching video summaries with strong generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of video summarization technology, specifically relating to a method for generating multimodal teaching video summaries based on heterogeneous graphs. Background Technology

[0002] With the development of digital educational resources, instructional videos, as a primary form of knowledge dissemination, are widely used in online education and smart classrooms. However, instructional videos generally suffer from problems such as long duration, uneven information density, and content redundancy, making it difficult for learners to efficiently and quickly acquire core knowledge points within a limited time. Therefore, multimodal video summarization methods have gradually become a research hotspot. Their basic framework typically includes an encoder for extracting features from each modality, a multimodal fusion module, and a decoder for generating summaries. However, since the encoder often models features from different modalities separately, the fusion stage often relies on temporal alignment and late fusion strategies. While this approach ensures alignment in the temporal dimension, it struggles to capture deeper semantic relationships between modalities, resulting in multimodal summaries that, while temporally logical, often lack semantic consistency and expressive diversity. Therefore, how to simultaneously consider cross-modal semantic alignment and intramodal content diversity during summarization remains a key challenge that urgently needs to be overcome. Summary of the Invention

[0003] The purpose of this invention is to address the above-mentioned problems by proposing a multimodal teaching video summarization method based on heterogeneous graphs, which can generate semantically consistent and content-rich teaching video summaries with strong generalization ability.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] This invention proposes a method for generating multimodal teaching video summaries based on heterogeneous graphs, comprising the following steps:

[0006] S1. Obtain several video samples to form a training set. The video samples include teaching videos, as well as corresponding sampled video frame sequences and sentence sequences.

[0007] S2. Establish a multimodal summarization generation model and train it using the training set. The multimodal summarization generation model performs the following operations:

[0008] S21. Input the video frame sequence into the visual feature extraction model to obtain a set of visual feature vectors, and input the sentence sequence into the language model to obtain a set of text feature vectors. Concatenate the set of visual feature vectors and the set of text feature vectors to form a multimodal feature representation.

[0009] S22. Treat each video frame in the video frame sequence and each sentence in the sentence sequence as a node, and initialize the adjacency matrix so that there is a connection relationship between any nodes.

[0010] S23. Calculate the intramodal connectivity and intermodal time alignment between nodes respectively. Construct the intramodal constraint matrix and intermodal constraint matrix accordingly, and then perform a Hadamard product of these three matrices with the adjacency matrix to form the adjacency representation. Obtain optimized heterogeneous graph , The set of all nodes;

[0011] S24. Execute a two-stage fusion strategy, that is, after obtaining a shallow fusion representation of multimodal feature representation through a global attention mechanism, input the shallow fusion representation and the optimized heterogeneous graph into a two-layer graph attention network to obtain a unified multimodal representation.

[0012] S3. Use the multimodal unified representation output by the trained multimodal summarization generation model to filter out the set of key video frame nodes. and key sentence node set The corresponding connectivity relationships within the optimized heterogeneous graph are retained as sub-adjacency matrices. Obtain multimodal summary graph This is the result of generating a summary of the teaching video.

[0013] Preferably, the visual feature extraction model is a pre-trained ResNet model or a pre-trained ViT model, and the language model is a pre-trained BERT model or a pre-trained RoBERTa model.

[0014] Preferably, the intramodal connectivity relationships and intermodal time alignment relationships between nodes are calculated separately, and the corresponding intramodal constraint matrices and intermodal constraint matrices are constructed as follows:

[0015] S231. Traverse the entire set of nodes to extract node pairs;

[0016] S232. When the nodes in a node pair are of the same modality, calculate the cosine similarity of the corresponding nodes to determine the intramodal connectivity. When the nodes in a node pair are of different modalities, calculate the intermodal temporal alignment of the corresponding nodes.

[0017] S233. Obtain the modal constraint matrix. and intermodal constraint matrix Among them, the intramodal constraint matrix top left corner area and bottom right corner The elements of a region are represented by the modal inner connectivity of corresponding node pairs, bottom left corner. area and top right corner All elements of the region are set to 0, and the intermodal constraint matrix is... bottom left corner area and top right corner The elements of a region are represented by the intermodal temporal alignment of the corresponding node pairs, with the top left corner as an example. area and bottom right corner All elements in the region are set to 0, where, The total number of video frames in the video frame sequence. This represents the total number of sentences in the sentence sequence.

[0018] Preferably, when the nodes in a node pair are of the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intramodal connectivity. When the nodes in a node pair are of different modalities, the intermodal temporal alignment of the corresponding nodes is calculated, as follows:

[0019] 1) When all nodes in a node pair are sentence nodes, perform the following operations:

[0020] Obtain the text feature vector pairs corresponding to the node pairs;

[0021] Calculate the cosine similarity between text feature vectors and Chinese text feature vectors;

[0022] The sentence connection relationship between node pairs is determined by constraining the modal intra-connection relationship according to the first preset threshold. That is, when the cosine similarity between text feature vectors is greater than the first preset threshold, the sentence connection relationship between node pairs is represented as 1, otherwise it is represented as 0.

[0023] 2) When all nodes in a node pair are video frame nodes, perform the following operations:

[0024] Obtain the visual feature vector pairs corresponding to the node pairs;

[0025] Calculate the cosine similarity between visual feature vectors in a visual feature vector pair;

[0026] The video frame connection relationship of the node pair is determined by the second preset threshold constraint modal intra-connection relationship. That is, when the cosine similarity between visual feature vectors is greater than the second preset threshold, the sentence connection relationship of the node pair is represented as 1, otherwise it is represented as 0.

[0027] 3) When the nodes in a node pair are a video frame node and a sentence node, perform the following operations:

[0028] The intermodal time alignment relationship of the node pairs is calculated. The intermodal time alignment relationship includes the intermodal time alignment relationship from the video frame node to the sentence node and the intermodal time alignment relationship from the sentence node to the video frame node. That is, when the time windows of the video frame node and the sentence node are consistent, the intermodal time alignment relationship from the video frame node to the sentence node and the intermodal time alignment relationship from the sentence node to the video frame node are represented as 1, otherwise, they are represented as 0.

[0029] Preferably, a shallow fusion representation of multimodal feature representations is obtained through a global attention mechanism, as follows:

[0030] S2411. Calculate the query matrix, key matrix, and value matrix based on the multimodal feature representation;

[0031] S2412. Calculate the attention weight matrix based on the query matrix, key matrix, and adjacency representation;

[0032] S2413. Multiply the attention weight matrix and the value matrix to obtain the first updated feature, and then multiply the multimodal feature representation with the first learnable linear transformation matrix and perform a residual connection with the first updated feature to obtain a shallow fusion representation.

[0033] By inputting the shallow fusion representation and optimized heterogeneous graph into a two-layer graph attention network, a unified multimodal representation is obtained, as follows:

[0034] S2421. Optimize the heterogeneous graph by splitting it into four subgraphs according to the node connection type;

[0035] S2422. Input the shallow fusion representation into a two-layer graph attention network to propagate node features on each subgraph and then aggregate them to obtain a multimodal unified representation.

[0036] Preferably, the two-layer graph attention network includes a first graph attention module and a second graph attention module connected in sequence, and each graph attention module performs the following operations:

[0037] The input features of the corresponding graph attention module are propagated through the graph attention layer in each subgraph to obtain updated features, and the updated features of all nodes in the corresponding subgraph are concatenated to generate the updated feature matrix of the corresponding subgraph.

[0038] The updated feature matrix of the first subgraph and the updated feature matrix of the third subgraph are concatenated and multiplied by the first learnable weight matrix to obtain the video frame node representation. At the same time, the updated feature matrix of the second subgraph and the updated feature matrix of the fourth subgraph are concatenated and multiplied by the second learnable weight matrix to obtain the sentence node representation.

[0039] By concatenating the video frame node representations and sentence node representations, the output features of the corresponding graph attention module are obtained.

[0040] Preferably, the multimodal summarization generation model also performs the following operations:

[0041] S25. Based on the optimized heterogeneous graph, a restarted random walk is performed to construct a semantically enhanced directed graph;

[0042] S26. Calculate the joint loss function based on the semantically enhanced directed graph for backpropagation to optimize the model parameters of the multimodal summarization generation model, as follows:

[0043] S261. Perform a linear transformation on the unified multimodal representation to obtain the node importance score prediction results;

[0044] S262. After sorting the node importance score prediction results in descending order, select the top scores from the descending order. Each video frame node and the previous Each sentence node corresponds to a set of key video frame nodes. and key sentence node set , , , The total number of video frames in the video frame sequence. The total number of sentences in the sentence sequence;

[0045] S263, Based on the set of key video frame nodes and key sentence node set Construct a set of inter-modal contrastive triplets and a set of intra-modal contrastive triplets;

[0046] S264. Calculate the joint loss function based on the node importance score prediction results, the inter-modal comparison sample triplet set, and the intra-modal comparison sample triplet set;

[0047] S265. Optimize the model parameters of the multimodal summarization generation model using the backpropagation joint loss function.

[0048] Preferably, a semantically enhanced directed graph is constructed by performing a restarted random walk based on the optimized heterogeneous graph, as follows:

[0049] S251, Adjacency representation Normalize by column, i.e., make adjacent representations The sum of the elements in each column is 1, thus obtaining the random walk transition matrix. ;

[0050] S252. Set the initial random walk probability matrix. It is the identity matrix;

[0051] S253, Based on the preset restart probability Random walk transition matrix Initial random walk probability matrix The random walk probability matrix for the current iteration is updated using the random walk probability matrix from the previous iteration. Then, the... The probability matrix of the random walk in the next iteration The updated formula is as follows:

[0052]

[0053] in, For the first The probability matrix of random walks in the next iteration For the first The probability matrix of random walks in the next iteration , This represents the total number of iterations.

[0054] S254, Place Return to step S253 and continue until the random walk restarts and converges or the preset number of iterations is reached, to obtain the result. The probability matrix of the random walk in the next iteration As a probability distribution matrix;

[0055] S255, Arrange the nodes according to their connection type. The probability matrix of the random walk in the next iteration It is split into four sub-matrices, and the elements in the sub-matrices represent semantic alignment relationships;

[0056] S256. Determine the intermodal semantic alignment relationship and intramodal semantic alignment relationship of each node based on the four sub-matrices of the split. That is, select the different modal nodes with the largest corresponding semantic alignment relationship to generate the intermodal semantic alignment relationship, and select the same modal nodes with the largest corresponding semantic alignment relationship to generate the intramodal semantic alignment relationship. Both the intermodal semantic alignment relationship and the intramodal semantic alignment relationship are represented as 1.

[0057] S257. Use the inter-modal semantic alignment relationships and intra-modal semantic alignment relationships of each node as edge sets. Construct a semantically enhanced directed graph by assigning the corresponding element to the element at the specified position, while declaring all other elements as 0. .

[0058] Preferably, the set of inter-modal comparison sample triples includes the set of inter-modal comparison sample triples of video frames. The set of sample triples for comparison between sentence modalities ,in:

[0059] 1) Set of triplet samples for intermodal comparison of video frames The following information was obtained:

[0060] With key video frame node set As a set of video anchor points;

[0061] Set of key sentence nodes The nodes in the graph will enhance the semantics of the directed graph. Corresponding edge set Video frame nodes with an element of 1 are added to the video positive sample set. , contains One element;

[0062] From video frame node set Remove video anchor point set With the set of positive video samples , the remaining Each node constitutes a set of negative samples for the video. ;

[0063] Construct a set of triplet samples for intermodal comparison of video frames. ;

[0064] 2) Set of sentence modality comparison sample triplets The following information was obtained:

[0065] With key sentence node set As a set of sentence anchor points;

[0066] Set of key video frame nodes The nodes in the graph will enhance the semantics of the directed graph. Corresponding edge set Sentence nodes with an element of 1 are added to the sentence positive sample set. , contains One element;

[0067] From the set of sentence nodes Remove sentence anchor set With the set of positive sentences , the remaining Each node constitutes a negative sample set for the sentence. ;

[0068] Construct a set of sentence intermodal comparison sample triples ;

[0069] The intra-modal contrast sample triple set includes the video frame intra-modal contrast sample triple set. The set of comparison samples within sentence modality ,in:

[0070] 1) Set of intramodal contrast sample triples for video frames The following information was obtained:

[0071] In the set of key video frame nodes Randomly select one node as a video anchor point sample;

[0072] For video anchor point samples, semantically enhanced directed graphs are generated. Corresponding edge set The video frame node with an element of 1 is used as the positive sample of the video;

[0073] From the set of key video frame nodes After removing the video anchor point samples, a video frame node is randomly selected as the video negative sample, forming a video contrast triplet consisting of the video anchor sample, the video positive sample, and the video negative sample.

[0074] Traverse the set of key video frame nodes form Each video comparison triplet constitutes a set of intra-modal comparison sample triplets for each video frame. ;

[0075] 2) Set of sentence-modal comparison sample triplets The following information was obtained:

[0076] In the set of key sentence nodes Randomly select one node as a sentence anchor point sample;

[0077] For sentence anchor point samples, semantically enhanced directed graphs are generated. Corresponding edge set The sentence node whose element is 1 is used as the positive sample of the sentence;

[0078] From the set of key sentence nodes After removing sentence anchor samples, a sentence node is randomly selected as the sentence negative sample, forming a sentence comparison triplet consisting of sentence anchor samples, sentence positive samples, and sentence negative samples;

[0079] Traversing the set of key sentence nodes form The sentence comparison triples, and all sentence comparison triples constitute the sentence intra-modal comparison sample triple set. .

[0080] Preferably, the joint loss function is calculated based on the node importance score prediction results, the inter-modal comparison sample triplet set, and the intra-modal comparison sample triplet set, as follows:

[0081] S2631. Based on the node importance score prediction results, the classification loss is calculated using the Focal Loss function. ;

[0082] S2632, based on the set of triplet samples for inter-modal comparison of video frames respectively. The set of sample triples for comparison between sentence modalities The visual modality consistency loss is calculated using the Triplet Loss function. Text modality consistency loss ;

[0083] S2633, based on the set of triplets of intra-modal contrast samples in video frames respectively. The set of comparison samples within sentence modality The Triplet Loss function is used to calculate the intramodal diversity loss of visual modes. and sentence modality diversity loss ;

[0084] S2634. Obtain the joint loss function :

[0085]

[0086] in, For visual modal consistency loss The weight, Text modality consistency loss The weight, For visual modality diversity loss The weight, Loss of intramodal diversity of sentences The weight.

[0087] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0088] This method effectively integrates deep semantic relationships across modalities and diverse information within modalities into the summarization process through heterogeneous graph modeling and semantic enhancement mechanisms. This improves the semantic alignment and content richness of the summaries, resulting in more accurate and representative summaries. Specifically, video frame sequences and sentence sequences are obtained from the input teaching video to form corresponding nodes, and an adjacency matrix is ​​initialized to construct an initial heterogeneous graph. Then, the connection relationships between nodes are optimized to form an optimized heterogeneous graph, making the semantic connections within and between modalities more accurate. Next, a two-stage fusion strategy is executed on the optimized heterogeneous graph. First, a global attention mechanism is used to achieve shallow feature fusion, and then deep semantic interaction is achieved through subgraph propagation, thereby obtaining a unified multimodal representation. Subsequently, a restarted random walk is used to infer the semantic alignment relationships across modalities and within modalities, constructing a semantically enhanced directed graph. Based on this, key node screening and comparative sample sampling are performed, and classification loss, consistency loss, and diversity loss are combined for joint optimization. Finally, a summary graph containing key information is generated, and the resulting multimodal summary graph serves as the final teaching video summary generation result. This method can generate semantically consistent and content-rich teaching video summaries, and has strong generalization ability, making it particularly suitable for scenarios such as online education and smart classrooms. Attached Figure Description

[0089] Figure 1This is a flowchart of the multimodal teaching video summarization method based on heterogeneous graphs according to the present invention;

[0090] Figure 2 This is a flowchart of the two-stage fusion strategy of the present invention;

[0091] Figure 3 This is a flowchart of steps S25 and S26 of the present invention;

[0092] Figure 4 The images show the visualization of the summary generation results of the present invention. The left image is a comparison of the frame-by-frame summary generation results of the present invention method and different existing methods, and the right image is a comparison of the sentence-by-sentence summary generation results of the present invention method and different existing methods. Detailed Implementation

[0093] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0094] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0095] like Figures 1-4 As shown, a method for generating multimodal teaching video summaries based on heterogeneous graphs includes the following steps:

[0096] S1. Obtain several video samples to form a training set. The video samples include teaching videos, as well as corresponding sampled video frame sequences and sentence sequences.

[0097] Specifically, sampling each teaching video yields corresponding video frame sequences and sentence sequences. The sentences in the sentence sequences correspond to the text transcribed from the video frames within the sampling time period. The video frame sequences contain... The video frames contain a sentence sequence containing Each sentence corresponds to an element in a sequence that is generally sorted by time. By sampling the teaching video, two modalities are extracted: images (corresponding video frames) and text (corresponding sentences).

[0098] S2. Establish a multimodal summarization generation model and train it using the training set. The multimodal summarization generation model performs the following operations:

[0099] S21. Input the video frame sequence into the visual feature extraction model to obtain a set of visual feature vectors, and input the sentence sequence into the language model to obtain a set of text feature vectors. Concatenate the set of visual feature vectors and the set of text feature vectors to form a multimodal feature representation.

[0100] In one embodiment, the visual feature extraction model is a pre-trained ResNet model or a pre-trained ViT model, and the language model is a pre-trained BERT model or a pre-trained RoBERTa model.

[0101] Specifically, a visual feature extraction model (such as a pre-trained ResNet model or a ViT model; in this embodiment, a pre-trained ResNet model is used) is employed to encode the video frame sequence, thereby obtaining a set of visual feature vectors. The sentence sequence is encoded using a language model (such as a pre-trained BERT model or RoBERTa model; this embodiment uses a pre-trained BERT model) to obtain a set of text feature vectors. The set of visual feature vectors and text feature vector set Perform concatenation to form a multimodal feature representation of the teaching video. , , The total number of video frames in the video frame sequence. The total number of sentences in the sentence sequence. This represents the length of the visual feature vectors in the visual feature vector set or the length of the text feature vectors in the text feature vector set.

[0102] S22. Treat each video frame in the video frame sequence and each sentence in the sentence sequence as a node, and initialize the adjacency matrix so that there is a connection relationship between any nodes.

[0103] Specifically, the initialization process ensures that all nodes are connected to form an adjacency matrix of all 1s. Construct the initial heterogeneous graph , For the set of all nodes, , A set of video frame nodes, that is, including Each video frame node A set of sentence nodes, that is, including Each sentence node.

[0104] S23. Calculate the intramodal connectivity and intermodal time alignment between nodes respectively. Construct the intramodal constraint matrix and intermodal constraint matrix accordingly, and then perform a Hadamard product of these three matrices with the adjacency matrix to form the adjacency representation. Obtain optimized heterogeneous graph , This is the set of all nodes.

[0105] In one embodiment, the intramodal connectivity and intermodal time alignment relationships between nodes are calculated respectively, and the corresponding intramodal constraint matrices and intermodal constraint matrices are constructed as follows:

[0106] S231. Traverse the entire set of nodes to extract node pairs;

[0107] S232. When the nodes in a node pair are of the same modality, calculate the cosine similarity of the corresponding nodes to determine the intramodal connectivity. When the nodes in a node pair are of different modalities, calculate the intermodal temporal alignment of the corresponding nodes.

[0108] S233. Obtain the modal constraint matrix. and intermodal constraint matrix Among them, the intramodal constraint matrix top left corner area and bottom right corner The elements of a region are represented by the modal inner connectivity of corresponding node pairs, bottom left corner. area and top right corner All elements of the region are set to 0, and the intermodal constraint matrix is... bottom left corner area and top right corner The elements of a region are represented by the intermodal temporal alignment of the corresponding node pairs, with the top left corner as an example. area and bottom right corner All elements in the region are set to 0, where, The total number of video frames in the video frame sequence. This represents the total number of sentences in the sentence sequence.

[0109] In one embodiment, when the nodes in a node pair belong to the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intramodal connectivity. When the nodes in a node pair belong to different modalities, the intermodal temporal alignment relationship of the corresponding nodes is calculated, as follows:

[0110] 1) When all nodes in a node pair are sentence nodes, perform the following operations:

[0111] Obtain the text feature vector pairs corresponding to the node pairs;

[0112] Calculate the cosine similarity between text feature vectors and Chinese text feature vectors;

[0113] The sentence connection relationship between node pairs is determined by constraining the modal intra-connection relationship according to the first preset threshold. That is, when the cosine similarity between text feature vectors is greater than the first preset threshold, the sentence connection relationship between node pairs is represented as 1, otherwise it is represented as 0.

[0114] 2) When all nodes in a node pair are video frame nodes, perform the following operations:

[0115] Obtain the visual feature vector pairs corresponding to the node pairs;

[0116] Calculate the cosine similarity between visual feature vectors in a visual feature vector pair;

[0117] The video frame connection relationship of the node pair is determined by the second preset threshold constraint modal intra-connection relationship. That is, when the cosine similarity between visual feature vectors is greater than the second preset threshold, the sentence connection relationship of the node pair is represented as 1, otherwise it is represented as 0.

[0118] 3) When the nodes in a node pair are a video frame node and a sentence node, perform the following operations:

[0119] The intermodal time alignment relationship of the node pairs is calculated. The intermodal time alignment relationship includes the intermodal time alignment relationship from the video frame node to the sentence node and the intermodal time alignment relationship from the sentence node to the video frame node. That is, when the time windows of the video frame node and the sentence node are consistent, the intermodal time alignment relationship from the video frame node to the sentence node and the intermodal time alignment relationship from the sentence node to the video frame node are represented as 1, otherwise, they are represented as 0.

[0120] Specifically, the intramodal connectivity between nodes and the intermodal time alignment are calculated respectively, and the corresponding intramodal constraint matrices are constructed. and intermodal constraint matrix ,include:

[0121] Traverse the entire set of nodes Extract from the first The node and the first A node pair consisting of 1 node , , This represents the total number of nodes.

[0122] When node pair When nodes belong to the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intramodal connectivity. When nodes represent different modes, calculate the intermodal time alignment relationship of the corresponding nodes, i.e.:

[0123] 1) In node pairs When all nodes are sentence nodes, perform the following operations:

[0124] Obtain node pairs Corresponding text feature vector pairs ,in, For the first The text feature vector of each node, For the first The text feature vector of each node;

[0125] Calculate text feature vector pairs Cosine similarity between Chinese text feature vectors ;

[0126] According to the first preset threshold (e.g., 0.95) Constraint modal intra-connection relationships as node pairs Sentence connection relationship :

[0127]

[0128] 2) In node pairs When all nodes are video frame nodes, perform the following operations:

[0129] Obtain node pairs Corresponding visual feature vector pairs ,in, For the first The visual feature vector of each node, For the first Visual feature vectors of each node;

[0130] Calculate visual feature vector pairs Cosine similarity between visual feature vectors ;

[0131] According to the second preset threshold (e.g., 0.9) Constraint modal intra-connection relationships are treated as node pairs. Inter-frame connectivity of video :

[0132]

[0133] 3) In node pairs When the nodes are both video frame nodes and sentence nodes, the following operations are performed:

[0134] Compute node pairs The The node to the first Intermodal time alignment of nodes and node pairs The The node to the first Intermodal time alignment of nodes :

[0135]

[0136] in, For the first The time window of each node For the first The time window for each node.

[0137] Obtain the modal constraint matrix and intermodal constraint matrix Among them, the intramodal constraint matrix top left corner The elements of the region are It indicates the bottom right corner. The elements of the region are It indicates that the bottom left corner area and top right corner All elements of the region are set to 0, and the intermodal constraint matrix is... bottom left corner The elements of the region are It indicates that the upper right corner The elements of the region are It indicates that the top left corner area and bottom right corner All elements in the region are set to 0.

[0138] Using the Hadamard product to transform the intramodal constraint matrix Intermodal constraint matrix Merge and then connect with the adjacency matrix Performing the Hadamard product yields a more robust adjacency representation. :

[0139] .

[0140] S24. Execute a two-stage fusion strategy, that is, after obtaining a shallow fusion representation of multimodal feature representation through a global attention mechanism, input the shallow fusion representation and the optimized heterogeneous graph into a two-layer graph attention network to obtain a unified multimodal representation.

[0141] In one embodiment, a shallow fusion representation of multimodal feature representations is obtained through a global attention mechanism, as follows:

[0142] S2411. Calculate the query matrix, key matrix, and value matrix based on the multimodal feature representation;

[0143] S2412. Calculate the attention weight matrix based on the query matrix, key matrix, and adjacency representation;

[0144] S2413. Multiply the attention weight matrix and the value matrix to obtain the first updated feature, and then multiply the multimodal feature representation with the first learnable linear transformation matrix and perform a residual connection with the first updated feature to obtain a shallow fusion representation.

[0145] By inputting the shallow fusion representation and optimized heterogeneous graph into a two-layer graph attention network, a unified multimodal representation is obtained, as follows:

[0146] S2421. Optimize the heterogeneous graph by splitting it into four subgraphs according to the node connection type;

[0147] S2422. Input the shallow fusion representation into a two-layer graph attention network to propagate node features on each subgraph and then aggregate them to obtain a multimodal unified representation.

[0148] In one embodiment, the two-layer graph attention network includes a first graph attention module and a second graph attention module connected in sequence, and each graph attention module performs the following operations:

[0149] The input features of the corresponding graph attention module are propagated through the graph attention layer in each subgraph to obtain updated features, and the updated features of all nodes in the corresponding subgraph are concatenated to generate the updated feature matrix of the corresponding subgraph.

[0150] The updated feature matrix of the first subgraph and the updated feature matrix of the third subgraph are concatenated and multiplied by the first learnable weight matrix to obtain the video frame node representation. At the same time, the updated feature matrix of the second subgraph and the updated feature matrix of the fourth subgraph are concatenated and multiplied by the second learnable weight matrix to obtain the sentence node representation.

[0151] By concatenating the video frame node representations and sentence node representations, the output features of the corresponding graph attention module are obtained.

[0152] Specifically, a two-stage fusion strategy is implemented, which involves using a global attention mechanism to process multimodal feature representations. Perform shallow fusion to obtain shallow fusion representation Then, the shallow fusion representation is... and optimization of heterogeneous graphs Deep fusion of input two-layer graph attention networks yields a multimodal unified representation. ,in:

[0153] 1) Representing multimodal features through a global attention mechanism Perform shallow fusion to obtain shallow fusion representation The details are as follows:

[0154] Based on multimodal feature representation Calculate the query matrix separately Key matrix Sum matrix :

[0155]

[0156] in, , The learnable linear transformation matrices are, in order, the query matrix, the key matrix, and the value matrix. The second dimension of the query matrix, key matrix, or value matrix;

[0157] Based on the query matrix Key matrix Adjacency representation Calculate the attention weight matrix Then the attention weight matrix Middle The node and the first Attention weights of each node The calculation is as follows:

[0158]

[0159] in, Adjacency representation Middle The node and the first The connection representation of each node. Adjacency representation Middle The node and the first The connection representation of each node. , For query matrix Middle The value of each node, Key matrix Middle Transpose of the values ​​of each node Key matrix Middle Transpose of the values ​​of each node It is an exponential function;

[0160] attention weight matrix Sum matrix Multiplication yields the first updated feature, and the multimodal features are then represented. With the first learnable linear transformation matrix After multiplication, a residual connection is performed with the first updated feature to obtain a shallow fused representation. :

[0161] .

[0162] 2) Shallow layer fusion representation and optimization of heterogeneous graphs Deep fusion of input two-layer graph attention networks yields a multimodal unified representation. The details are as follows:

[0163] The heterogeneous graph will be optimized according to the node connection type. It is split into four subgraphs to form the first subgraph. Second subgraph Third subgraph and the fourth subgraph Adjacency representation The breakdown is as follows:

[0164]

[0165] in, Adjacency representation The undirected edge matrix between mid-video frame nodes, Adjacency representation The matrix of undirected edges between sentence nodes in the text. Adjacency representation The directed edge matrix from mid-video frame nodes to sentence nodes. Adjacency representation The directed edge matrix from sentence nodes to video frame nodes, in adjacency representation. Each of the split matrices represents a unique edge type. , A set of video frame nodes. This is a set of sentence nodes.

[0166] Shallow fusion representation Input a two-layer graph attention network to obtain a multimodal unified representation. A two-layer graph attention network includes a first graph attention module and a second graph attention module connected sequentially, wherein:

[0167] The attention module in the first image performs the following operations:

[0168] Shallow fusion representation The updated features are obtained by propagating node features on each subgraph, and the updated features of all nodes in the corresponding subgraph are concatenated to obtain the updated feature matrix of the corresponding subgraph. Specifically:

[0169] 1) Calculate the third subgraph The update characteristics of each video frame node, i.e. Then the first Update characteristics of each node The formula is as follows:

[0170]

[0171]

[0172] in, For the third subgraph The node to the first The aggregation weight coefficient of each node, , To point to the third subgraph A set of sentence nodes with n nodes. For normalization function, For activation functions (such as the Sigmoid activation function, etc.). For splicing operations, The learnable linear mapping parameter matrix for the third subgraph. The learnable attention weight matrix for the third subgraph. The second dimension is the dimension of the learnable linear mapping parameter matrix in the attention module of the first graph; Shallow fusion representation Middle A shallow fusion feature, Shallow fusion representation Middle A shallow fusion feature, ;

[0173] splicing the third subgraph The update feature matrix of the third subgraph is obtained by analyzing the update features of all video frame nodes. .

[0174] 2) Calculate the first subgraph The update characteristics of each video frame node, i.e. Then the first Update characteristics of each node The formula is as follows:

[0175]

[0176]

[0177] in, For the first subgraph The node to the first The aggregation weight coefficient of each node, , To point to the first subgraph A set of video frame nodes of nodes. Let be the learnable linear mapping parameter matrix of the first subgraph. This is the learnable attention weight matrix for the first subgraph;

[0178] splicing the first sub-image The update feature matrix of the first subgraph is obtained by analyzing the update features of all video frame nodes. .

[0179] 3) Calculate the second subgraph The update features of each sentence node in the text, namely Then the first Update characteristics of each node The formula is as follows:

[0180]

[0181]

[0182] in, In the second subgraph The node to the first The aggregation weight coefficient of each node, , To point to the second subgraph A set of video frame nodes of nodes. The learnable linear mapping parameter matrix for the second subgraph. This is the learnable attention weight matrix for the second subgraph;

[0183] splicing the second sub-graph The update feature matrix of the second subgraph is obtained by updating the features of all sentence nodes in the graph. .

[0184] 4) Calculate the fourth subgraph The update features of each sentence node in the text, namely Then the first Update characteristics of each node The formula is as follows:

[0185]

[0186]

[0187] in, For the fourth subgraph The node to the first The aggregation weight coefficient of each node, , To point to the fourth subgraph A set of sentence nodes with n nodes. The learnable linear mapping parameter matrix for the fourth subgraph. This is the learnable attention weight matrix for the fourth subgraph;

[0188] splicing the fourth sub-image The update feature matrix of the fourth subgraph is obtained by updating the features of all sentence nodes in the graph. .

[0189] Update the feature matrix of the first subgraph The updated feature matrix of the third subgraph After concatenation, it is compared with the first learnable weight matrix. Multiply to obtain the video frame node representation. At the same time, update the feature matrix of the second subgraph. The updated feature matrix of the fourth subgraph After concatenation, it is compared with the second learnable weight matrix. Multiply to obtain the sentence node representation. ,in:

[0190] , .

[0191] Representing video frame nodes Sentence node representation Perform a concatenation operation to obtain the output features of the attention module in the first image. .

[0192] Similarly, the output features of the attention module in the first image are... Input the attention module in the second graph, and perform the propagation and aggregation operations as in the attention module in the first graph to obtain the output features of the attention module in the second graph, i.e., the multimodal unified representation. , Let be the second dimension of the learnable linear mapping parameter matrix in the attention module of the second graph. All are preset values.

[0193] In one embodiment, the multimodal summarization generation model also performs the following operations:

[0194] S25. Based on the optimized heterogeneous graph, a restarted random walk is performed to construct a semantically enhanced directed graph;

[0195] S26. Calculate the joint loss function based on the semantically enhanced directed graph for backpropagation to optimize the model parameters of the multimodal summarization generation model, as follows:

[0196] S261. Perform a linear transformation on the unified multimodal representation to obtain the node importance score prediction results;

[0197] S262. After sorting the node importance score prediction results in descending order, select the top scores from the descending order. Each video frame node and the previous Each sentence node corresponds to a set of key video frame nodes. and key sentence node set , , , The total number of video frames in the video frame sequence. The total number of sentences in the sentence sequence;

[0198] S263, Based on the set of key video frame nodes and key sentence node set Construct a set of inter-modal contrastive triplets and a set of intra-modal contrastive triplets;

[0199] S264. Calculate the joint loss function based on the node importance score prediction results, the inter-modal comparison sample triplet set, and the intra-modal comparison sample triplet set;

[0200] S265. Optimize the model parameters of the multimodal summarization generation model using the backpropagation joint loss function.

[0201] Specifically, based on the semantically enhanced directed graph Calculate the joint loss function The model parameters used for backpropagation to optimize the multimodal summarization generation model, i.e., the unified representation of multimodality. Perform a linear transformation (using a multilayer perceptron, MLP) to obtain the node importance score prediction results. Prediction results of node importance scores After sorting the fractions in descending order, select the first few fractions. Each video frame node and the previous Each sentence node corresponds to a set of key video frame nodes. and key sentence node set , , To construct the intermodal contrast sample triplet set and the intramodal contrast sample triplet set.

[0202] In one embodiment, a semantically enhanced directed graph is constructed by performing a restarted random walk based on an optimized heterogeneous graph, as detailed below:

[0203] S251, Adjacency representation Normalize by column, i.e., make adjacent representations The sum of the elements in each column is 1, thus obtaining the random walk transition matrix. ;

[0204] S252. Set the initial random walk probability matrix. It is the identity matrix;

[0205] S253, Based on the preset restart probability Random walk transition matrix Initial random walk probability matrix The random walk probability matrix for the current iteration is updated using the random walk probability matrix from the previous iteration. Then, the... The probability matrix of the random walk in the next iteration The updated formula is as follows:

[0206]

[0207] in, For the first The probability matrix of random walks in the next iteration For the first The probability matrix of random walks in the next iteration , This represents the total number of iterations.

[0208] S254, Place Return to step S253 and continue until the random walk restarts and converges or the preset number of iterations is reached, to obtain the result. The probability matrix of the random walk in the next iteration As a probability distribution matrix;

[0209] S255, Arrange the nodes according to their connection type. The probability matrix of the random walk in the next iteration It is split into four sub-matrices, and the elements in the sub-matrices represent semantic alignment relationships;

[0210] S256. Determine the intermodal semantic alignment relationship and intramodal semantic alignment relationship of each node based on the four sub-matrices of the split. That is, select the different modal nodes with the largest corresponding semantic alignment relationship to generate the intermodal semantic alignment relationship, and select the same modal nodes with the largest corresponding semantic alignment relationship to generate the intramodal semantic alignment relationship. Both the intermodal semantic alignment relationship and the intramodal semantic alignment relationship are represented as 1.

[0211] S257. Use the inter-modal semantic alignment relationships and intra-modal semantic alignment relationships of each node as edge sets. Construct a semantically enhanced directed graph by assigning the corresponding element to the element at the specified position, while declaring all other elements as 0. .

[0212] Specifically, based on optimizing heterogeneous graphs Perform a restarted random walk to construct a semantically enhanced directed graph. The details are as follows:

[0213] adjacency representation Normalize by column, i.e., make adjacent representations The sum of the elements in each column is 1, thus obtaining the random walk transition matrix. ;

[0214] Set the initial random walk probability matrix It is the identity matrix;

[0215] Based on the preset restart probability Random walk transition matrix Initial random walk probability matrix The random walk probability matrix for the current iteration is updated using the random walk probability matrix from the previous iteration. Then, the... The probability matrix of the random walk in the next iteration The updated formula is as follows:

[0216]

[0217] in, For the first The probability matrix of random walks in the next iteration For the first The probability matrix of random walks in the next iteration , This represents the total number of iterations.

[0218] Place Return to step S253 and continue until the random walk restarts and converges or the preset number of iterations is reached, to obtain the result. The probability matrix of the random walk in the next iteration As a probability distribution matrix The last iteration number at which the random walk converges or reaches the preset number of iterations;

[0219] The first The probability matrix of the random walk in the next iteration Split according to node connection type to obtain four sub-matrices:

[0220]

[0221] in, This refers to the semantic alignment relationships between video frame nodes. This refers to the semantic alignment relationship between video frame nodes and sentence nodes. This represents the semantic alignment relationship between sentence nodes and video frame nodes. This refers to the semantic alignment relationships between sentence nodes;

[0222] Determine the inter-modal semantic alignment relationship and intra-modal semantic alignment relationship for each node. Specifically, select the different modal nodes with the largest corresponding semantic alignment relationship to generate the inter-modal semantic alignment relationship, and select the same modal nodes with the largest corresponding semantic alignment relationship to generate the intra-modal semantic alignment relationship. Both inter-modal and intra-modal semantic alignment relationships are represented as 1. Specifically, the following describes the... Each node will be explained:

[0223] 1) No. Each node is a video frame node:

[0224] Select and the first The node with the largest semantic alignment relationship among nodes. , , ,and for Middle The node to the first The node whose semantic alignment relationship between nodes reaches its maximum value will be the node with the highest semantic alignment relationship among all nodes. The intermodal semantic alignment relationship of the nodes is represented as follows: At the same time, select the one with the first The node with the largest semantic alignment relationship among nodes. , , ,and for Middle The node to the first The node whose semantic alignment relationship between nodes reaches its maximum value will be the node with the highest semantic alignment relationship among all nodes. The intramodal alignment relationship of the nodes is represented as follows: .

[0225] 2) No. Each node is a sentence node:

[0226] Select and the first The node with the largest semantic alignment relationship among nodes. , , ,and for Middle The node to the first The node whose semantic alignment relationship between nodes reaches its maximum value will be the node with the highest semantic alignment relationship among all nodes. The intermodal semantic alignment relationship of nodes is represented as follows: At the same time, select the one with the first The node with the largest semantic alignment relationship among nodes. , , ,and for Middle The node to the first The node whose semantic alignment relationship between nodes reaches its maximum value will be the node with the highest semantic alignment relationship among all nodes. The intramodal alignment relationship of the nodes is represented as follows: .

[0227] Based on the above four types of semantic alignment relationships edge set The elements of the corresponding nodes are set to 0, while the elements of other nodes are set to 0, to construct a semantically enhanced directed graph. That is, edge set. An element of 1 indicates that the corresponding nodes are semantically aligned; otherwise, the corresponding nodes are not semantically aligned.

[0228] In one embodiment, the inter-modal contrast sample triplet set includes the video frame inter-modal contrast sample triplet set. The set of sample triples for comparison between sentence modalities ,in:

[0229] 1) Set of triplet samples for intermodal comparison of video frames The following information was obtained:

[0230] With key video frame node set As a set of video anchor points;

[0231] Set of key sentence nodes The nodes in the graph will enhance the semantics of the directed graph. Corresponding edge set Video frame nodes with an element of 1 are added to the video positive sample set. , contains One element;

[0232] From video frame node set Remove video anchor point set With the set of positive video samples , the remaining Each node constitutes a set of negative samples for the video. ;

[0233] Construct a set of triplet samples for intermodal comparison of video frames. ;

[0234] 2) Set of sentence modality comparison sample triplets The following information was obtained:

[0235] With key sentence node set As a set of sentence anchor points;

[0236] Set of key video frame nodes The nodes in the graph will enhance the semantics of the directed graph. Corresponding edge set Sentence nodes with an element of 1 are added to the sentence positive sample set. , contains One element;

[0237] From the sentence node set Remove sentence anchor set With the set of positive sentences , the remaining Each node constitutes a negative sample set for the sentence. ;

[0238] Construct a set of sentence intermodal comparison sample triples ;

[0239] The intra-modal contrast sample triple set includes the video frame intra-modal contrast sample triple set. The set of comparison samples within sentence modality ,in:

[0240] 1) Set of intramodal contrast sample triples for video frames The following information was obtained:

[0241] In the set of key video frame nodes Randomly select one node as a video anchor point sample;

[0242] For video anchor point samples, semantically enhanced directed graphs are generated. Corresponding edge set The video frame node with an element of 1 is used as the positive sample of the video;

[0243] From the set of key video frame nodes After removing the video anchor point samples, a video frame node is randomly selected as the video negative sample, forming a video contrast triplet consisting of the video anchor sample, the video positive sample, and the video negative sample.

[0244] Traverse the set of key video frame nodes form Each video comparison triplet constitutes a set of intra-modal comparison sample triplets for each video frame. ;

[0245] 2) Set of sentence-modal comparison sample triplets The following information was obtained:

[0246] In the set of key sentence nodes Randomly select one node as a sentence anchor point sample;

[0247] For sentence anchor point samples, semantically enhanced directed graphs are generated. The sentence nodes when the elements in the corresponding edge set are 1 are used as sentence positive samples;

[0248] Remove the sentence anchor samples from the key sentence node set and randomly select a sentence node as a sentence negative sample to form a sentence contrast triple consisting of sentence anchor samples, sentence positive samples and sentence negative samples;

[0249] Traverse the key sentence node set to form sentence contrast triples, and all sentence contrast triples constitute the sentence intra-modal contrast sample triple set .

[0250] Specifically, the video contrast triple composed of video anchor samples, video positive samples and video negative samples is represented as , where is the video anchor sample corresponding to the th node, , is the video positive sample corresponding to the th node, <000107...... is the video negative sample corresponding to the th node, , and the sentence contrast triple composed of sentence anchor samples, sentence positive samples and sentence negative samples is represented as , where is the sentence anchor sample corresponding to the th node, , is the sentence positive sample corresponding to the th node <...... is the sentence negative sample corresponding to the th node, , .

[0251] In one embodiment, calculate the joint loss function according to the node importance score prediction result, the inter-modal contrast sample triple set and the intra-modal contrast sample triple set, specifically as follows:

[0252] S2631. Calculate the classification loss using the Focal Loss function based on the node importance score prediction result ;

[0253] S26...... and the sentence inter-modal contrast sample triple set [[ID=...... The visual modality consistency loss is calculated using the Triplet Loss function. Text modality consistency loss ;

[0254] S2633, based on the set of triplets of intra-modal contrast samples in video frames respectively. The set of comparison samples within sentence modality The Triplet Loss function is used to calculate the intramodal diversity loss of visual modes. and sentence modality diversity loss ;

[0255] S2634. Obtain the joint loss function :

[0256]

[0257] in, For visual modal consistency loss The weight, Text modality consistency loss The weight, For visual modality diversity loss The weight, Loss of intramodal diversity of sentences The weight.

[0258] Specifically, the joint classification loss, inter-modal consistency loss, and intra-modal diversity loss generate a joint loss function. The details are as follows:

[0259] Based on the prediction results of node importance scores The classification loss is calculated using the Focal Loss function. :

[0260]

[0261] in, Indicates the first The gold standard label for each node, Indicates the first Each node represents a summary video frame or a summary sentence. Indicates the first The node is neither a summary video frame nor a summary sentence. , For hyperparameters, , , Node importance score prediction results Middle The score of each node.

[0262] Based on the set of inter-modal comparison samples of video frames respectively The set of sample triples for comparison between sentence modalities The visual modality consistency loss is calculated using the Triplet Loss function. Text modality consistency loss :

[0263]

[0264]

[0265] in, For a unified representation of multimodal Middle The feature values ​​of each node (deep fusion features) For a unified representation of multimodal Middle The feature values ​​of each node For a unified representation of multimodal Middle The feature values ​​of each node ,and , , It is a Euclidean distance function. This represents the minimum discriminant distance between the preset positive and negative samples.

[0266] Based on the set of intramodal contrast sample triples of video frames respectively The set of comparison samples within sentence modality The Triplet Loss function is used to calculate the intramodal diversity loss of visual modes. and sentence modality diversity loss :

[0267]

[0268]

[0269] Obtain the joint loss function :

[0270]

[0271] in, For visual modal consistency loss The weight, Text modality consistency loss The weight, For visual modality diversity loss The weight, Loss of intramodal diversity of sentences The weights, in this embodiment , , , The values ​​are 0.05, 0.03, 0.02, and 0.01, respectively. The quality of the generated summary is improved through model parameter optimization. Visual modality refers to image modality.

[0272] S3. Use the multimodal unified representation output by the trained multimodal summarization generation model to filter out the set of key video frame nodes. and key sentence node set The corresponding connectivity relationships within the optimized heterogeneous graph are retained as sub-adjacency matrices. Obtain multimodal summary graph This is the result of generating a summary of the teaching video.

[0273] Once training is complete using the training set, the trained multimodal summary is used to generate the multimodal unified representation output by the model. From the complete set of nodes Select key video frame nodes from the data. and key sentence node set And retain the corresponding set of key video frame nodes. and key sentence node set In optimizing heterogeneous graphs The internal connectivity is used as a sub-adjacency matrix Obtain multimodal summary graph This is the final result of generating the teaching video summary.

[0274] The effectiveness of the method of this invention was verified on the VT-SSum dataset. The VT-SSum dataset is derived from large-scale teaching video resources, covering a variety of courses and topics, and provides high-quality transcribed text and summary annotations. It is widely used in the research and evaluation of automatic summarization of teaching videos. On the VT-SSum dataset, the method of this invention was compared with two multimodal summarization methods, A2Summ and TaleSumm, and two graph representation-based video learning methods, MGL and VideoSAGE. References for A2Summ are: B. He, J. Wang, J. Qiu, T. Bui, A. Shrivastava, and Z. Wang, “Align and attend: Multimodal summarization with dual contrastive losses,” in CVPR, 2023, pp. 14 867–14 878.; References for TaleSumm are: AK Singh, D. Srivastava, and M. Tapaswi, “Previously on... from recaps tostory summarization,” in CVPR, 2024, pp. 13 635–13 646.; References for MGL are: J. Xie, Z. Zhao, Z. Lin, and Y. Shen, “Multimodal graph learning for cross-modal retrieval,” in… SDM, 2023, pp. 145–153.; VideoSAGE Reference: JMR Chaves and S. Tripathi, “Videosage: Video summarization with graph representation learning,” in CVPR, 2024, pp. 2527–2534. Experimental results show that the method of this invention achieves excellent performance in terms of summarization accuracy, content richness, and semantic consistency. Taking a lecture video introducing the English vocabulary database WordNet from the VT-SSum dataset (numbered “22gipsfuw7sfapbpjtl3jg4vankygwyk” in the VT-SSum dataset) as an example, as follows... Figure 4 As shown, importance prediction visualizations were first performed frame-by-frame and sentence-by-sentence for both image and text modalities. Figure 4 The left-middle figure is a comparison of the frame-by-frame summary generation results of the method of the present invention and different existing methods. Figure 4 The right-middle figure is a comparison of the sentence-by-sentence summarization results generated by the method of this invention and different existing methods. Figure 4 Middle left image and Figure 4 In the right-hand graph, the black line represents the gold standard label for the summarized video frame (corresponding to the left graph) or summarized sentence (corresponding to the right graph). The green line represents the score of the summarized video frame or summarized sentence predicted by the method of this invention (i.e., the score corresponding to the predicted node importance score). The other colored lines correspond to the scores of the summarized video frames or summarized sentences predicted by the comparison methods. For example, the yellow line represents the score of the summarized video frame or summarized sentence predicted by the A2Summ method, the red line represents the score of the summarized video frame or summarized sentence predicted by the TaleSumm method, the purple line represents the score of the summarized video frame or summarized sentence predicted by the MGL method, and the cyan line represents the score of the summarized video frame or summarized sentence predicted by the VideoSAGE method. It can be seen that the method of this invention is superior to the comparison methods in fitting the summarized result label lines, and the prediction curves of different modalities show highly similar trends, maintaining the same key segments and crucial information at the semantic level. Figure 4 The advantages of this method in terms of summary extraction accuracy and multimodal alignment capability have been demonstrated. Results show that the summaries generated by the method of this invention are significantly superior to the comparative methods in terms of both hit rate and diversity, verifying the effectiveness and superiority of the method of this invention in teaching video summarization tasks.

[0275] This method effectively integrates deep semantic relationships across modalities and diverse information within modalities into the summarization process through heterogeneous graph modeling and semantic enhancement mechanisms. This improves the semantic alignment and content richness of the summaries, resulting in more accurate and representative summaries. Specifically, video frame sequences and sentence sequences are obtained from the input teaching video to form corresponding nodes, and an adjacency matrix is ​​initialized to construct an initial heterogeneous graph. Then, the connection relationships between nodes are optimized to form an optimized heterogeneous graph, making the semantic connections within and between modalities more accurate. Next, a two-stage fusion strategy is executed on the optimized heterogeneous graph. First, a global attention mechanism is used to achieve shallow feature fusion, and then deep semantic interaction is achieved through subgraph propagation, thereby obtaining a unified multimodal representation. Subsequently, a restarted random walk is used to infer the semantic alignment relationships across modalities and within modalities, constructing a semantically enhanced directed graph. Based on this, key node screening and comparative sample sampling are performed, and classification loss, consistency loss, and diversity loss are combined for joint optimization. Finally, a summary graph containing key information is generated, and the resulting multimodal summary graph serves as the final teaching video summary generation result. This method can generate semantically consistent and content-rich teaching video summaries, and has strong generalization ability, making it particularly suitable for scenarios such as online education and smart classrooms.

[0276] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0277] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for generating multimodal teaching video summaries based on heterogeneous graphs, characterized in that: Includes the following steps: S1. Obtain several video samples to form a training set, wherein the video samples include teaching videos, as well as corresponding sampled video frame sequences and sentence sequences; S2. Establish a multimodal summarization generation model and train it using a training set. The multimodal summarization generation model performs the following operations: S21. Input the video frame sequence into the visual feature extraction model to obtain a set of visual feature vectors, and input the sentence sequence into the language model to obtain a set of text feature vectors. Concatenate the set of visual feature vectors and the set of text feature vectors to form a multimodal feature representation. S22. Treat each video frame in the video frame sequence and each sentence in the sentence sequence as a node, and initialize the adjacency matrix so that there is a connection relationship between any nodes. S23. Calculate the intramodal connectivity and intermodal time alignment between nodes respectively. Construct the intramodal constraint matrix and intermodal constraint matrix accordingly, and then perform a Hadamard product of these three matrices with the adjacency matrix to form the adjacency representation. Obtain optimized heterogeneous graph , The set of all nodes; S24. Execute a two-stage fusion strategy, that is, after obtaining a shallow fusion representation of multimodal feature representation through a global attention mechanism, input the shallow fusion representation and the optimized heterogeneous graph into a two-layer graph attention network to obtain a unified multimodal representation. S3. Use the multimodal unified representation output by the trained multimodal summarization generation model to filter out the set of key video frame nodes. and key sentence node set The corresponding connectivity relationships within the optimized heterogeneous graph are retained as sub-adjacency matrices. Obtain multimodal summary graph This is the result of generating a summary of the teaching video.

2. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 1, characterized in that: The visual feature extraction model is a pre-trained ResNet model or a pre-trained ViT model, and the language model is a pre-trained BERT model or a pre-trained RoBERTa model.

3. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 1, characterized in that: The calculation of intramodal connectivity and intermodal time alignment between nodes is performed separately, and the corresponding intramodal constraint matrix and intermodal constraint matrix are constructed as follows: S231. Traverse the entire set of nodes to extract node pairs; S232. When the nodes in a node pair are of the same modality, calculate the cosine similarity of the corresponding nodes to determine the intramodal connectivity. When the nodes in a node pair are of different modalities, calculate the intermodal temporal alignment of the corresponding nodes. S233. Obtain the modal constraint matrix. and intermodal constraint matrix Among them, the intramodal constraint matrix top left corner area and bottom right corner The elements of a region are represented by the modal inner connectivity of corresponding node pairs, bottom left corner. area and top right corner All elements of the region are set to 0, and the intermodal constraint matrix is... bottom left corner area and top right corner The elements of a region are represented by the intermodal temporal alignment of the corresponding node pairs, with the top left corner as an example. area and bottom right corner All elements in the region are set to 0, where, This represents the total number of video frames in the video frame sequence. This represents the total number of sentences in the sentence sequence.

4. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 3, characterized in that: When the nodes in a node pair are of the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intramodal connectivity. When the nodes in a node pair are of different modalities, the intermodal temporal alignment of the corresponding nodes is calculated, as follows: 1) When all nodes in a node pair are sentence nodes, perform the following operations: Obtain the text feature vector pairs corresponding to the node pairs; Calculate the cosine similarity between text feature vectors and Chinese text feature vectors; The sentence connection relationship between node pairs is determined by constraining the modal intra-connection relationship according to the first preset threshold. That is, when the cosine similarity between text feature vectors is greater than the first preset threshold, the sentence connection relationship between node pairs is represented as 1, otherwise it is represented as 0. 2) When all nodes in a node pair are video frame nodes, perform the following operations: Obtain the visual feature vector pairs corresponding to the node pairs; Calculate the cosine similarity between visual feature vectors in a visual feature vector pair; The video frame connection relationship of the node pair is determined by the second preset threshold constraint modal intra-connection relationship. That is, when the cosine similarity between visual feature vectors is greater than the second preset threshold, the sentence connection relationship of the node pair is represented as 1, otherwise it is represented as 0. 3) When the nodes in a node pair are a video frame node and a sentence node, perform the following operations: The intermodal time alignment relationship of the node pairs is calculated. The intermodal time alignment relationship includes the intermodal time alignment relationship from the video frame node to the sentence node and the intermodal time alignment relationship from the sentence node to the video frame node. That is, when the time windows of the video frame node and the sentence node are consistent, the intermodal time alignment relationship from the video frame node to the sentence node and the intermodal time alignment relationship from the sentence node to the video frame node are represented as 1, otherwise, they are represented as 0.

5. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 1, characterized in that: The shallow fusion representation obtained through the global attention mechanism for multimodal feature representation is as follows: S2411. Calculate the query matrix, key matrix, and value matrix based on the multimodal feature representation; S2412. Calculate the attention weight matrix based on the query matrix, key matrix, and adjacency representation; S2413. Multiply the attention weight matrix and the value matrix to obtain the first updated feature, and then multiply the multimodal feature representation with the first learnable linear transformation matrix and perform a residual connection with the first updated feature to obtain a shallow fusion representation. The process of inputting shallow fusion representations and optimized heterogeneous graphs into a two-layer graph attention network to obtain a multimodal unified representation is as follows: S2421. Optimize the heterogeneous graph by splitting it into four subgraphs according to the node connection type; S2422. Input the shallow fusion representation into a two-layer graph attention network to propagate node features on each subgraph and then aggregate them to obtain a multimodal unified representation.

6. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 5, characterized in that: The two-layer graph attention network includes a first graph attention module and a second graph attention module connected in sequence. Each graph attention module performs the following operations: The input features of the corresponding graph attention module are propagated through the graph attention layer in each subgraph to obtain updated features, and the updated features of all nodes in the corresponding subgraph are concatenated to generate the updated feature matrix of the corresponding subgraph. The updated feature matrix of the first subgraph and the updated feature matrix of the third subgraph are concatenated and multiplied by the first learnable weight matrix to obtain the video frame node representation. At the same time, the updated feature matrix of the second subgraph and the updated feature matrix of the fourth subgraph are concatenated and multiplied by the second learnable weight matrix to obtain the sentence node representation. By concatenating the video frame node representations and sentence node representations, the output features of the corresponding graph attention module are obtained.

7. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 1, characterized in that: The multimodal summary generation model also performs the following operations: S25. Based on the optimization of the heterogeneous graph, a restarted random walk is performed to construct a semantically enhanced directed graph; S26. Calculate the joint loss function based on the semantically enhanced directed graph for backpropagation to optimize the model parameters of the multimodal summarization generation model, as follows: S261. Perform a linear transformation on the unified multimodal representation to obtain the node importance score prediction results; S262. After sorting the node importance score prediction results in descending order, select the top scores from the descending order. Each video frame node and the previous Each sentence node corresponds to a set of key video frame nodes. and key sentence node set , , , This represents the total number of video frames in the video frame sequence. The total number of sentences in the sentence sequence; S263, Based on the set of key video frame nodes and key sentence node set Construct a set of inter-modal contrastive triplets and a set of intra-modal contrastive triplets; S264. Calculate the joint loss function based on the node importance score prediction results, the inter-modal comparison sample triplet set, and the intra-modal comparison sample triplet set; S265. Optimize the model parameters of the multimodal summarization generation model using the backpropagation joint loss function.

8. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 7, characterized in that: The method of resuming random walks based on optimized heterogeneous graphs to construct semantically enhanced directed graphs is as follows: S251, Adjacency representation Normalize by column, i.e., make adjacent representations The sum of the elements in each column is 1, thus obtaining the random walk transition matrix. ; S252. Set the initial random walk probability matrix. It is the identity matrix; S253, Based on the preset restart probability Random walk transition matrix Initial random walk probability matrix The random walk probability matrix for the current iteration is updated using the random walk probability matrix from the previous iteration. Then, the... The probability matrix of the random walk in the next iteration The updated formula is as follows: ； in, For the first The probability matrix of random walks in the next iteration For the first The probability matrix of random walks in the next iteration , This represents the total number of iterations. S254, Place Return to step S253 and continue until the random walk restarts and converges or the preset number of iterations is reached, to obtain the result. The probability matrix of the random walk in the next iteration As a probability distribution matrix; S255, Arrange the nodes according to their connection type. The probability matrix of the random walk in the next iteration It is split into four sub-matrices, and the elements in the sub-matrices represent semantic alignment relationships; S256. Determine the intermodal semantic alignment relationship and intramodal semantic alignment relationship of each node based on the four sub-matrices of the splitting. That is, select the different modal nodes with the largest corresponding node semantic alignment relationship to generate the intermodal semantic alignment relationship, and select the same modal nodes with the largest corresponding node semantic alignment relationship to generate the intramodal semantic alignment relationship. The intermodal semantic alignment relationship and the intramodal semantic alignment relationship are both represented as 1. S257. Use the inter-modal semantic alignment relationships and intra-modal semantic alignment relationships of each node as edge sets. Construct a semantically enhanced directed graph by assigning the corresponding element to the element at the specified position, while declaring all other elements as 0. .

9. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 7, characterized in that: The inter-modal contrast sample triplet set includes the video frame inter-modal contrast sample triplet set. The set of sample triples for comparison between sentence modalities ,in: 1) Set of triplet samples for intermodal comparison of video frames The following information was obtained: With key video frame node set As a set of video anchor points; Set of key sentence nodes The nodes in the graph will enhance the semantics of the directed graph. Corresponding edge set Video frame nodes with an element of 1 are added to the video positive sample set. , contains One element; From video frame node set Remove video anchor point set With the set of positive video samples , the remaining Each node constitutes a set of negative samples for the video. ; Construct a set of triplet samples for intermodal comparison of video frames. ; 2) Set of sentence modality comparison sample triplets The following information was obtained: With key sentence node set As a set of sentence anchor points; Set of key video frame nodes The nodes in the graph will enhance the semantics of the directed graph. Corresponding edge set Sentence nodes with an element of 1 are added to the sentence positive sample set. , contains One element; From the sentence node set Remove sentence anchor set With the set of positive sentences , the remaining Each node constitutes a negative sample set for the sentence. ; Construct a set of sentence intermodal comparison sample triples ; The intra-modal contrast sample triplet set includes the video frame intra-modal contrast sample triplet set. The set of comparison samples within sentence modality ,in: 1) Set of intramodal contrast sample triples for video frames The following information was obtained: In the set of key video frame nodes Randomly select one node as a video anchor point sample; For video anchor point samples, semantically enhanced directed graphs are generated. Corresponding edge set The video frame node with an element of 1 is used as the positive sample of the video; From the set of key video frame nodes After removing the video anchor point samples, a video frame node is randomly selected as the video negative sample, forming a video contrast triplet consisting of the video anchor sample, the video positive sample, and the video negative sample. Traverse the set of key video frame nodes form Each video comparison triplet constitutes a set of intra-modal comparison sample triplets for each video frame. ; 2) Set of sentence-modal comparison sample triplets The following information was obtained: In the set of key sentence nodes Randomly select one node as a sentence anchor point sample; For sentence anchor point samples, semantically enhanced directed graphs are generated. Corresponding edge set The sentence node whose element is 1 is used as the positive sample of the sentence; From the set of key sentence nodes After removing sentence anchor samples, a sentence node is randomly selected as the sentence negative sample, forming a sentence comparison triplet consisting of sentence anchor samples, sentence positive samples, and sentence negative samples; Traversing the set of key sentence nodes form The sentence comparison triples, and all sentence comparison triples constitute the sentence intra-modal comparison sample triple set. .

10. The method for generating multimodal teaching video summaries based on heterogeneous graphs as described in claim 7, characterized in that: The joint loss function is calculated based on the node importance score prediction results, the inter-modal comparison sample triplet set, and the intra-modal comparison sample triplet set, as detailed below: S2631. Based on the node importance score prediction results, the classification loss is calculated using the Focal Loss function. ; S2632, based on the set of triplet samples for inter-modal comparison of video frames respectively. The set of sample triples for comparison between sentence modalities The visual modality consistency loss is calculated using the Triplet Loss function. Text modality consistency loss ; S2633, based on the set of triplets of intra-modal contrast samples in video frames respectively. The set of comparison samples within sentence modality The Triplet Loss function is used to calculate the intramodal diversity loss of visual vision. and sentence modality diversity loss ; S2634. Obtain the joint loss function : ； in, For visual modal consistency loss The weight, Text modality consistency loss The weight, For visual modality diversity loss The weight, Loss of intramodal diversity of sentences The weight.

Images