Heterogeneous graph-based multi-modal teaching video abstract generation method

Abstract

The invention discloses a heterogeneous graph-based multi-modal teaching video abstract generation method. The method comprises the steps of obtaining a plurality of video samples to form a training set; establishing a multi-modal abstract generation model, training by using a training set, and executing the following operations by the model: respectively inputting a video frame sequence and a sentence sequence into a visual feature extraction model and a language model to obtain a visual feature vector set and a text feature vector set to form multi-modal feature representation; initializing an adjacent matrix; performing Hadamard product on the intra-modal constraint matrix, the inter-modal constraint matrix and the adjacent matrix to obtain an optimized heterogeneous graph; executing a double-stage fusion strategy; and screening a key video frame node set and a key sentence node set by using multi-modal unified representation output by the trained multi-modal abstract generation model, and correspondingly reserving a connection relationship in the optimized heterogeneous graph as a sub-adjacency matrix to obtain a multi-modal abstract graph. According to the method, teaching video abstracts with consistent semantics and rich contents can be generated, and the generalization ability is high.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video summary generation, and particularly relates to a multi-modal teaching video summary generation method based on a heterogeneous graph. BACKGROUND

[0002] With the development of digitalization of education resources, teaching videos, as a main knowledge dissemination form, are widely used in online education and smart classrooms. However, teaching videos generally have problems such as long length, uneven information density, and content redundancy, which make it difficult for learners to efficiently and quickly obtain core knowledge points in a limited time. Therefore, a multi-modal video summary method has gradually become a research hotspot. The basic framework of the multi-modal video summary method generally includes an encoder for extracting features from each modality, a multi-modal fusion module, and a decoder for generating a summary. However, since the encoder usually models the features of different modalities separately, the fusion stage mainly relies on time alignment and late fusion strategies. Although this method can ensure alignment in the time dimension, it is difficult to capture deeper semantic associations between cross-modalities, which leads to the fact that the generated multi-modal summary is reasonable in time sequence, but often lacks semantic consistency and expression diversity. Therefore, how to simultaneously consider cross-modal semantic alignment and content diversity within modalities during the summary generation process is still a key problem to be solved. SUMMARY

[0003] The application aims to solve the above problems and provides a multi-modal teaching video summary generation method based on a heterogeneous graph, which can generate a teaching video summary with semantic consistency and rich content and has strong generalization ability.

[0004] To achieve the above object, the application adopts the following technical scheme:

[0005] The multi-modal teaching video summary generation method based on a heterogeneous graph provided by the application includes the following steps:

[0006] S1, a plurality of video samples are obtained to form a training set, the video samples including teaching videos, and corresponding sampled video frame sequences and sentence sequences;

[0007] S2, a multi-modal summary generation model is established and trained using the training set, and the multi-modal summary generation model performs the following operations:

[0008] S21, the video frame sequence is input into a visual feature extraction model to obtain a set of visual feature vectors, and the sentence sequence is input into a language model to obtain a set of text feature vectors, and the set of visual feature vectors and the set of text feature vectors are spliced to form a multi-modal feature representation;

[0009] S22, each video frame in the video frame sequence and each sentence in the sentence sequence are regarded as a node, and an adjacency matrix is initialized by allowing any nodes to have a connection relationship;

[0010] S23, respectively calculate the intra-modal connection relationship and the inter-modal time alignment relationship between nodes, and perform Hadamard product on the three of the constructed intra-modal constraint matrix and inter-modal constraint matrix and the adjacency matrix to form an adjacency representation , obtain an optimized heterogeneous graph , is the set of all nodes;

[0011] S24, perform a two-stage fusion strategy, that is, obtain a shallow fusion representation of the multi-modal feature representation through a global attention mechanism, then input the shallow fusion representation and the optimized heterogeneous graph into a double-layer graph attention network to obtain a multi-modal unified representation;

[0012] S3, filter out a set of key video frame nodes and a set of key sentence nodes from the multi-modal unified representation output by the trained multi-modal summary generation model and , and reserve the connection relationship in the optimized heterogeneous graph as a sub-adjacency matrix , obtain a multi-modal summary graph as a teaching video summary generation result.

[0013] Preferably, the visual feature extraction model is a pre-trained ResNet model or a pre-trained ViT model, and the language model is a pre-trained BERT model or a pre-trained RoBERTa model.

[0014] Preferably, the intra-modal connection relationship and the inter-modal time alignment relationship between nodes are respectively calculated, and the intra-modal constraint matrix and the inter-modal constraint matrix are constructed, as follows:

[0015] S231, traverse the set of all nodes to extract node pairs;

[0016] S232, when the nodes in the node pair are of the same modality, calculate the cosine similarity of the corresponding nodes to determine the intra-modal connection relationship, and when the nodes in the node pair are of different modalities, calculate the inter-modal time alignment relationship of the corresponding nodes;

[0017] S233, obtain the intra-modal constraint matrix and the inter-modal constraint matrix , wherein the elements in the upper left corner region and the lower right corner region of the intra-modal constraint matrix are represented by the intra-modal connection relationship of the corresponding node pair, the elements in the lower left corner region and the upper right corner region are all set to 0, and the elements in the lower left corner region and the upper right corner region of the inter-modal constraint matrix elements of the region are set to 0, wherein, the region and the bottom right corner elements of the region are set to 0, wherein, is a total number of video frames in the sequence of video frames, is a total number of sentences in the sequence of sentences.

[0018] Preferably, the cosine similarity of the corresponding nodes is calculated to determine the intra-modal connection relationship when the nodes in the node pair are of the same modality, and the inter-modal time alignment relationship of the corresponding nodes is calculated when the nodes in the node pair are of different modalities, as follows:

[0019] 1) When the nodes in the node pair are both sentence nodes, the following operations are performed:

[0020] obtain a pair of text feature vectors corresponding to the node pair;

[0021] calculate the cosine similarity between the text feature vectors in the pair of text feature vectors;

[0022] constrain the intra-modal connection relationship as the inter-sentence connection relationship of the node pair according to a first preset threshold, that is, when the cosine similarity between the text feature vectors is greater than the first preset threshold, the inter-sentence connection relationship of the node pair is represented as 1, otherwise, it is represented as 0;

[0023] 2) When the nodes in the node pair are both video frame nodes, the following operations are performed:

[0024] obtain a pair of visual feature vectors corresponding to the node pair;

[0025] calculate the cosine similarity between the visual feature vectors in the pair of visual feature vectors;

[0026] constrain the intra-modal connection relationship as the inter-video frame connection relationship of the node pair according to a second preset threshold, that is, when the cosine similarity between the visual feature vectors is greater than the second preset threshold, the inter-sentence connection relationship of the node pair is represented as 1, otherwise, it is represented as 0;

[0027] 3) When the nodes in the node pair are respectively a video frame node and a sentence node, the following operations are performed:

[0028] calculate the inter-modal time alignment relationship of the node pair, which includes the inter-modal time alignment relationship from the video frame node to the sentence node and the inter-modal time alignment relationship from the sentence node to the video frame node, that is, when the time windows of the video frame node and the sentence node are consistent, the inter-modal time alignment relationship from the video frame node to the sentence node and the inter-modal time alignment relationship from the sentence node to the video frame node are represented as 1, otherwise, they are represented as 0.

[0029] Preferably, the shallow fusion representation of the multi-modal feature representation is obtained by a global attention mechanism, specifically as follows:

[0030] S2411、According to the multi-modal feature representation, a query matrix, a key matrix and a value matrix are calculated respectively;

[0031] S2412、According to the query matrix, the key matrix and the adjacency representation, an attention weight matrix is calculated;

[0032] S2413、The attention weight matrix and the value matrix are multiplied to obtain a first updated feature, and the multi-modal feature representation is multiplied by a first learnable linear transformation matrix and then connected in residual with the first updated feature to obtain a shallow fusion representation;

[0033] The shallow fusion representation and the optimized heterogeneous graph are input into a double-layer graph attention network to obtain a multi-modal unified representation, specifically as follows:

[0034] S2421、The optimized heterogeneous graph is split into four subgraphs according to the node connection type;

[0035] S2422、The shallow fusion representation is input into the double-layer graph attention network to propagate and aggregate the node features on each subgraph respectively to obtain a multi-modal unified representation.

[0036] Preferably, the double-layer graph attention network includes a first graph attention module and a second graph attention module connected in sequence, and each graph attention module performs the following operations:

[0037] The input features of the corresponding graph attention module are propagated through the graph attention layer on each subgraph to obtain updated features, and the updated features of all nodes of the corresponding subgraph are spliced to generate an updated feature matrix of the corresponding subgraph;

[0038] The updated feature matrix of the first subgraph and the updated feature matrix of the third subgraph are spliced and multiplied by a first learnable weight matrix to obtain a video frame node representation, and the updated feature matrix of the second subgraph and the updated feature matrix of the fourth subgraph are spliced and multiplied by a second learnable weight matrix to obtain a sentence node representation;

[0039] The video frame node representation and the sentence node representation are spliced to obtain the output features of the corresponding graph attention module.

[0040] Preferably, the multi-modal summary generation model further performs the following operations:

[0041] S25、Restart random walk is performed based on the optimized heterogeneous graph to construct a semantic enhanced directed graph;

[0042] S26、A joint loss function is calculated according to the semantic enhanced directed graph for backpropagation to optimize the model parameters of the multi-modal summary generation model, specifically as follows:

[0043] S261, linearly transforming the multi-modal unified representation to obtain a node importance score prediction result;

[0044] S262, after arranging scores in the node importance score prediction result in descending order, selecting the first K video frame nodes and the first K sentence nodes according to the scores arranged in descending order, respectively, to form a set of key video frame nodes and a set of key sentence nodes. is the total number of video frames in the video frame sequence, is the total number of sentences in the sentence sequence.

[0045] S263, constructing a set of inter-modal contrast sample triplets and a set of intra-modal contrast sample triplets according to the set of key video frame nodes and the set of key sentence nodes.

[0046] S264, calculating a joint loss function according to the node importance score prediction result, the set of inter-modal contrast sample triplets and the set of intra-modal contrast sample triplets.

[0047] S265, back-propagating the joint loss function to optimize model parameters of the multi-modal summary generation model.

[0048] Preferably, the restart random walk is performed based on the optimized heterogeneous graph to construct a semantic-enhanced directed graph, and the construction is specifically as follows:

[0049] S251, normalizing the adjacency representation by column, i.e., setting the sum of each column element in the adjacency representation to 1 to obtain a random walk transition matrix

[0050] S252, setting an initial random walk probability matrix as an identity matrix.

[0051] S253, updating a current iteration random walk probability matrix according to a preset restart probability , the random walk transition matrix , the initial random walk probability matrix and a last iteration random walk probability matrix, so that the random walk probability matrix of the nth iteration is updated as follows:

[0052] ​​​​​​​​​​​​​

[0053] wherein, is the random walk probability matrix of the first iteration, is the random walk probability matrix of the first iteration, is the random walk probability matrix of the first iteration, is the random walk probability matrix of the first iteration, , is the total number of iterations;

[0054] S254, set , return to step S253 until the random walk converges or the preset number of iterations is reached, and obtain the random walk probability matrix of the first iteration as the probability distribution matrix;

[0055] S255, split the random walk probability matrix of the first iteration into four sub-matrices according to the node connection type, and the elements in the sub-matrices represent semantic alignment relationships;

[0056] S256, determine the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship of each node according to the four split sub-matrices, that is, select different modal nodes with the largest corresponding node semantic alignment relationship to generate the inter-modal semantic alignment relationship, and select the same modal nodes with the largest corresponding node semantic alignment relationship to generate the intra-modal semantic alignment relationship, and the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship are both represented as 1.

[0057] S257, take the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship of each node as the elements of the corresponding positions in the edge set , and other position elements are represented as 0, and construct a semantic enhanced directed graph .

[0058] Preferably, the inter-modal contrast sample triple set includes a video frame inter-modal contrast sample triple set and a sentence inter-modal contrast sample triple set , wherein:

[0059] 1) The video frame inter-modal contrast sample triple set is obtained as follows:

[0060] Take the key video frame node set as the video anchor point set;

[0061] For the nodes in the key sentence node set , if the element of the corresponding edge set in the semantic enhanced directed graph is 1, add the video frame node to the video positive sample set , which contains ​​elements;

[0062] remove the video anchor point set from the video frame node set with the video positive sample set , the remaining nodes constitute the video negative sample set ;

[0063] Construct a video frame inter-modal contrast sample triple set ;

[0064] 2) Sentence inter-modal contrast sample triple set , get as follows:

[0065] Take the key sentence node set as the sentence anchor point set;

[0066] For the nodes in the key video frame node set , if the element of the corresponding edge set in the semantic enhanced directed graph is 1, add the sentence node to the sentence positive sample set , which contains elements in total;

[0067] Remove the sentence anchor point set from the sentence node set and the sentence positive sample set , and the remaining nodes constitute the sentence negative sample set ;

[0068] Construct a sentence inter-modal contrast sample triple set ;

[0069] The intra-modal contrast sample triple set includes the video frame intra-modal contrast sample triple set and the sentence intra-modal contrast sample triple set , wherein:

[0070] 1) Video frame intra-modal contrast sample triple set , get as follows:

[0071] Take any node in the key video frame node set as the video anchor point sample;

[0072] For the video anchor point sample, take the video frame node whose element in the corresponding edge set in the semantic enhanced directed graph is 1 as the video positive sample;

[0073] ​​​​From the key video frame node collection After removing the video anchor sample, a video frame node is randomly selected as the video negative sample to form a video comparison triplet consisting of the video anchor sample, the video positive sample and the video negative sample;

[0074] Traverse the key video frame node collection form Video comparison triplets, all video comparison triplets constitute the set of comparison sample triplets within the video frame modality ;

[0075] 2) Set of contrast sample triples within sentence modality , obtained as follows:

[0076] In the key sentence node set Take any node as the sentence anchor sample;

[0077] For sentence anchor samples, the semantic enhancement directed graph The corresponding edge set The sentence node when the element of is 1 is regarded as a positive sentence sample;

[0078] From the key sentence node set After removing the sentence anchor sample, a sentence node is randomly selected as the sentence negative sample to form a sentence comparison triple consisting of the sentence anchor sample, sentence positive sample and sentence negative sample;

[0079] Traverse the key sentence node set form Sentence comparison triples, all sentence comparison triples constitute the set of comparison sample triples within the sentence modality .

[0080] Preferably, a joint loss function is calculated based on the node importance score prediction results, the inter-modality comparison sample triple set, and the intra-modality comparison sample triple set, as follows:

[0081] S2631, based on the node importance score prediction results, the Focal Loss loss function is used to calculate the classification loss ;

[0082] S2632, based on the video frame inter-modality comparison sample triplet set Comparison sample triples between sentence modalities The Triplet Loss loss function is used to calculate the visual modality consistency loss. and text modality consistency loss ;

[0083] S2633, respectively based on the video frame intra-modal contrast sample triple set and the sentence intra-modal contrast sample triple set The Triplet Loss loss function is used to calculate the visual intra-modal diversity loss and the sentence intra-modal diversity loss .

[0084] S2634, obtain a joint loss function :

[0085]

[0086] wherein, is the weight of the visual modal consistency loss , is the weight of the text modal consistency loss , is the weight of the visual intra-modal diversity loss , is the weight of the sentence intra-modal diversity loss .

[0087] Compared with the prior art, the beneficial effects of the present application are:

[0088] The method effectively integrates the cross-modal deep semantic relationship and the intra-modal diversity information into the summary generation process through the heterogeneous graph modeling and semantic enhancement mechanism, thereby improving the semantic alignment degree and content richness of the summary, and the final obtained summary has higher accuracy and representativeness. Specifically, video frame sequences and sentence sequences are obtained from the input teaching video to form corresponding nodes, and an adjacency matrix is initialized to construct an initial heterogeneous graph; then the connection relationship between the nodes is optimized to form an optimized heterogeneous graph, so that the semantic connection between the modal and the modal is more accurate; then a two-stage fusion strategy is performed on the optimized heterogeneous graph, first the shallow feature fusion is realized by using the global attention mechanism, and then the deep semantic interaction is realized by subgraph propagation, thereby obtaining a unified representation of multiple modalities; subsequently, the semantic alignment relationship between the cross-modal and the intra-modal is inferred by using the restart random walk, and a semantic enhanced directed graph is constructed; on this basis, key node screening and contrast sample sampling are performed, and joint optimization is performed in combination with the classification loss, the consistency loss and the diversity loss; finally, a summary graph containing key information is generated, and a multi-modal summary graph is obtained as the final generated teaching video summary generation result. That is, the method can generate a teaching video summary with consistent semantics and rich content, has strong generalization ability, and is especially suitable for online education and smart classroom scenes. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1Flowchart of the method for generating summaries of multimodal teaching videos based on heterogeneous graphs according to the present invention;

[0090] Figure 2 is a flow chart of the two-stage fusion strategy of the present invention;

[0091] Figure 3 This is a flow chart of steps S25 and S26 of the present invention;

[0092] Figure 4 This is a visualization of the summary generation results of the present invention. The left figure is a comparison of the frame-by-frame summary generation results of the method of the present invention and different existing methods, and the right figure is a comparison of the sentence-by-sentence summary generation results of the method of the present invention and different existing methods. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0094] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of this application. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0095] like Figures 1-4 As shown in FIG, a method for generating summaries of multimodal teaching videos based on heterogeneous graphs includes the following steps:

[0096] S1. Obtain several video samples to form a training set. The video samples include teaching videos, and corresponding sampled video frame sequences and sentence sequences.

[0097] Specifically, sampling each teaching video can obtain the corresponding video frame sequence and sentence sequence. The sentences in the sentence sequence correspond to the text transcribed from the video frames of the sampling time period. The video frame sequence contains video frames, the sentence sequence contains The elements in the corresponding sequence are generally sorted by time. By sampling the teaching video, two modalities, image (corresponding to video frames) and text (corresponding to sentences), are extracted.

[0098] S2. Establish a multimodal summary generation model and train it using the training set. The multimodal summary generation model performs the following operations:

[0099] S21, input the video frame sequence into a visual feature extraction model respectively to obtain a visual feature vector set, and input the sentence sequence into a language model to obtain a text feature vector set, and concatenate the visual feature vector set and the text feature vector set to form a multi-modal feature representation.

[0100] In an embodiment, the visual feature extraction model is a pre-trained ResNet model or a pre-trained ViT model, and the language model is a pre-trained BERT model or a pre-trained RoBERTa model.

[0101] Specifically, the visual feature extraction model (such as a pre-trained ResNet model or a ViT model, and a pre-trained ResNet model is used in this embodiment) is used to encode the video frame sequence to obtain a visual feature vector set ; the language model (such as a pre-trained BERT model or a RoBERTa model, and a pre-trained BERT model is used in this embodiment) is used to encode the sentence sequence to obtain a text feature vector set . The visual feature vector set and the text feature vector set are concatenated by Concat to form a multi-modal feature representation of the teaching video , , is the total number of video frames in the video frame sequence, is the total number of sentences in the sentence sequence, is the length of the visual feature vector in the visual feature vector set or the length of the text feature vector in the text feature vector set.

[0102] S22, each video frame in the video frame sequence and each sentence in the sentence sequence are regarded as a node, and an adjacency matrix is initialized by making any nodes have a connection relationship.

[0103] Specifically, the initialized adjacency matrix is a full 1 matrix , and an initial heterogeneous graph is constructed, is the set of all nodes, , is the set of video frame nodes, that is, it includes video frame nodes, is the set of sentence nodes, that is, it includes sentence nodes.

[0104] S23, the intra-modal connection relationship and the inter-modal time alignment relationship between nodes are calculated respectively, and the intra-modal constraint matrix and the inter-modal constraint matrix are constructed correspondingly, and then the Hadamard product of the three matrices is performed to form an adjacency representation , and an optimized heterogeneous graph is obtained. , The set of all nodes.

[0105] In one embodiment, the intra-modal connection relationship and the inter-modal time alignment relationship between nodes are calculated respectively, and the intra-modal constraint matrix and the inter-modal constraint matrix are constructed accordingly, as follows:

[0106] S231, traverse all node sets to extract node pairs;

[0107] S232. When the nodes in the node pair are of the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intra-modal connection relationship; when the nodes in the node pair are of different modalities, the inter-modal time alignment relationship of the corresponding nodes is calculated;

[0108] S233. Obtaining the modal constraint matrix and the intermodal constraint matrix , where the intra-modal constraint matrix is upper left corner area and lower right corner The elements of the region are represented by the modal intra-connection relationship of the corresponding node pairs, the lower left corner area and upper right corner The elements of the region are all set to 0, and the inter-modal constraint matrix lower left corner area and upper right corner The elements of the region are represented by the temporal alignment relationship between the corresponding node pairs, the upper left corner area and lower right corner The elements of the region are all set to 0, where is the total number of video frames in the video frame sequence, is the total number of sentences in the sentence sequence.

[0109] In one embodiment, when the nodes in a node pair are of the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intra-modal connection relationship. When the nodes in the node pair are of different modalities, the inter-modal time alignment relationship of the corresponding nodes is calculated, as follows:

[0110] 1) When both nodes in a node pair are sentence nodes, perform the following operations:

[0111] Get the text feature vector pair corresponding to the node pair;

[0112] Calculate the cosine similarity between text feature vectors in a text feature vector pair;

[0113] Constraining the intra-modal connection relationship as the inter-sentence connection relationship of the node pair according to a first preset threshold, that is, when the cosine similarity between the text feature vectors is greater than the first preset threshold, the inter-sentence connection relationship of the node pair is represented as 1, otherwise it is represented as 0;

[0114] 2) When the nodes in the node pair are both video frame nodes, the following operations are performed:

[0115] Obtain a pair of visual feature vectors corresponding to the node pair;

[0116] Calculate the cosine similarity between the visual feature vectors in the pair of visual feature vectors;

[0117] According to the second preset threshold, the intra-modal connection relationship is constrained as the inter-modal connection relationship between the video frames in the node pair, that is, when the cosine similarity between the visual feature vectors is greater than the second preset threshold, the inter-sentence connection relationship in the node pair is represented as 1, otherwise, it is represented as 0;

[0118] 3) When the nodes in the node pair are a video frame node and a sentence node respectively, the following operations are performed:

[0119] Calculate the inter-modal time alignment relationship of the node pair, which includes the inter-modal time alignment relationship of the video frame node to the sentence node and the inter-modal time alignment relationship of the sentence node to the video frame node, that is, when the time windows of the video frame node and the sentence node are consistent, the inter-modal time alignment relationship of the video frame node to the sentence node and the inter-modal time alignment relationship of the sentence node to the video frame node are represented as 1, otherwise, they are represented as 0.

[0120] Specifically, the intra-modal connection relationship and the inter-modal time alignment relationship between the nodes are calculated respectively, and the intra-modal constraint matrix and the inter-modal constraint matrix are constructed accordingly, including:

[0121] Traverse all node sets Extract a node pair consisting of the first node and the second node , , , The total number of nodes is

[0122] When the nodes in the node pair are of the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intra-modal connection relationship, and when the nodes in the node pair are of different modalities, the inter-modal time alignment relationship of the corresponding nodes is calculated, that is:

[0123] 1) When the nodes in the node pair are both sentence nodes, the following operations are performed:

[0124] Obtain a pair of text feature vectors corresponding to the node pair , wherein is the th text feature vector of the first node text feature vector of the i-th node, text feature vector of the i-th node; text feature vector of the i-th node;

[0125] calculate the cosine similarity between the text feature vectors of the node pair ;

[0126] According to the first preset threshold (such as 0.95), the intra-modal connection relationship is constrained as the inter-sentence connection relationship of the node pair :

[0127]

[0128] 2) When the nodes in the node pair are both video frame nodes, the following operations are performed: obtain the visual feature vector pair corresponding to the node pair

[0129] , wherein the visual feature vector of the i-th node is , and the visual feature vector of the j-th node is ;

[0130] calculate the cosine similarity between the visual feature vectors in the visual feature vector pair ;

[0131] According to the second preset threshold (such as 0.9), the intra-modal connection relationship is constrained as the inter-video frame connection relationship of the node pair :

[0132]

[0133] 3) When the nodes in the node pair are respectively a video frame node and a sentence node, the following operations are performed: calculate the inter-modal time alignment relationship from the i-th node to the j-th node of the node pair

[0134] and the inter-modal time alignment relationship from the i-th node to the j-th node of the node pair :

[0135] ​​​​​​​​​​​​​​​​​

[0136] wherein, is the time window of the th node, is the time window of the th node.

[0137] obtain an intra-modal constraint matrix and an inter-modal constraint matrix , wherein, the elements in the upper-left corner region of the intra-modal constraint matrix are denoted by , the elements in the lower-right corner region are denoted by , the elements in the left-lower corner region and the right-upper corner region are both set to 0, the elements in the lower-left corner region of the inter-modal constraint matrix are denoted by , the elements in the upper-right corner region are denoted by , the elements in the upper-left corner region and the lower-right corner region are both set to 0.

[0138] The intra-modal constraint matrix , the inter-modal constraint matrix are combined by Hadamard product, and then Hadamard product is performed with the adjacency matrix to obtain a more robust adjacency representation :

[0139] .

[0140] S24, a two-stage fusion strategy is executed, that is, after obtaining the shallow fusion representation of the multi-modal feature representation by the global attention mechanism, the shallow fusion representation and the optimized heterogeneous graph are input into the double-layer graph attention network to obtain the multi-modal unified representation.

[0141] In an embodiment, the shallow fusion representation of the multi-modal feature representation is obtained by the global attention mechanism, specifically as follows:

[0142] S2411, query matrix, key matrix and value matrix are calculated according to the multi-modal feature representation respectively;

[0143] S2412, attention weight matrix is calculated according to the query matrix, the key matrix and the adjacency representation;

[0144] S2413, multiply the attention weight matrix and the value matrix to obtain a first updated feature, and perform residual connection between the first updated feature and the multi-modal feature representation after multiplying the multi-modal feature representation and a first learnable linear transformation matrix to obtain a shallow fusion representation;

[0145] Input the shallow fusion representation and the optimized heterogeneous graph into the double-layer graph attention network to obtain a multi-modal unified representation, specifically as follows:

[0146] S2421, split the optimized heterogeneous graph into four subgraphs according to the node connection type;

[0147] S2422, input the shallow fusion representation into the double-layer graph attention network to perform node feature propagation on each subgraph respectively after aggregation, and obtain a multi-modal unified representation.

[0148] In an embodiment, the double-layer graph attention network includes a first graph attention module and a second graph attention module connected in sequence, and each graph attention module performs the following operations:

[0149] The input feature corresponding to the graph attention module is propagated through the graph attention layer on each subgraph to obtain an updated feature, and the updated features of all nodes of the corresponding subgraph are spliced to generate an updated feature matrix of the corresponding subgraph;

[0150] The updated feature matrix of the first subgraph and the updated feature matrix of the third subgraph are spliced and multiplied by a first learnable weight matrix to obtain a video frame node representation, and the updated feature matrix of the second subgraph and the updated feature matrix of the fourth subgraph are spliced and multiplied by a second learnable weight matrix to obtain a sentence node representation;

[0151] The video frame node representation and the sentence node representation are spliced to obtain the output feature of the corresponding graph attention module.

[0152] Specifically, a two-stage fusion strategy is performed, that is, the multi-modal feature representation is shallowly fused through a global attention mechanism to obtain a shallow fusion representation , and then the shallow fusion representation and the optimized heterogeneous graph are input into the double-layer graph attention network for deep fusion to obtain a multi-modal unified representation , wherein:

[0153] 1) The multi-modal feature representation is shallowly fused through a global attention mechanism to obtain a shallow fusion representation , specifically as follows:

[0154] According to the multi-modal feature representation , a query matrix , a key matrix and value matrix :

[0155]

[0156] wherein, , are the learnable linear transformation matrices of the query matrix, the key matrix, and the value matrix, respectively, is the dimension of the second dimension of the query matrix or the key matrix or the value matrix;

[0157] According to the query matrix , the key matrix , and the adjacency representation , the attention weight matrix is calculated, and the attention weight of the i-th node and the j-th node in the attention weight matrix is calculated as follows:

[0158]

[0159] wherein, is the connection representation of the i-th node and the j-th node in the adjacency representation , is the connection representation of the i-th node and the j-th node in the adjacency representation , , is the value of the i-th node in the query matrix , is the transpose of the value of the i-th node in the key matrix , is the transpose of the value of the i-th node in the key matrix , is an exponential function; The attention weight matrix is multiplied by the value matrix to obtain the first updated feature, and the multi-modal feature representation is multiplied by the first learnable linear transformation matrix and then connected in residual with the first updated feature to obtain the shallow fusion representation :

[0160]

[0161] .

[0162] ​​​​​​​​​2) input the shallow fusion representation into a two-layer graph attention network to obtain a multimodal unified representation and optimize the heterogeneous graph input the shallow fusion representation into a two-layer graph attention network to obtain a multimodal unified representation , specifically as follows:

[0163] split the optimized heterogeneous graph according to the node connection type into four subgraphs, forming a first subgraph , a second subgraph , a third subgraph and a fourth subgraph , the adjacency representation is split as follows:

[0164]

[0165] wherein, is the undirected edge matrix between video frame nodes in the adjacency representation , is the undirected edge matrix between sentence nodes in the adjacency representation , is the directed edge matrix from video frame nodes to sentence nodes in the adjacency representation , is the directed edge matrix from sentence nodes to video frame nodes in the adjacency representation , each matrix split from the adjacency representation represents an independent edge type, , is the set of video frame nodes, is the set of sentence nodes. input the shallow fusion representation into a two-layer graph attention network to obtain a multimodal unified representation , the two-layer graph attention network includes a first graph attention module and a second graph attention module connected in turn, wherein: the first graph attention module performs the following operations:

[0166] input the shallow fusion representation into a two-layer graph attention network to obtain a multimodal unified representation , the two-layer graph attention network includes a first graph attention module and a second graph attention module connected in turn, wherein:

[0167] the first graph attention module performs the following operations:

[0168] input the shallow fusion representation into a two-layer graph attention network to obtain a multimodal unified representation , the two-layer graph attention network includes a first graph attention module and a second graph attention module connected in turn, wherein:

[0169] 1) calculate the updated features of each video frame node in the third subgraph , i.e. , the updated feature of the j-th node is , the formula is as follows:

[0170] ​​​

[0171]

[0172] wherein, is the aggregated weight coefficient of the th node to the th node under the third sub-graph, , is the set of sentence nodes pointing to the th node under the third sub-graph, is a normalization function, is an activation function (such as a Sigmoid activation function, etc.), is a concatenation operation, is a learnable linear mapping parameter matrix of the third sub-graph, is a learnable attention weight matrix of the third sub-graph, is the dimension of the second dimension of the learnable linear mapping parameter matrix in the first graph attention module; is the th shallow fusion feature in the shallow fusion representation , is the th shallow fusion feature in the shallow fusion representation , ;

[0173] concatenates the updated features of all video frame nodes in the third sub-graph to obtain the updated feature matrix of the third sub-graph. .

[0174] 2) Calculate the updated features of each video frame node in the first sub-graph, i.e. , then the updated feature of the th node is , and the formula is as follows:

[0175]

[0176]

[0177] wherein, is the aggregated weight coefficient of the th node to the th node under the first sub-graph, , is the set of video frame nodes pointing to the th node under the first sub-graph, is a learnable linear mapping parameter matrix of the first sub-graph, is a learnable attention weight matrix of the first sub-graph;​

[0178] Splice the first sub-image Update features of all video frame nodes in the first subgraph to obtain the updated feature matrix .

[0179] 3) Calculate the second subgraph The updated features of each sentence node in , then Update characteristics of nodes , the formula is as follows:

[0180]

[0181]

[0182] in, For the second subgraph Node to The aggregation weight coefficient of each node, , To point to the first A collection of video frame nodes of nodes, is the learnable linear mapping parameter matrix of the second subgraph, is the learnable attention weight matrix of the second subgraph;

[0183] Splice the second sub-image Update the features of all sentence nodes in the second subgraph to obtain the updated feature matrix .

[0184] 4) Calculate the fourth subgraph The updated features of each sentence node in , then Update characteristics of nodes , the formula is as follows:

[0185]

[0186]

[0187] in, For the fourth subgraph Node to The aggregation weight coefficient of each node, , To point to the fourth subgraph The sentence node set of nodes, is the learnable linear mapping parameter matrix of the fourth subgraph, is the learnable attention weight matrix of the fourth subgraph;

[0188] concatenate the fourth subgraph obtain the updated feature matrix of the fourth subgraph by updating the features of all sentence nodes .

[0189] concatenate the updated feature matrix of the first subgraph and the updated feature matrix of the third subgraph , and multiply the concatenated result by the first learnable weight matrix to obtain the video frame node representation , and concatenate the updated feature matrix of the second subgraph and the updated feature matrix of the fourth subgraph , and multiply the concatenated result by the second learnable weight matrix to obtain the sentence node representation , wherein

[0190] , .

[0191] concatenate the video frame node representation and the sentence node representation to obtain the output feature of the first graph attention module .

[0192] Similarly, input the output feature of the first graph attention module to the second graph attention module to perform the propagation and aggregation operations as in the first graph attention module, and obtain the output feature of the second graph attention module, i.e., the multi-modal unified representation , is the dimension of the second dimension of the learnable linear mapping parameter matrix in the second graph attention module. are all preset values.

[0193] In an embodiment, the multi-modal summary generation model further performs the following operations:

[0194] S25, performing restart random walk based on the optimized heterogeneous graph to construct a semantic enhanced directed graph;

[0195] S26, calculating a joint loss function based on the semantic enhanced directed graph for back propagation to optimize the model parameters of the multi-modal summary generation model, specifically as follows:

[0196] S261, performing linear transformation on the multi-modal unified representation to obtain a node importance score prediction result;

[0197] S262, after arranging the scores in the node importance score prediction result in descending order, selecting the top video frame nodes and the top a sentence node, to form a set of key video frame nodes and a set of key sentence nodes and a set of key sentence nodes , 、 , is a total number of video frames in the video frame sequence, is a total number of sentences in the sentence sequence;

[0198] S263, constructing a set of inter-modal contrastive sample triplets and a set of intra-modal contrastive sample triplets according to the set of key video frame nodes and the set of key sentence nodes .

[0199] S264, calculating a joint loss function according to the node importance score prediction result, the set of inter-modal contrastive sample triplets and the set of intra-modal contrastive sample triplets;

[0200] S265, back-propagating the joint loss function to optimize model parameters of the multi-modal summary generation model.

[0201] Specifically, the joint loss function is calculated according to a semantic enhanced directed graph . for back-propagation to optimize model parameters of the multi-modal summary generation model, i.e., performing linear transformation (using a multi-layer perception, MLP) on the multi-modal unified representation to obtain a node importance score prediction result . After arranging scores in the node importance score prediction result in descending order, the first video frame nodes and the first sentence nodes are selected respectively, to form a set of key video frame nodes and a set of key sentence nodes , 、 to construct a set of inter-modal contrastive sample triplets and a set of intra-modal contrastive sample triplets.

[0202] In an embodiment, a semantic enhanced directed graph is constructed by restarting random walk based on an optimized heterogeneous graph, specifically as follows:

[0203] S251, normalizing each column in the adjacency representation , i.e., setting a sum of elements in each column in the adjacency representation to 1, to obtain a random walk transition matrix .

[0204] S252, setting an initial random walk probability matrix as an identity matrix;

[0205] S253, according to the preset restart probability , random walk transition matrix , initial random walk probability matrix and the random walk probability matrix of the last iteration random walk probability matrix of the current iteration, the random walk probability matrix of the first iteration is updated as follows:

[0206]

[0207] wherein, is the random walk probability matrix of the first iteration, is the random walk probability matrix of the first iteration, , is the total number of iterations;

[0208] S254, set , return to step S253 until the random walk converges or the preset number of iterations is reached, and obtain the random walk probability matrix of the first iteration as the probability distribution matrix;

[0209] S255, split the random walk probability matrix of the first iteration into four sub-matrices according to the node connection type, and the elements in the sub-matrices represent semantic alignment relationships; S256, determine the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship of each node according to the four split sub-matrices, that is, select different modal nodes with the largest corresponding node semantic alignment relationship to generate the inter-modal semantic alignment relationship, and select the same modal nodes with the largest corresponding node semantic alignment relationship to generate the intra-modal semantic alignment relationship, and the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship are both represented as 1;

[0210] S257, take the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship of each node as the elements of the corresponding positions in the edge set

[0211] , and other position elements are represented as 0, and construct a semantic enhanced directed graph .

[0212] Specifically, based on the optimized heterogeneous graph , restart random walk is performed to construct a semantic enhanced directed graph , specifically as follows:

[0213] According to column normalization, the adjacency representation is normalized, that is, the adjacency representation ​​The sum of the elements in each column is 1, and the random walk transfer matrix is ​​obtained ;

[0214] Set the initial random walk probability matrix is the identity matrix;

[0215] Based on the preset restart probability , random walk transfer matrix , initial random walk probability matrix The random walk probability matrix of the previous iteration is used to update the random walk probability matrix of the current iteration, then The random walk probability matrix of the iteration The update formula is as follows:

[0216]

[0217] in, For the The random walk probability matrix of the iteration, For the The random walk probability matrix of the iteration, , is the total number of iterations;

[0218] Place , return to step S253, until the random walk is restarted and converges or the preset number of iterations is reached, and the first The random walk probability matrix of the iteration As a probability distribution matrix, The last iteration number when the restart random walk converges or reaches the preset number of iterations;

[0219] The first The random walk probability matrix of the iteration Split by node connection type to obtain four sub-matrices:

[0220]

[0221] in, is the semantic alignment relationship between video frame nodes, is the semantic alignment relationship between video frame nodes and sentence nodes, is the semantic alignment relationship between sentence nodes and video frame nodes, is the semantic alignment relationship between sentence nodes;

[0222] Determine the inter-modal semantic alignment relationship and intra-modal semantic alignment relationship of each node, that is, select the different modal nodes with the largest corresponding node semantic alignment relationship to generate the inter-modal semantic alignment relationship, and select the same modal nodes with the largest corresponding node semantic alignment relationship to generate the intra-modal semantic alignment relationship. The inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship are both represented as 1. Specifically, the following is for the first Node description:

[0223] 1) No. The nodes are video frame nodes:

[0224] Select and The node with the largest semantic alignment relationship , , ,and for Middle Node to The node where the semantic alignment relationship between nodes takes the maximum value is the node The semantic alignment relationship between the modalities of nodes is expressed as At the same time, select The node with the largest semantic alignment relationship , , ,and for Middle Node to The node where the semantic alignment relationship between nodes takes the maximum value is the node The intra-modal alignment relationship of nodes is expressed as .

[0225] 2) No. The nodes are sentence nodes:

[0226] Select and The node with the largest semantic alignment relationship , , ,and for Middle Node to The node where the semantic alignment relationship between nodes takes the maximum value is the node The semantic alignment relationship between the modalities of nodes is expressed as At the same time, select The node with the largest semantic alignment relationship , , ,and for Middle Node to The node where the semantic alignment relationship between nodes takes the maximum value is the node The intra-modal alignment relationship of nodes is expressed as .

[0227] Based on the above four types of semantic alignment relationships Edge Set The elements of the corresponding nodes in the graph are 0, and the elements of other nodes are 0, constructing a semantically enhanced directed graph. . That is, the edge set The element in the string is 1, which means that the corresponding nodes are semantically aligned. Otherwise, the corresponding nodes are semantically misaligned.

[0228] In one embodiment, the inter-modality contrast sample triplet set includes a video frame inter-modality contrast sample triplet set. Comparison sample triples between sentence modalities ,in:

[0229] 1) Video frame inter-modality comparison sample triplet set , obtained as follows:

[0230] Key video frame node collection As a collection of video anchors;

[0231] For the key sentence node set The nodes in the semantically enhanced directed graph The corresponding edge set The video frame node when the element is 1 is added to the video positive sample set , including elements;

[0232] From the video frame node collection Remove the video anchor set and the video positive sample set , the remaining Nodes constitute the video negative sample set ;

[0233] Construct a set of video frame inter-modal comparison sample triplets ;

[0234] 2) Sentence modality comparison sample triple set , obtained as follows:

[0235] Key sentence node set As a set of sentence anchors;

[0236] Key video frame node collection The nodes in the semantically enhanced directed graph The corresponding edge set The sentence node with the element of 1 is added to the sentence positive sample set , including elements;

[0237] From the sentence node set Remove sentence anchor set and the sentence positive sample set , the remaining Nodes constitute the sentence negative sample set ;

[0238] Construct a set of sentence modality comparison sample triples ;

[0239] The intra-modal contrast sample triplet set includes the intra-modal contrast sample triplet set of the video frame Compare the sample triples set within the sentence modality ,in:

[0240] 1) Set of contrast sample triplets within the video frame modality , obtained as follows:

[0241] In the key video frame node collection Take any node as the video anchor sample;

[0242] For video anchor samples, the semantic enhancement directed graph The corresponding edge set The video frame node when the element of is 1 is regarded as the positive video sample;

[0243] From the key video frame node collection After removing the video anchor sample, a video frame node is randomly selected as the video negative sample to form a video comparison triplet consisting of the video anchor sample, the video positive sample and the video negative sample;

[0244] Traverse the key video frame node collection form Video comparison triplets, all video comparison triplets constitute the set of comparison sample triplets within the video frame modality ;

[0245] 2) Set of contrast sample triples within sentence modality , obtained as follows:

[0246] In the key sentence node set Take any node as the sentence anchor sample;

[0247] For sentence anchor samples, the semantic enhancement directed graph The corresponding edge set The sentence node when the element of is 1 is regarded as a positive sentence sample;

[0248] From the key sentence node set After removing the sentence anchor sample, a sentence node is randomly selected as the sentence negative sample to form a sentence comparison triple consisting of the sentence anchor sample, sentence positive sample and sentence negative sample;

[0249] Traverse the key sentence node set form Sentence comparison triples, all sentence comparison triples constitute the set of comparison sample triples within the sentence modality .

[0250] Specifically, the video comparison triplet consisting of video anchor samples, video positive samples and video negative samples is expressed as ,in, For the The video anchor samples corresponding to the nodes, , For the The video positive samples corresponding to the nodes are , For the The video negative samples corresponding to the nodes, , the sentence contrast triple consisting of sentence anchor sample, sentence positive sample and sentence negative sample is expressed as ,in, For the The sentence anchor samples corresponding to the nodes, , For the The sentence positive sample corresponding to the node , For the Negative sentence samples corresponding to nodes, , .

[0251] In one embodiment, a joint loss function is calculated based on the node importance score prediction results, the inter-modality comparison sample triple set, and the intra-modality comparison sample triple set, as follows:

[0252] S2631, based on the node importance score prediction results, the Focal Loss loss function is used to calculate the classification loss ;

[0253] S2632, based on the video frame inter-modality comparison sample triplet set Comparison sample triples between sentence modalities The Triplet Loss loss function is used to calculate the visual modality consistency loss. and text modality consistency loss ;

[0254] S2633, respectively based on the contrast sample triplet set within the video frame modality Compare the sample triples set within the sentence modality The Triplet Loss loss function is used to calculate the diversity loss within the visual modality and intra-sentence modality diversity loss ;

[0255] S2634. Obtain joint loss function :

[0256]

[0257] in, is the visual modality consistency loss The weight of is the text modality consistency loss The weight of is the intra-visual modality diversity loss The weight of is the sentence modality diversity loss The weight of .

[0258] Specifically, the joint classification loss, inter-modality consistency loss and intra-modality diversity loss generate a joint loss function , as follows:

[0259] Prediction results based on node importance scores Use Focal Loss loss function to calculate classification loss :

[0260]

[0261] in, Indicates the The gold standard label of each node, Indicates the A node is a summary video frame or a summary sentence, Indicates the nodes are not summary video frames or summary sentences, 、 is a hyperparameter, , , Prediction results for node importance scores Middle The score of nodes.

[0262] respectively based on the inter-modal contrastive sample triplet set of video frames and the inter-modal contrastive sample triplet set of sentences The visual modality consistency loss and the text modality consistency loss are respectively calculated by using a Triplet Loss loss function :

[0263]

[0264]

[0265] wherein, is a feature value of the i-th node in the multi-modal unified representation , is a feature value of the i-th node in the multi-modal unified representation , is a feature value of the i-th node in the multi-modal unified representation , is a feature value of the i-th node in the multi-modal unified representation , and , , , is a Euclidean distance function, represents a minimum distance of differentiation between preset positive and negative samples.

[0266] respectively based on the intra-modal contrastive sample triplet set of video frames and the intra-modal contrastive sample triplet set of sentences The intra-modal diversity loss of the visual modality and the intra-modal diversity loss of the text modality are respectively calculated by using a Triplet Loss loss function :

[0267]

[0268]

[0269] obtaining a joint loss function :

[0270]

[0271] wherein, is a weight of the visual modality consistency loss , is a weight of the text modality consistency loss , is a weight of the intra-modal diversity loss of the visual modality ​​the weight of the sentence modality diversity loss the weight of the sentence modality diversity loss the weight of the sentence modality diversity loss, the embodiment , , , are 0.05, 0.03, 0.02, and 0.01, respectively. The model parameters are optimized to improve the quality of the generated summary. The visual modality is the image modality.

[0272] S3, the multi-modal unified representation output by the trained multi-modal summary generation model is used to filter out a set of key video frame nodes and a set of key sentence nodes , and the connection relationships corresponding thereto are reserved in the optimized heterogeneous graph as sub-adjacency matrices , to obtain a multi-modal summary graph as the teaching video summary generation result.

[0273] When the training set is trained, the multi-modal unified representation output by the trained multi-modal summary generation model is used to filter out a set of key video frame nodes and a set of key sentence nodes from the entire node set , and the connection relationships corresponding thereto are reserved in the optimized heterogeneous graph as sub-adjacency matrices , , to obtain a multi-modal summary graph , which is the final generated teaching video summary generation result.

[0274] ​​The effectiveness of the method of the present application is verified on the VT-SSum dataset. The VT-SSum dataset is derived from large-scale teaching video resources, covers various courses and topics, and provides high-quality transcription text and summary annotations, and is widely used in the research and evaluation of automatic summary of teaching videos. On the VT-SSum dataset, the method of the present application is compared with A2Summ, TaleSumm two multi-modal summary methods, and MGL, VideoSAGE two video learning methods based on graph representation, wherein A2Summ reference: B. He, J. Wang, J. Qiu, T. Bui, A. Shrivastava, and Z. Wang, “Align and attend: Multimodal summarization with dual contrastive losses,” in CVPR, 2023, pp. 14 867–14 878.; TaleSumm reference: A. K. Singh, D. Srivastava, and M. Tapaswi, “Previously on... from recaps to story summarization,” in CVPR, 2024, pp. 13 635–13 646.; MGL reference: J. Xie, Z. Zhao, Z. Lin, and Y. Shen, “Multimodal graph learning for cross-modal retrieval,” in SDM, 2023, pp. 145–153.; VideoSAGE reference: J. M. R. Chaves and S. Tripathi, “Videosage: Video summarization with graph representation learning,” in CVPR, 2024, pp. 2527–2534. The experimental results show that the method of the present application has excellent performance in the accuracy, content richness and semantic consistency of the summary. Taking a lecture video (numbered “22gipsfuw7sfapbpjtl3jg4vankygwyk” in the VT-SSum dataset) introducing the English vocabulary database WordNet in the VT-SSum dataset as an example, as shown in Figure 4 the left image in the middle is a comparison chart of the frame-by-frame summary generation results of the method of the present application and different existing methods, Figure 4 the right image in the middle is a comparison chart of the sentence-by-sentence summary generation results of the method of the present application and different existing methods.Figure 4 The middle right graph is a comparison chart of the sentence-by-sentence summary generation results of the method of the present application and different existing methods, wherein, Figure 4 The middle left graph and Figure 4 In the middle right graph, the black broken line represents the gold standard label of the summary video frame (corresponding to the left graph) or the summary sentence (corresponding to the right graph), the green broken line represents the score of the summary video frame or the summary sentence predicted by the method of the present application (i.e. the score corresponding to the predicted node importance score in the prediction result), and the broken lines of other colors represent the scores of the summary video frame or the summary sentence predicted by the comparison methods, such as the yellow broken line representing the score of the summary video frame or the summary sentence predicted by the A2Summ method, the red broken line representing the score of the summary video frame or the summary sentence predicted by the TaleSumm method, the purple broken line representing the score of the summary video frame or the summary sentence predicted by the MGL method, and the cyan broken line representing the score of the summary video frame or the summary sentence predicted by the VideoSAGE method. As can be seen, the method of the present application is superior to the comparison methods in fitting the summary result label broken line, and the prediction curves of different modalities show a highly similar change trend, maintaining the same key segments and key information at the semantic level, Figure 4 proving its advantages in summary extraction accuracy and multi-modal alignment capability. The results show that the summary generated by the method of the present application is significantly better than the comparison methods in terms of hit rate and diversity, verifying the effectiveness and superiority of the method of the present application in the teaching video summary task.

[0275] The method effectively integrates the cross-modal deep semantic relationship and the intra-modal diversity information into the summary generation process through the heterogeneous graph modeling and semantic enhancement mechanism, thereby improving the semantic alignment degree and content richness of the summary, and finally obtaining a summary with higher accuracy and representativeness. Specifically, video frame sequences and sentence sequences are obtained from the input teaching video to form corresponding nodes, and an adjacency matrix is initialized to construct an initial heterogeneous graph; then the connection relationship between the nodes is optimized to form an optimized heterogeneous graph, so that the semantic connection within and between modalities is more accurate; then a two-stage fusion strategy is performed on the optimized heterogeneous graph, first the shallow feature fusion is realized by using the global attention mechanism, and then the deep semantic interaction is realized by subgraph propagation, thereby obtaining a unified multi-modal representation; then the semantic alignment relationship between cross-modal and intra-modal is inferred by using the restart random walk, and a semantic enhanced directed graph is constructed; on this basis, key node screening and comparison sample sampling are performed, and joint optimization is performed in combination with the classification loss, consistency loss and diversity loss; finally, a summary graph containing key information is generated, and a multi-modal summary graph is obtained as the final generated teaching video summary generation result. That is, the method can generate a teaching video summary that is semantically consistent and content-rich, has strong generalization ability, and is especially suitable for online education and smart classroom scenarios.

[0276] Any technical features in the above-described embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations are described, however, any combination of the technical features is considered to be within the scope of the present disclosure.

[0277] The above-described embodiments are merely illustrative and exemplary, and are not to be interpreted in a limiting manner. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, and these are all within the scope of the present application. Therefore, the scope of the present application should be subject to the appended claims.

Claims

1.A method for generating a multi-modal teaching video summary based on a heterogeneous graph, characterized in that: The method comprises the following steps: S1, obtaining a plurality of video samples to form a training set, wherein the video samples comprise teaching videos, and corresponding sampled video frame sequences and sentence sequences; S2, establishing a multi-modal summary generation model and training the model using the training set, wherein the multi-modal summary generation model performs the following operations: S21, inputting the video frame sequences into a visual feature extraction model to obtain a set of visual feature vectors, and inputting the sentence sequences into a language model to obtain a set of text feature vectors, and splicing the set of visual feature vectors and the set of text feature vectors to form a multi-modal feature representation; S22, regarding each video frame in the video frame sequences and each sentence in the sentence sequences as a node, and initializing an adjacency matrix by allowing any nodes to have a connection relationship; S23, respectively calculate the inter-node modal intra-connection relationship and the inter-modal time alignment relationship, and after corresponding construction of the inter-modal constraint matrix and the inter-modal constraint matrix, perform Hadamard product of the three to form the adjacency representation , obtain the optimized heterogeneous graph , is the set of all nodes S24, performing a two-stage fusion strategy, that is, obtaining a shallow fusion representation of the multi-modal feature representation through a global attention mechanism, and then inputting the shallow fusion representation and an optimized heterogeneous graph into a double-layer graph attention network to obtain a multi-modal unified representation; S3, filtering out a key video frame node set from the multi-modal unified representation output by the trained multi-modal summary generation model and a key sentence node set , and corresponding connection relationships reserved in the optimized heterogeneous graph as a sub-adjacency matrix , obtaining a multi-modal summary graph as a teaching video summary generation result. 2.The method of claim 1, wherein: The visual feature extraction model is a pre-trained ResNet model or a pre-trained ViT model, and the language model is a pre-trained BERT model or a pre-trained RoBERTa model. 3.The method of claim 1, wherein: The intra-modal connection relationship and the inter-modal time alignment relationship between the nodes are calculated respectively, and an intra-modal constraint matrix and an inter-modal constraint matrix are constructed correspondingly, specifically as follows: S231, traversing all node sets to extract node pairs; S232, when the nodes in the node pair are of the same modality, calculating the cosine similarity of the corresponding nodes to determine the intra-modal connection relationship, and when the nodes in the node pair are of different modalities, calculating the inter-modal time alignment relationship of the corresponding nodes; S233, obtaining an intra-modal constraint matrix and an inter-modal constraint matrix wherein the intra-modal constraint matrix has elements in the upper-left corner region and the lower-right corner region represented by the intra-modal connection relationship of the corresponding node pair, elements in the lower-left corner region and the upper-right corner region are all set to 0, and the inter-modal constraint matrix has elements in the lower-left corner region and the upper-right corner region represented by the inter-modal time alignment relationship of the corresponding node pair, elements in the upper-left corner region and the lower-right corner region are all set to 0, wherein is the total number of video frames in the video frame sequence, is the total number of sentences in the sentence sequence. 4.The method of claim 3, wherein: When the nodes in the node pair are of the same modality, the cosine similarity of the corresponding nodes is calculated to determine the intra-modal connection relationship, and when the nodes in the node pair are of different modalities, the inter-modal time alignment relationship of the corresponding nodes is calculated, specifically as follows: 1) when the nodes in the node pair are both sentence nodes, the following operations are performed: obtaining a pair of text feature vectors corresponding to the node pair; calculating the cosine similarity between the text feature vectors in the pair of text feature vectors; constraining the intra-modal connection relationship as the inter-sentence connection relationship of the node pair according to a first preset threshold, that is, when the cosine similarity between the text feature vectors is greater than the first preset threshold, the inter-sentence connection relationship of the node pair is represented as 1, otherwise, as 0; 2) when the nodes in the node pair are both video frame nodes, the following operations are performed: obtaining a pair of visual feature vectors corresponding to the node pair; calculating the cosine similarity between the visual feature vectors in the pair of visual feature vectors; constraining the intra-modal connection relationship as the inter-video frame connection relationship of the node pair according to a second preset threshold, that is, when the cosine similarity between the visual feature vectors is greater than the second preset threshold, the inter-sentence connection relationship of the node pair is represented as 1, otherwise, as 0; 3) when the nodes in the node pair are a video frame node and a sentence node respectively, the following operations are performed: inter-modal time alignment relationships between the pairs of computing nodes, the inter-modal time alignment relationships including inter-modal time alignment relationships from video frame nodes to sentence nodes and inter-modal time alignment relationships from sentence nodes to video frame nodes, i.e., when time windows of the video frame nodes and the sentence nodes are consistent, the inter-modal time alignment relationships from the video frame nodes to the sentence nodes and the inter-modal time alignment relationships from the sentence nodes to the video frame nodes are represented as 1, otherwise, represented as 0. 5.The isomorphic graph based multi-modal teaching video summarization generation method of claim 1, wherein: The shallow fusion representation of the multi-modal feature representation is obtained through a global attention mechanism, and the specific process is as follows: S2411, calculating a query matrix, a key matrix and a value matrix according to the multi-modal feature representation; S2412, calculating an attention weight matrix according to the query matrix, the key matrix and the adjacency representation; S2413, multiplying the attention weight matrix and the value matrix to obtain a first updated feature, and performing residual connection between the multi-modal feature representation and the first updated feature after multiplying the multi-modal feature representation with a first learnable linear transformation matrix, to obtain a shallow fusion representation; The multi-modal unified representation is obtained by inputting the shallow fusion representation and the optimized heterogeneous graph into a double-layer graph attention network, and the specific process is as follows: S2421, splitting the optimized heterogeneous graph into four sub-graphs according to node connection types; S2422, inputting the shallow fusion representation into the double-layer graph attention network to propagate and aggregate node features on each sub-graph to obtain a multi-modal unified representation. 6.The method for generating a multi-modal teaching video abstract based on a heterogeneous graph according to claim 5, wherein: The double-layer graph attention network includes a first graph attention module and a second graph attention module connected in sequence, and each graph attention module performs the following operations: The input features of the corresponding graph attention module are propagated through a graph attention layer on each sub-graph to obtain updated features, and the updated features of all nodes of the corresponding sub-graph are spliced to generate an updated feature matrix of the corresponding sub-graph; The updated feature matrix of the first sub-graph and the updated feature matrix of the third sub-graph are spliced and multiplied by a first learnable weight matrix to obtain a video frame node representation, and the updated feature matrix of the second sub-graph and the updated feature matrix of the fourth sub-graph are spliced and multiplied by a second learnable weight matrix to obtain a sentence node representation; The video frame node representation and the sentence node representation are spliced to obtain the output features of the corresponding graph attention module. 7.The isomorphic graph based multi-modal teaching video summarization generation method of claim 1, wherein: The multi-modal summary generation model further performs the following operations: S25, performing restart random walk based on the optimized heterogeneous graph to construct a semantic enhanced directed graph; S26, calculating a joint loss function based on the semantic enhanced directed graph for back propagation to optimize model parameters of the multi-modal summary generation model, and the specific process is as follows: S261, performing linear transformation on the multi-modal unified representation to obtain node importance score prediction results; S262, after arranging the scores in the node importance score prediction results in descending order, select the top scores according to the scores after descending order. video frame nodes and the previous sentence nodes, corresponding to the key video frame node set and key sentence node set , 、 , is the total number of video frames in the video frame sequence, is the total number of sentences in the sentence sequence; S263、according to the key video frame node set and the key sentence node set constructing the inter-modal contrast sample triple set and the intra-modal contrast sample triple set; S264, calculating a joint loss function according to the node importance score prediction results, a set of inter-modal contrast sample triplets and a set of intra-modal contrast sample triplets; S265, optimizing the model parameters of the multi-modal summary generation model by back propagating the joint loss function. 8.The method of claim 7, wherein: The restart random walk based on the optimized heterogeneous graph is performed to construct a semantic enhanced directed graph, and the specific process is as follows: S251、to the adjacent representation normalizing by column, i.e. letting the sum of elements in each column of the adjacent representation to obtain a random walk transition matrix ; S252, set initial random walk probability matrix is the identity matrix; S253、according to the preset restart probability , random walk transition matrix , initial random walk probability matrix and the random walk probability matrix of the last iteration, the random walk probability matrix of the current iteration is updated, then the random walk probability matrix of the first iteration is updated The update formula is as follows: ； wherein, is the first iteration of the random walk probability matrix, is the first iteration of the random walk probability matrix, is the first iteration of the random walk probability matrix, is the first iteration of the random walk probability matrix, , is the total number of iterations; S254, set , return to step S253 until the random walk convergence is restarted or the preset iteration number is reached, and the random walk probability matrix of the iteration is obtained as the probability distribution matrix. ​ S255、per node connection type the first random walk probability matrix of the second iteration split into four sub-matrices, the elements in the sub-matrices representing semantic alignment relationships; S256、determine the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship of each node according to the four split sub-matrices, that is, select the different modal nodes with the largest corresponding node semantic alignment relationship to generate the inter-modal semantic alignment relationship, and select the same modal nodes with the largest corresponding node semantic alignment relationship to generate the intra-modal semantic alignment relationship, wherein the inter-modal semantic alignment relationship and the intra-modal semantic alignment relationship are both represented as 1; S257, take the inter-modal semantic alignment relationship and intra-modal semantic alignment relationship of each node as the edge set The elements at the corresponding positions in the graph are represented as 0, and the elements at other positions are represented as 0 to construct a semantically enhanced directed graph. . 9.The isomorphic graph based multi-modal teaching video summarization generation method of claim 7, wherein: The inter-modal contrastive sample triple set includes an inter-video frame modal contrastive sample triple set and an inter-sentence modal contrastive sample triple set wherein: 1) a set of video frame inter-modal contrastive sample triplets are obtained as follows: A set of key video frame nodes as a set of video anchor points; For the key sentence node set The nodes in the semantically enhanced directed graph The corresponding edge set The video frame node when the element is 1 is added to the video positive sample set , including elements; remove the video anchor node set from the video frame node set ;​​​​ Constructing a set of video frame inter-modal contrast sample triplets ; 2) Sentence inter-modal contrast sample triple set , is obtained as follows: with a set of key sentence nodes as a set of sentence anchor points; Nodes in a set of key video frame nodes are added to a set of sentence positive samples when elements of a corresponding edge set in a semantic-enhanced directed graph are 1 , and the set of sentence positive samples contains elements in total. remove the sentence anchor set from the sentence node set remove the sentence anchor set from the sentence node set remove the sentence anchor set from the sentence node set , the remaining nodes form the sentence negative sample set ; constructing sentence inter-modal contrast sample triple set ; The intra-modality contrastive sample triple set includes an intra-video frame modality contrastive sample triple set and an intra-sentence modality contrastive sample triple set wherein: 1) Set of contrast sample triplets within the video frame modality , obtained as follows: In a set of key video frame nodes any node in the set as a video anchor sample; Video anchor point samples, semantic-enhanced directed graphs with elements of 1 in the middle corresponding edge set Video frame nodes with elements of 1 in the middle corresponding edge set as video positive samples; Removing video anchor point samples from a set of key video frame nodes Randomly selecting one video frame node as a video negative sample after removing the video anchor point samples from the set of key video frame nodes, to form a video comparison triple consisting of a video anchor point sample, a video positive sample, and a video negative sample; Traversing a set of key video frame nodes Forming a video contrast triple, all video contrast triples forming a set of video frame intra-modal contrast sample triples ; 2) a set of sentence-in-modal contrastive sample triplets are obtained as follows: In the key sentence node set Take any node as a sentence anchor point sample For sentence anchor samples, the semantic enhancement directed graph The corresponding edge set The sentence node when the element of is 1 is regarded as a positive sentence sample; remove sentence anchor point samples from the key sentence node set After removing the sentence anchor point samples from the key sentence node set, a sentence node is randomly selected as a sentence negative sample, forming a sentence comparison triple consisting of a sentence anchor point sample, a sentence positive sample, and a sentence negative sample. traversing the key sentence node set forming a sentence pair comparison triple, all the sentence pair comparison triples form a sentence intra-modal comparison sample triple set . 10.The method of claim 7, wherein: The joint loss function is calculated according to the node importance score prediction result, the inter-modal contrast sample triple set and the intra-modal contrast sample triple set, and specifically as follows: S2631, the classification loss is calculated by using a Focal Loss loss function based on the node importance score prediction result ; S2632, respectively based on the video frame inter-modal contrast sample triple set and the sentence inter-modal contrast sample triple set The Triplet Loss loss function is used to calculate the visual modal consistency loss and the text modal consistency loss ; S2633, respectively based on intra-video frame modality contrastive sample triplet sets and intra-sentence modality contrastive sample triplet sets Triplet Loss loss function is used to calculate the intra-visual modality diversity loss and intra-sentence modality diversity loss ; S2634, obtaining a joint loss function : ； in, is the visual modality consistency loss The weight of is the text modality consistency loss The weight of is the intra-visual modality diversity loss The weight of is the sentence modality diversity loss The weight of .

Images