Video abstract generation method and system

Abstract

The invention relates to the field of video processing, in particular to a video abstract generation method and system, and the method specifically comprises the steps: extracting an I frame in a video, extracting an audio between the current I frame and the next I frame, building a corresponding relation between the current I frame and the audio, and extracting the features of the I frame and the audio; constructing a visual hyperedge according to the I frame feature similarity and the occurrence time of the I frame in the video, and generating the weight of the visual hyperedge by using the video text abstract; constructing an audio hyperedge based on the audio feature similarity and the occurrence time of the audio in the video, and generating the weight of the audio hyperedge by using the video text abstract; constructing a hypergraph through an audio hyperedge and a visual hyperedge, wherein nodes of the hypergraph are I frames; and obtaining a key frame based on the video text abstract and the hypergraph, and generating a video abstract by using the key frame.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a video summary generation method and system. Background Art

[0002] Video summarization technology is to extract the core content of the video through an automated method to generate concise and informative short videos, so that users can quickly obtain the key information of the video. Traditional video summarization methods are mainly divided into content-based summarization and structure-based summarization. Content-based summarization methods generate summaries by analyzing the visual content of the video, while structure-based summarization methods rely more on the semantic information of the video, such as objects, events, plot development, etc. However, traditional methods are often limited to the processing of simple visual features or speech text information, and it is difficult to capture higher-order and multi-dimensional information relationships in the video, especially when facing complex scenes and multi-task videos. The intrinsic information of the video has multi-level and multi-dimensional characteristics. For example, a video not only contains image information, but also involves multiple modalities such as time and sound, and there are often complex multiple relationships between these information. How to extract key information from the video and effectively present it to users is of great significance for the promotion of the video. Summary of the invention

[0003] In order to extract a video summary by utilizing multiple modal information in a video, in a first aspect of the present invention, a video summary generation method is provided, the method comprising: Extract the I frame in the video, extract the audio between the current I frame and the next I frame, establish the corresponding relationship between the current I frame and the audio, and extract the features of the I frame and the audio; A visual hyperedge is constructed based on the similarity of I-frame features and the time when the I-frame appears in the video, and the weight of the visual hyperedge is generated using the video text summary; an audio hyperedge is constructed based on the similarity of audio features and the time when the audio appears in the video, and the weight of the audio hyperedge is generated using the video text summary; a hypergraph is constructed through the audio hyperedge and the visual hyperedge, and the nodes of the hypergraph are I-frames; Key frames are obtained based on the video text summary and the hypergraph, and the video summary is generated using the key frames.

[0004] Preferably, the weight of generating the visual hyperedge by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each visual hyperedge, calculate the average of the features of all I frames it contains to obtain the average vector of the hyperedge; The encoding vector of the sentence or keyword and the average vector of the hyperedge are input into the trained neural network, the output value of the neural network is used as the similarity between the hyperedge and the sentence or keyword, and the maximum value of the similarity between the hyperedge and all sentences or keywords is used as the weight of the visual hyperedge.

[0005] Preferably, the weight of the audio hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each audio hyperedge, extract the fusion features of the corresponding audio features of all I frames it contains; The similarity between the fused feature and the encoding vector of each sentence or keyword is calculated, and the maximum similarity is used as the weight of the audio hyperedge.

[0006] Preferably, the step of obtaining key frames based on the video text summary and the hypergraph is as follows: Calculate all hyperedges to which the hypergraph node belongs, and take the sum of the weights of all hyperedges as the centrality score of the node; Perform topic modeling on the video text summary and calculate the relevance of the audio features corresponding to the nodes with each topic; The importance of the hypergraph nodes is obtained using the centrality score and the maximum correlation, and the key frames are obtained based on the importance.

[0007] Preferably, obtaining the key frame according to the importance is specifically as follows: Divide all I frames of the video into multiple intervals in sequence; Determine the hyperedge set included in the interval according to the time relationship, calculate the average value of the hyperedge weights in the hyperedge set, determine the range to which the average value of the hyperedge weights belongs, and use the number of key frames corresponding to the range as the number of key frames in the interval; The I frames with the greatest number of key frames of importance are selected from the sorted I frames as key frames.

[0008] In a second aspect of the present invention, a video summary generation system is provided, the system comprising: The feature extraction module is used to extract the I frame in the video, extract the audio between the current I frame and the next I frame, establish the corresponding relationship between the current I frame and the audio, and extract the features of the I frame and the audio; A hypergraph construction module is used to construct a visual hyperedge based on the similarity of I-frame features and the time when the I-frame appears in the video, and generate the weight of the visual hyperedge using the video text summary; construct an audio hyperedge based on the similarity of audio features and the time when the audio appears in the video, and generate the weight of the audio hyperedge using the video text summary; construct a hypergraph through the audio hyperedge and the visual hyperedge, wherein the nodes of the hypergraph are I-frames; The summary generation module is used to obtain key frames based on the video text summary and the hypergraph, and generate a video summary using the key frames.

[0009] Preferably, the weight of generating the visual hyperedge by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each visual hyperedge, calculate the average of the features of all I frames it contains to obtain the average vector of the hyperedge; The encoding vector of the sentence or keyword and the average vector of the hyperedge are input into the trained neural network, the output value of the neural network is used as the similarity between the hyperedge and the sentence or keyword, and the maximum value of the similarity between the hyperedge and all sentences or keywords is used as the weight of the visual hyperedge.

[0010] Preferably, the weight of the audio hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each audio hyperedge, extract the fusion features of the corresponding audio features of all I frames it contains; The similarity between the fused feature and the encoding vector of each sentence or keyword is calculated, and the maximum similarity is used as the weight of the audio hyperedge.

[0011] Preferably, the step of obtaining key frames based on the video text summary and the hypergraph is as follows: Calculate all hyperedges to which the hypergraph node belongs, and take the sum of the weights of all hyperedges as the centrality score of the node; Perform topic modeling on the video text summary and calculate the relevance of the audio features corresponding to the nodes with each topic; The importance of the hypergraph nodes is obtained using the centrality score and the maximum correlation, and the key frames are obtained based on the importance.

[0012] Preferably, obtaining the key frame according to the importance is specifically as follows: Divide all I frames of the video into multiple intervals in sequence; Determine the hyperedge set included in the interval according to the time relationship, calculate the average value of the hyperedge weights in the hyperedge set, determine the range to which the average value of the hyperedge weights belongs, and use the number of key frames corresponding to the range as the number of key frames in the interval; The I frames with the greatest number of key frames of importance are selected from the sorted I frames as key frames.

[0013] The video summary generation method provided by the present invention integrates the visual information and audio information of the video, and constructs a hypergraph of I frames through the visual information and audio information, and uses the video text summary to generate the weight of the hyperedge, so as to better reflect the dynamic changes and internal connections of the video content, effectively use the structural information of the hypergraph and the semantic information of the text summary, select the key frames that can summarize the video content and highlight the text theme, and then obtain a more accurate video summary. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is a schematic diagram of the process of embodiment 1; Figure 2 A schematic diagram of extracting I frames and audio from a video is shown; Figure 3 A schematic diagram of constructing an I frame set using a sliding window is shown; Figure 4 is a schematic diagram of a hypergraph; Figure 5 A structural diagram of another embodiment. DETAILED DESCRIPTION

[0014] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0015] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of the embodiments of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0016] Embodiment 1 provides a method for generating a video summary. Figure 1 As shown, the method includes: S1, extract the I frame in the video, and extract the audio between the current I frame and the next I frame, establish the corresponding relationship between the current I frame and the audio, and extract the features of the I frame and the audio; Videos are composed of a series of continuous image frames, including I frames, P frames, and B frames. I frames contain complete image information, while P frames and B frames both require I frames to restore the complete image. Changes in I frames usually correspond to changes in the scene. Extracting I frames from videos can not only reduce the amount of subsequent calculations, but also minimize the omission of video information. In addition to visual information, videos also contain auditory information, namely audio, such as character dialogue. An I frame and the audio clip that follows it are related in content. The audio between the current I frame and the next I frame is used as the audio feature information of the current I frame, such as Figure 2For example, the audio information between the second I frame and the third I frame is the corresponding audio of the second I frame. After obtaining the I frame and the audio corresponding to the I frame, the features of the I frame are extracted by a convolutional neural network, and the features of the audio are extracted by a Mel frequency cepstral coefficient or a spectrogram. In another embodiment, the audio is converted into text, the text is encoded, and the encoding result is used as the audio feature.

[0017] S2, constructing a visual hyperedge based on the similarity of I-frame features and the time when the I-frame appears in the video, and generating the weight of the visual hyperedge using the video text summary; constructing an audio hyperedge based on the similarity of audio features and the time when the audio appears in the video, and generating the weight of the audio hyperedge using the video text summary; constructing a hypergraph through the audio hyperedge and the visual hyperedge, wherein the nodes of the hypergraph are I-frames; In a video, even if the scene is the same, the video content may be different if it appears at different times. In order to prevent inaccurate extraction caused by associating I frames with a long time interval, the present invention constructs a visual hyperedge based on the similarity of I frame features and the time when the I frame appears in the video. The method of constructing a visual hyperedge includes but is not limited to calculating the similarity of I frames based on their features, and then dividing the I frames into multiple sets according to the time when they appear in the video, and connecting the I frames in each set with a similarity greater than a certain value to a hyperedge. In another embodiment, a sliding window method is used to construct an I frame set, that is, the I frames in each sliding window are connected with a hyperedge based on their similarity. Preferably, the length of the sliding window is greater than the sliding step size. Figure 3 A schematic diagram showing the use of a sliding window to construct an I frame set.

[0018] The video text summary is a text description of the video content, which can provide high-level semantic information of the video. In one embodiment, the video text summary is a text describing the video. In another embodiment, if there is no text describing the video, the title of the video is used as the video text summary. For example, a news video, the video text summary is that a 7.0 magnitude earthquake occurred in a certain place early this morning local time, and the epicenter was located in the XX area, which has caused many houses to collapse and some roads to be interrupted. The rescue team has rushed to the disaster area to carry out rescue work. A product evaluation video, the video text summary is that this phone uses the XX processor, with strong performance, excellent photo effects, and particularly outstanding night scene performance. The battery life is average and basically meets daily use.

[0019] Determine whether a visual hyperedge is important based on the content of the text summary. If a scene or event is mentioned in the text summary, and the I frame corresponding to the scene or event is included in a visual hyperedge, then the hyperedge should have a higher weight. For example, if a road interruption is mentioned in the video text summary, and the image content corresponding to the visual hyperedge {I1, I2} is the scene of the road interruption, then the hyperedge should be given a higher weight; while the hyperedge {I3, I4} corresponds to the sky, then the weight of the hyperedge is relatively low.

[0020] In an optional embodiment, the weight of the visual hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each visual hyperedge, calculate the average of the features of all I frames it contains to obtain the average vector of the hyperedge; The encoding vector of the sentence or keyword and the average vector of the hyperedge are input into the trained neural network, the output value of the neural network is used as the similarity between the hyperedge and the sentence or keyword, and the maximum value of the similarity between the hyperedge and all sentences or keywords is used as the weight of the visual hyperedge.

[0021] Preprocess the text summary of the video and decompose it into smaller units, which are separate sentences or keywords. For example, the video text summary "The protagonist talks in the room and then walks outside." can be divided into two sentences: "The protagonist talks in the room" and "Then walks outside." Or extract the keywords "room", "talk", "outdoors". Use text embedding models such as Word2Vec, GloVe, BERT, Sentence-BERT, etc. to convert each sentence or keyword into a vector representation, also known as a word vector or sentence vector. These vectors can capture the semantic information of the text, and similar texts are closer in the vector space. For a visual hyperedge, it contains several I frames, each of which has a previously extracted feature vector. The feature vectors of all I frames in the hyperedge are averaged to obtain the average vector of the hyperedge to represent the visual content of the hyperedge. Pre-train a neural network whose input is the text vector and the visual vector, and the output is the similarity between them. Input the encoding vector of each sentence or keyword and the average vector of each visual hyperedge into the trained neural network, and the neural network will output the similarity. The similarity is a value between 0 and 1, and the larger the value, the more similar it is. If the video text summary is divided into sentences, the encoding vector of each sentence and the average vector of each visual hyperedge are input into the trained neural network; if the video text summary is divided into keywords, the encoding vector of each keyword and the average vector of each visual hyperedge are input into the trained neural network. A visual hyperedge has a similarity with each sentence or keyword. The maximum value of the similarity between this visual hyperedge and all sentences or keywords is found as the weight of this visual hyperedge.

[0022] An audio hyperedge is constructed in the same manner as a visual hyperedge. Specifically, an audio hyperedge is constructed based on the similarity of audio features and the time when the audio appears in the video. The method for constructing an audio hyperedge includes, but is not limited to, calculating the audio similarity based on the audio features, and then dividing the audio into multiple sets according to the time when the audio appears in the video, and connecting the audio in each set with a similarity greater than a certain value to a hyperedge. In another embodiment, an audio set is constructed using a sliding window, that is, the audio in each sliding window is connected with a hyperedge based on the similarity. Preferably, the length of the sliding window is greater than the sliding step size. Since the audio and I frames are one-to-one corresponding, after the hyperedge is constructed for the audio, the audio can be directly replaced with the I frame based on the corresponding relationship between the audio and the I frame, thereby realizing the construction of an audio hyperedge for the I frame. Audio is the content between two I frames, and the content between the two I frames can be retained through the audio hyperedge.

[0023] Similar to visual hyperedges, the importance of an audio hyperedge is determined based on the content of the text summary. If a sound or conversation is mentioned in the text summary, and the I-frame associated with the audio clip corresponding to the sound or conversation is included in an audio hyperedge, then the hyperedge should have a higher weight.

[0024] In an optional embodiment, the weight of the audio hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each audio hyperedge, extract the fusion features of the corresponding audio features of all I frames it contains; The similarity between the fused feature and the encoding vector of each sentence or keyword is calculated, and the maximum similarity is used as the weight of the audio hyperedge.

[0025] Similarly, the text summary of the video is preprocessed and decomposed into smaller units, which are separate sentences or keywords. Use a text embedding model to convert each sentence or keyword into a vector representation. The audio features corresponding to all I frames in the audio hyperedge are fused to obtain the fused features of the audio hyperedge. The fusion method preferably concatenates all audio feature vectors into a longer vector, and then converts them to a certain length through a fully connected layer. The cosine similarity, dot product or fully connected layer is used to calculate the similarity between the fused features of the audio hyperedge and the encoding vector of each sentence or keyword in the text summary. Similar to the visual hyperedge, the maximum value of the similarity between the audio hyperedge and all sentences or keywords is selected as its weight. All I frames are used as nodes, and the constructed visual hyperedges and audio hyperedges are used as hyperedges to form a hypergraph together. The hypergraph not only includes the visual information of the video, but also the auditory information. Figure 4 A schematic diagram of a hypergraph.

[0026] S3, obtaining key frames based on the video text summary and the hypergraph, and generating a video summary using the key frames.

[0027] After the hypergraph is obtained, the key frames are obtained using the video text summary and the hypergraph. In one embodiment, the key frames are obtained based on the video text summary and the hypergraph, specifically: Calculate all hyperedges to which the hypergraph node belongs, and take the sum of the weights of all hyperedges as the centrality score of the node; Perform topic modeling on the video text summary and calculate the relevance of the audio features corresponding to the nodes with each topic; The importance of the hypergraph nodes is obtained using the centrality score and the maximum correlation, and the key frames are obtained based on the importance.

[0028] For each node in the hypergraph, add the weights of all hyperedges containing the node, and the sum is used as the centrality score of the node. If a node belongs to multiple high-weight hyperedges, it means that it is more important. Then, perform topic modeling on the video text summary, and decompose the text summary into several topics. Preferably, LDA (Latent Dirichlet Allocation) is used to perform topic modeling on the video text summary. For example, the constructed topics are: Topic 1 Mobile phone performance (keywords: processor, Snapdragon 8Gen2, performance, powerful), Topic 2 Battery life (keywords: battery, battery life, daily use), etc. Calculate the correlation between the audio features corresponding to each node and each topic. The correlation measures the degree of correlation between the audio content of the node and each topic in the text summary. The correlation score of the topic with the highest correlation with the node is used as the maximum correlation score of the node. Combine the centrality score and the maximum correlation score to calculate the importance of each node. Preferably, the centrality score and the maximum correlation score are multiplied, or they are weighted summed to obtain the importance of the node. Sort all nodes according to the calculated importance score, and select several nodes with the highest importance scores as key frames.

[0029] In one embodiment, obtaining the key frame according to the importance is specifically: Divide all I frames of the video into multiple intervals in sequence; Determine the hyperedge set included in the interval according to the time relationship, calculate the average value of the hyperedge weights in the hyperedge set, determine the range to which the average value of the hyperedge weights belongs, and use the number of key frames corresponding to the range as the number of key frames in the interval; The I frames with the greatest number of key frames of importance are selected from the sorted I frames as key frames.

[0030] All I frames of the video are divided into multiple intervals in chronological order, and the division method includes but is not limited to equal interval division and adaptive division according to the change of video content. After division, each interval contains a certain number of continuous I frames. The set of hyperedges contained in each interval is determined according to the time relationship. Preferably, if the I frame in the interval is in a hyperedge, the hyperedge is added to the hyperedge set of the interval. The average value of these hyperedge weights is calculated, and this average value reflects the overall importance of the video content in the interval, and this average value is mapped to a pre-set range. In one embodiment, several weight ranges are pre-defined, and a corresponding number of key frames is assigned to each range. The larger the weight, the larger the corresponding number of key frames. If the average weight of a certain interval falls into a certain range, the number of key frames corresponding to the range is used as the number of key frames required to be extracted in the interval. In each interval, the I frames are sorted according to the importance of the I frames, and the I frames with the largest number of key frames are selected from the sorted I frames as key frames. The key frames of all intervals are merged to obtain a set of key frames for the entire video. The selected key frames are arranged in their chronological order in the original video and spliced ​​together to form a final video summary. In another optional embodiment, a short transition effect is added between adjacent key frames to make the video summary smooth.

[0031] Embodiment 2 provides a video summary generation system, such as Figure 5 As shown, the system comprises: The feature extraction module is used to extract the I frame in the video, extract the audio between the current I frame and the next I frame, establish the corresponding relationship between the current I frame and the audio, and extract the features of the I frame and the audio; A hypergraph construction module is used to construct a visual hyperedge based on the similarity of I-frame features and the time when the I-frame appears in the video, and generate the weight of the visual hyperedge using the video text summary; construct an audio hyperedge based on the similarity of audio features and the time when the audio appears in the video, and generate the weight of the audio hyperedge using the video text summary; construct a hypergraph through the audio hyperedge and the visual hyperedge, wherein the nodes of the hypergraph are I-frames; The summary generation module is used to obtain key frames based on the video text summary and the hypergraph, and generate a video summary using the key frames.

[0032] Preferably, the weight of generating the visual hyperedge by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each visual hyperedge, calculate the average of the features of all I frames it contains to obtain the average vector of the hyperedge; The encoding vector of the sentence or keyword and the average vector of the hyperedge are input into the trained neural network, the output value of the neural network is used as the similarity between the hyperedge and the sentence or keyword, and the maximum value of the similarity between the hyperedge and all sentences or keywords is used as the weight of the visual hyperedge.

[0033] Preferably, the weight of the audio hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each audio hyperedge, extract the fusion features of the corresponding audio features of all I frames it contains; The similarity between the fused feature and the encoding vector of each sentence or keyword is calculated, and the maximum similarity is used as the weight of the audio hyperedge.

[0034] Preferably, the step of obtaining key frames based on the video text summary and the hypergraph is as follows: Calculate all hyperedges to which the hypergraph node belongs, and take the sum of the weights of all hyperedges as the centrality score of the node; Perform topic modeling on the video text summary and calculate the relevance of the audio features corresponding to the nodes with each topic; The importance of the hypergraph nodes is obtained using the centrality score and the maximum correlation, and the key frames are obtained based on the importance.

[0035] Preferably, obtaining the key frame according to the importance is specifically as follows: Divide all I frames of the video into multiple intervals in sequence; Determine the hyperedge set included in the interval according to the time relationship, calculate the average value of the hyperedge weights in the hyperedge set, determine the range to which the average value of the hyperedge weights belongs, and use the number of key frames corresponding to the range as the number of key frames in the interval; The I frames with the greatest number of key frames of importance are selected from the sorted I frames as key frames.

[0036] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0037] As another aspect, the present invention further provides a computer-readable medium, wherein the computer-readable medium carries one or more programs, and when the one or more programs are executed by a device or a processor, the method described in the first embodiment is implemented.

[0038] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video summary generation method, characterized in that: The method comprises: Extract the I frame in the video, extract the audio between the current I frame and the next I frame, establish the corresponding relationship between the current I frame and the audio, and extract the features of the I frame and the audio; A visual hyperedge is constructed based on the similarity of I-frame features and the time when the I-frame appears in the video, and the weight of the visual hyperedge is generated using the video text summary; an audio hyperedge is constructed based on the similarity of audio features and the time when the audio appears in the video, and the weight of the audio hyperedge is generated using the video text summary; a hypergraph is constructed through the audio hyperedge and the visual hyperedge, and the nodes of the hypergraph are I-frames; Key frames are obtained based on the video text summary and the hypergraph, and the video summary is generated using the key frames.

2. The method according to claim 1, characterized in that The weight of generating the visual hyperedge using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each visual hyperedge, calculate the average of the features of all I frames it contains to obtain the average vector of the hyperedge; The encoding vector of the sentence or keyword and the average vector of the hyperedge are input into the trained neural network, the output value of the neural network is used as the similarity between the hyperedge and the sentence or keyword, and the maximum value of the similarity between the hyperedge and all sentences or keywords is used as the weight of the visual hyperedge.

3. The method according to claim 1, characterized in that The weight of the audio hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each audio hyperedge, extract the fusion features of the corresponding audio features of all I frames it contains; The similarity between the fused feature and the encoding vector of each sentence or keyword is calculated, and the maximum similarity is used as the weight of the audio hyperedge.

4. The method according to claim 1, characterized in that The key frames are obtained based on the video text summary and the hypergraph, specifically: Calculate all hyperedges to which the hypergraph node belongs, and take the sum of the weights of all hyperedges as the centrality score of the node; Perform topic modeling on the video text summary and calculate the relevance of the audio features corresponding to the nodes with each topic; The importance of the hypergraph nodes is obtained using the centrality score and the maximum correlation, and the key frames are obtained based on the importance.

5. The method according to claim 4, characterized in that The key frames are obtained according to the importance, specifically: Divide all I frames of the video into multiple intervals in sequence; Determine the hyperedge set included in the interval according to the time relationship, calculate the average value of the hyperedge weights in the hyperedge set, determine the range to which the average value of the hyperedge weights belongs, and use the number of key frames corresponding to the range as the number of key frames in the interval; The I frames with the greatest number of key frames of importance are selected from the sorted I frames as key frames.

6. A video summary generation system, characterized in that: The system comprises: The feature extraction module is used to extract the I frame in the video, extract the audio between the current I frame and the next I frame, establish the corresponding relationship between the current I frame and the audio, and extract the features of the I frame and the audio; A hypergraph construction module is used to construct a visual hyperedge based on the similarity of I-frame features and the time when the I-frame appears in the video, and generate the weight of the visual hyperedge using the video text summary; construct an audio hyperedge based on the similarity of audio features and the time when the audio appears in the video, and generate the weight of the audio hyperedge using the video text summary; construct a hypergraph through the audio hyperedge and the visual hyperedge, wherein the nodes of the hypergraph are I-frames; The summary generation module is used to obtain key frames based on the video text summary and the hypergraph, and generate a video summary using the key frames.

7. The system according to claim 6, characterized in that The weight of generating the visual hyperedge using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each visual hyperedge, calculate the average of the features of all I frames it contains to obtain the average vector of the hyperedge; The encoding vector of the sentence or keyword and the average vector of the hyperedge are input into the trained neural network, the output value of the neural network is used as the similarity between the hyperedge and the sentence or keyword, and the maximum value of the similarity between the hyperedge and all sentences or keywords is used as the weight of the visual hyperedge.

8. The system according to claim 6, characterized in that The weight of the audio hyperedge generated by using the video text summary is specifically: Segment the video text summary into sentences or keywords, and use the text embedding model to encode each sentence or keyword into a vector; For each audio hyperedge, extract the fusion features of the corresponding audio features of all I frames it contains; The similarity between the fused feature and the encoding vector of each sentence or keyword is calculated, and the maximum similarity is used as the weight of the audio hyperedge.

9. The system according to claim 6, characterized in that The key frames are obtained based on the video text summary and the hypergraph, specifically: Calculate all hyperedges to which the hypergraph node belongs, and take the sum of the weights of all hyperedges as the centrality score of the node; Perform topic modeling on the video text summary and calculate the relevance of the audio features corresponding to the nodes with each topic; The importance of the hypergraph nodes is obtained using the centrality score and the maximum correlation, and the key frames are obtained based on the importance.

10. The system according to claim 9, characterized in that The key frames are obtained according to the importance, specifically: Divide all I frames of the video into multiple intervals in sequence; Determine the hyperedge set included in the interval according to the time relationship, calculate the average value of the hyperedge weights in the hyperedge set, determine the range to which the average value of the hyperedge weights belongs, and use the number of key frames corresponding to the range as the number of key frames in the interval; The I frames with the greatest number of key frames of importance are selected from the sorted I frames as key frames.

Images