Video content extraction method and device, computer equipment and storage medium

Abstract

The invention relates to a video content extraction method and device, computer equipment and a storage medium. The method comprises the following steps: performing frame extraction processing on a downloaded target speech video to generate a continuous image sequence; screening the continuous image sequences after frame extraction, and identifying picture frames containing effective speech contents; inputting the picture frame containing the effective speech content into a multi-modal large model for semantic understanding, and generating structured semantic information; separating an audio track in the target speech video, carrying out voice transcription with a timestamp, generating a voice transcription text, and carrying out retouching processing on the voice transcription text; performing time sequence alignment processing on the timestamps corresponding to the picture frames containing the effective speech content and the timestamps corresponding to voice transcription; and calculating modal weights of the polished voice transcription text and the structured semantic information, and generating a text summary of the target speech video according to the modal weights. By adopting the method, the text summary with higher quality can be generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model, in particular to a video content extraction method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the continuous development of multimedia technology, multimedia content such as PPT (presentation video) plays an increasingly important role in information dissemination and knowledge sharing. However, how to efficiently and accurately extract and summarize the key information of these multimedia content has always been a hot and difficult research topic. The current technical solutions have made some progress in PPT and video content extraction, but there are still many deficiencies.

[0003] Traditional PPT content extraction methods can only extract text, lack of deep understanding of logical semantics and complex structure, leading to information fragmentation, making it difficult to analyze and apply. In video content extraction, the fusion of audio and video information is not good, making the extracted content lack of context support, affecting the accuracy and integrity. In addition, the video text extraction method fails to fully integrate multi-modal information, resulting in incoherent and incomplete text summary content, which cannot meet the user's demand for high-quality text summary.

[0004] Therefore, there is an urgent need for a video content extraction method, device, computer equipment, computer readable storage medium and computer program product that can generate higher quality text summary. SUMMARY

[0005] Therefore, it is necessary to provide a video content extraction method, device, computer equipment, computer readable storage medium and computer program product capable of generating higher quality text summary to solve the above technical problems.

[0006] In a first aspect, the present application provides a video content extraction method, comprising:

[0007] Frame extraction processing is performed on the downloaded target presentation video to generate a continuous image sequence;

[0008] The continuous image sequence after frame extraction is filtered to identify picture frames containing valid presentation content;

[0009] The picture frames containing valid presentation content are input into a multi-modal large model for semantic understanding to generate structured semantic information;

[0010] The audio track in the target presentation video is separated, and the time stamp is attached for speech transcription to generate speech transcription text, and the speech transcription text is polished;

[0011] align the timestamp corresponding to the picture frame containing the valid speech content with the timestamp corresponding to the speech transcription in time sequence;

[0012] calculate the modal weight of the polished speech transcription text and the structured semantic information respectively, and generate a text summary of the target speech video according to the modal weight.

[0013] In one embodiment, the inputting the picture frame containing the valid speech content into the multi-modal large model for semantic understanding to generate structured semantic information comprises:

[0014] inputting the picture frame containing the valid speech content into the multi-modal large model to extract image region features, wherein the image region features include title area, list items and chart area;

[0015] extracting text features from the image region features through a text encoder and a pre-defined text extraction instruction;

[0016] aligning the image region features and the text features through a cross-modal attention mechanism to generate structured semantic information.

[0017] In one embodiment, the aligning the timestamp corresponding to the picture frame containing the valid speech content with the timestamp corresponding to the speech transcription in time sequence comprises:

[0018] extending the timestamp of each picture frame containing valid speech content into a time window;

[0019] matching a first speech transcription text segment with overlapping timestamps in the speech transcription within the time window;

[0020] extracting semantic vectors of the picture frame containing the valid speech content and the speech transcription text respectively using a multi-modal large model, and calculating the cosine similarity between them;

[0021] According to the cosine similarity, the second speech transcription text segment with the highest similarity is selected from the first speech transcription text segment.

[0022] In one embodiment, the polishing the speech transcription text comprises:

[0023] cutting the speech context associated with the single-page speech content, inputting the speech context into a language large model for polishing, correcting redundant words and supplementing omitted information.

[0024] In one embodiment, the calculating the modal weight of the polished speech transcription text and the structured semantic information respectively comprises:

[0025] input the polished speech transcription text and the structured semantic information into a multi-modal semantic fusion model;

[0026] The key nodes of the structured semantic information are taken as query requests, and the word vectors of the polished speech transcription text are taken as key values. Through a cross-attention mechanism, the respective modal weights of the polished speech transcription text and the structured semantic information are calculated.

[0027] In one of the embodiments, the generating of the text summary of the target speech video according to the modal weights comprises:

[0028] According to the respective modal weights of the polished speech transcription text and the structured semantic information, the core text of the single-page speech content is generated.

[0029] The core text of each page of speech content is input into a language large model to be integrated into a hierarchical text summary report of the target speech video.

[0030] In a second aspect, the application further provides a video content extraction device, comprising:

[0031] A video image processing module is configured to perform frame extraction processing on the downloaded target speech video to generate a continuous image sequence.

[0032] The video image processing module is further configured to perform screening on the continuous image sequence after frame extraction to identify picture frames containing effective speech content.

[0033] A video and speech processing module is configured to input the picture frames containing effective speech content into a multi-modal large model for semantic understanding to generate structured semantic information.

[0034] The video and speech processing module is further configured to convert speech into text and polish the speech transcription text.

[0035] A time sequence alignment module is configured to perform time sequence alignment processing on the time stamp corresponding to the picture frames containing effective speech content and the time stamp corresponding to speech transcription.

[0036] A video content generation module is configured to calculate the respective modal weights of the polished speech transcription text and the structured semantic information, and generate a text summary of the target speech video according to the modal weights.

[0037] In a third aspect, the application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0038] Frame extraction processing is performed on the downloaded target speech video to generate a continuous image sequence.

[0039] screening the continuous image sequence after frame extraction, to identify picture frames containing effective speech content;

[0040] inputting the picture frames containing effective speech content into a multi-modal large model for semantic understanding, to generate structured semantic information;

[0041] separating the audio track in the target speech video, performing speech transcription with timestamps, to generate speech transcription text, and performing polishing processing on the speech transcription text;

[0042] performing time sequence alignment processing on the timestamps corresponding to the picture frames containing effective speech content and the timestamps corresponding to speech transcription;

[0043] calculating the modal weights of the polished speech transcription text and the structured semantic information respectively, and generating a text summary of the target speech video according to the modal weights.

[0044] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the following steps:

[0045] performing frame extraction processing on the downloaded target speech video, to generate a continuous image sequence;

[0046] screening the continuous image sequence after frame extraction, to identify picture frames containing effective speech content;

[0047] inputting the picture frames containing effective speech content into a multi-modal large model for semantic understanding, to generate structured semantic information;

[0048] separating the audio track in the target speech video, performing speech transcription with timestamps, to generate speech transcription text, and performing polishing processing on the speech transcription text;

[0049] performing time sequence alignment processing on the timestamps corresponding to the picture frames containing effective speech content and the timestamps corresponding to speech transcription;

[0050] calculating the modal weights of the polished speech transcription text and the structured semantic information respectively, and generating a text summary of the target speech video according to the modal weights.

[0051] In a fifth aspect, the present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the following steps:

[0052] performing frame extraction processing on the downloaded target speech video, to generate a continuous image sequence;

[0053] The continuous image sequence after frame extraction is screened to identify the picture frames containing effective speech content;

[0054] The picture frames containing effective speech content are input into a multi-modal large model for semantic understanding to generate structured semantic information;

[0055] The audio track in the target speech video is separated, and the timestamp is attached for speech transcription to generate a speech transcription text, and the speech transcription text is polished;

[0056] The timestamps corresponding to the picture frames containing effective speech content are time-aligned with the timestamps corresponding to the speech transcription;

[0057] The modal weights of the polished speech transcription text and the structured semantic information are calculated, and according to the modal weights, a text summary of the target speech video is generated.

[0058] The above video content extraction method, device, computer equipment, computer readable storage medium and computer program product first perform frame extraction on the downloaded target speech video to generate a continuous image sequence, and screen out picture frames containing effective speech content. Then, these picture frames are input into a multi-modal large model for semantic understanding to generate structured semantic information, thereby accurately extracting key content and avoiding the problem of information fragmentation in traditional methods. At the same time, the audio track in the target speech video is separated, and the timestamp is attached for speech transcription, and the transcription text is polished to correct redundant words and supplement omitted information, making the text more smooth and natural. By time-aligning the timestamps of the picture frames with the timestamps of the speech transcription, the accuracy of content extraction is further improved, ensuring that the generated text summary is close to the actual semantics of the video. In addition, the modal weights of the polished speech transcription text and the structured semantic information are calculated, and the text summary is generated according to the weights, which not only retains the single-page details, but also presents the overall logic, enhancing the completeness and logic of the content. These steps work together to not only improve the accuracy, readability and completeness of content extraction, but also improve the processing efficiency, enhance the adaptability and flexibility of the content, so that the generated text summary can quickly and accurately reflect the key content of the video, meeting the needs of various application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other related drawings without creative labor.

[0060] Figure 1 An application environment diagram of the video content extraction method in an embodiment;

[0061] Figure 2 A flowchart of the video content extraction method in an embodiment;

[0062] Figure 3 A main step diagram of the video content extraction method in an embodiment;

[0063] Figure 4 A flowchart of the video content extraction method in another embodiment;

[0064] Figure 5 A structure block diagram of the video content extraction device in an embodiment;

[0065] Figure 6 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0067] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the options, or any combination of multiple options.

[0068] The video content extraction method provided by the embodiments of the present application can be applied in the application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through the network. The data storage system can store the data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0069] The server 104 performs frame extraction processing on the downloaded target speech video to generate a continuous image sequence; screens the continuous image sequence after frame extraction, and identifies picture frames containing effective speech content; inputs the picture frames containing effective speech content into a multi-modal large model for semantic understanding to generate structured semantic information; separates the audio track in the target speech video, performs speech transcription with timestamps, generates speech transcription text, and performs polishing processing on the speech transcription text; performs time sequence alignment processing on the timestamps corresponding to the picture frames containing effective speech content and the timestamps corresponding to the speech transcription; calculates the modal weights of the polished speech transcription text and the structured semantic information respectively, and generates a text summary of the target speech video according to the modal weights.

[0070] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0071] In an exemplary embodiment, as shown in Figure 2 , a video content extraction method is provided. The method is applied to the server 104 in Figure 1 for example, and includes the following steps S202 to S212. Among them:

[0072] Step S202, performing frame extraction processing on the downloaded target speech video to generate a continuous image sequence.

[0073] Specifically, the target speech video is downloaded, and fixed frame rate (such as 1 frame per second) is used for frame extraction processing to generate a continuous image sequence.

[0074] Step S204, screening the continuous image sequence after frame extraction, and identifying picture frames containing effective speech content.

[0075] Specifically, a video is composed of a series of consecutive picture frames. Each frame is a static image. In a lecture video, not every frame contains useful information. Some frames may be repetitive, blurry, or irrelevant to the lecture content. Therefore, it is necessary to filter out those picture frames that contain effective lecture content. These frames usually contain important information such as the speaker's actions, expressions, PPT content, etc.

[0076] For example, a binary classifier based on ResNet-50 (pre-trained on a public dataset containing 100,000 PPT and non-PPT images) is used to filter the frame images. The classifier input is a 224x224x3 RGB image, and the output is a PPT picture probability value (threshold set to 0.85). The filtering criteria include whether there are horizontal / vertical dividing lines (identified by edge detection algorithm), bullets (HOG feature matching), or page label text (text detection model based on CRNN positioning) in the picture. Picture frames containing effective lecture content are identified, and other picture frames are removed.

[0077] Step S206, input the picture frames containing effective lecture content to the multi-modal large model for semantic understanding, and generate structured semantic information.

[0078] Specifically, the multi-modal large model is a deep learning model that can handle multiple modal data (such as text, image, audio, etc.). For example, models such as CLIP and Flamingo can handle both image and text data. For example, identify objects in the picture (such as the speaker, PPT slides, charts, etc.). Understand the relationship between these objects (such as the speaker is pointing to a certain part of the PPT). Extract key information in the picture (such as the text content on the PPT, the speaker's expression, etc.).

[0079] The result of structured semantic information is usually a structured data format, for example:

[0080] Text description: Convert the content in the picture frame into a natural language description. For example, "the speaker is showing a slide containing a data chart."

[0081] Label or classification: classify or label the content in the picture frame. For example, "speaker, PPT, chart."

[0082] Relationship graph: represents the relationship between different elements in the picture. For example, "the speaker points to the chart in the PPT."

[0083] Key information extraction: extract key information in the picture, such as the title on the PPT, important data, etc.

[0084] For example, input the selected PPT screen into the GPT-4V multimodal large model, extract a 14x14 image feature map through the ViT-L / 14 visual encoder, combine the text prompt "extract the title, sub-title, and logical relationship between each point of the current PPT", align the visual and language features through the cross-modal attention mechanism (head number 12, hidden layer dimension 768), and output structured JSON semantic information (such as {"title":"Deep Learning Basics","content":[{"point":"Neural Network Structure","associated points":"Activation Function Role"},{"point":"Backpropagation","associated points":"Gradient Calculation"}]}).

[0085] Step S208, separate the audio track in the target speech video, and perform speech transcription with timestamps, generate speech transcription text, and polish the speech transcription text.

[0086] Specifically, the target speech video refers to the speech video file that needs to be processed. Audio track separation refers to extracting the audio part from the video file to make it an independent audio file, so that the audio and video content can be processed separately, which is convenient for subsequent speech transcription and polishing. Timestamp refers to the specific time point of the beginning and end of each word or sentence in the audio. In the speech transcription process, the timestamp can help align the speech content with the specific frame of the video.

[0087] For example, use the Whisper-large-v3 model for speech transcription, output a timestamped text sequence (time precision 100ms). For example, the timestamp [00:01:23-00:01:35] corresponds to the text "Next we discuss the structure of neural networks".

[0088] After speech transcription, the resulting written text content usually contains some grammatical errors, redundant words, or incomplete sentences. Further editing and optimization of the generated speech transcription text can improve the quality and readability of the text. For example, input the speech transcription text into the LLaMA-3-70B model for polishing (maximum input length 2048, temperature parameter 0.3), correct redundant words and clarify references.

[0089] Step S210, align the timestamps corresponding to the frames containing the effective speech content with the timestamps corresponding to the speech transcription in time sequence.

[0090] Specifically, the timestamp refers to the specific time point of a frame or speech transcription text in the video. Each frame and each speech transcription text segment has a corresponding timestamp indicating its appearance time in the video.

[0091] The timestamps of the video frames are matched with the timestamps of the speech-to-text to ensure that the visual and auditory information are synchronized in time. The purpose of this step is to accurately match the video frames with the corresponding audio content, so that the generated text summary can accurately reflect the actual content of the video.

[0092] Algorithms or models are used to calculate the correspondence between the timestamps of video frames and the speech-to-text. Alignment methods can include time window expansion, text segment matching, and semantic similarity calculation to select the most relevant speech segments as the associated context.

[0093] Step S212: Calculate the modal weights of the polished speech-to-text and the structured semantic information, and generate a text summary of the target speech video based on the modal weights.

[0094] Specifically, modal weights can help the model better balance the contributions of each modality when fusing information from different modalities.

[0095] The polished speech-to-text and structured semantic information are input into a multimodal semantic fusion model. A cross-attention mechanism is used to calculate the contribution weight of each modality to the key content. Through the model's internal mechanism, the weight of each modality is calculated to determine the respective importance of the speech-to-text and structured semantic information when generating the text summary.

[0096] Based on the calculated modal weights, a comprehensive text summary is generated. This summary not only includes the content of the speech-to-text but also integrates structured semantic information from the video frames, making the summary more comprehensive, accurate, and logical. The speech-to-text and structured semantic information are fused according to the modal weights. The generated text summary retains the details of each page while presenting the overall logic, accurately reflecting the key content of the speech video.

[0097] In the above video content extraction methods, such as Figure 3As shown, these picture frames are input into a multi-modal large model for semantic understanding, generating structured semantic information, thereby accurately extracting key content, avoiding the problem of information fragmentation in traditional methods. At the same time, the audio track in the target speech video is separated, with timestamps for speech transcription, and the transcribed text is polished to correct redundant words and supplement omitted information, making the text more smooth and natural. By aligning the timestamps of the picture frames with the timestamps of the speech transcription, the accuracy of content extraction is further improved, ensuring that the generated text summary is close to the actual semantics of the video. In addition, the modal weight of the polished speech transcription text and the structured semantic information is calculated, and the text summary is generated according to the weight, not only retaining the details of a single page, but also presenting the overall logic, enhancing the completeness and logic of the content. These steps work together to not only improve the accuracy, readability, and completeness of content extraction, but also improve processing efficiency, enhance content adaptability and flexibility, so that the generated text summary can quickly and accurately reflect the key content of the video, meeting the needs of various application scenarios.

[0098] In one exemplary embodiment, as Figure 4 shown, picture frames containing effective speech content are input into a multi-modal large model for semantic understanding, generating structured semantic information, including:

[0099] Step S402, input picture frames containing effective speech content into a multi-modal large model, extract image region features, including title area, list items and chart area;

[0100] Step S404, according to the image region features, extract text features through the text encoder and the pre-defined text extraction instructions;

[0101] Step S406, using cross-modal attention mechanism, align the image region features and text features, and generate structured semantic information.

[0102] Specifically, picture frames containing effective speech content are input into a multi-modal large model. These picture frames are usually selected from videos and contain key content of the speech. The multi-modal large model extracts image region features from the picture frames, which can be divided into different regions, such as: title area: the part of the picture that displays the title. List items: the part of the picture that displays lists or bullets. Chart area: the part of the picture that displays charts.

[0103] The text content in the picture frame is encoded using a text encoder (such as BERT, GPT, etc.). The text encoder converts the text into high-dimensional feature vectors, which can represent the semantic information of the text. Through predefined text extraction instructions, the model is guided to extract the text content in specific areas, which can help the model more accurately identify and extract the text in the title, list items, and chart. The extracted text features are high-dimensional feature vectors output by the text encoder.

[0104] The cross-modal attention mechanism is a technique for processing the interaction between different modalities (such as images and text). The extracted image region features (such as the features of the title area, list items, and chart area) are aligned with the text features. The purpose of alignment is to find the correspondence between the image and the text, for example, aligning the title area in the image with the title content in the text. Through alignment processing, structured semantic information is generated. Structured semantic information can include: text description, label or classification, relationship graph, etc.

[0105] In this embodiment, by inputting the picture frame containing effective speech content into a multi-modal large model for semantic understanding, accurate content extraction and efficient feature alignment are achieved, and structured semantic information is generated. This not only improves the accuracy and completeness of content extraction, but also enhances the logicality and readability of text summary. The generated structured semantic information covers text description, label classification, relationship graph, and key information extraction, etc., providing strong support for application scenarios such as quick browsing, conference record organization, online education content refinement, etc., significantly improving content understanding and application efficiency.

[0106] In one exemplary embodiment, the timestamps corresponding to the picture frames containing effective speech content are time-aligned with the timestamps corresponding to the speech transcription, including:

[0107] The timestamp of each picture frame containing effective speech content is expanded into a time window;

[0108] Within the time window, match the first speech transcription text segment with overlapping timestamps in the speech transcription;

[0109] Using a multi-modal large model, respectively extract the semantic vectors of the picture frames and the speech transcription text containing effective speech content, and calculate the cosine similarity between them;

[0110] According to the cosine similarity, the second speech transcription text segment with the highest similarity is selected from the first speech transcription text segment.

[0111] Specifically, each picture frame has a specific timestamp indicating its appearance time in the video. To increase the flexibility of matching, the timestamp of each picture frame is extended to a time window. The speech transcription text is segmented into multiple text segments, each with its own timestamp indicating the appearance time of the text segment in the audio. Within the extended time window, find the speech transcription text segments that overlap with the picture frame timestamp, which are referred to as the first speech transcription text segments.

[0112] The multi-modal large model (such as CLIP, Flamingo, etc.) is used to extract the semantic vectors of the picture frames and the speech transcription text respectively. The semantic vector is a high-dimensional vector used to represent the semantic information of the input data. Calculate the cosine similarity between the semantic vector of the picture frame and the semantic vector of the speech transcription text. The closer the cosine similarity is to 1, the more similar the two vectors are. From the first speech transcription text segment, the text segment with the highest similarity is selected according to the cosine similarity, which is referred to as the "second speech transcription text segment". Through the screening of semantic similarity, the consistency of the picture frame and the speech transcription text in semantics is ensured, further improving the accuracy of the time alignment.

[0113] For example, the DTW algorithm (Dynamic Time Warping algorithm) is used to align the PPT frame timestamp (precision 1 second) and the speech timestamp (precision 100 ms), the extended time window is set to ±2 seconds, and the speech segment with a matching overlap rate ≥50% is matched; At the same time, calculate the cosine similarity of the PPT semantic vector (GPT-4V output CLS token) and the speech text vector (Sentence-BERT encoding), and select the speech segment with a similarity ≥0.7 as the associated context.

[0114] In this embodiment, through time window extension, text segment matching, semantic vector extraction and cosine similarity calculation, the timestamp of the picture frame is time-aligned with the timestamp of the speech transcription. This process not only considers the proximity in time, but also ensures the consistency in content through semantic similarity, significantly improving the accuracy and reliability of the time alignment.

[0115] In an exemplary embodiment, the speech transcription text is polished, including:

[0116] Cut the speech context associated with the content of a single page of speech, and input the speech context into the language large model for polishing to correct redundant words and supplement omitted information.

[0117] Specifically, the single-page speech content refers to the key information and topics contained in each page of PPT or speech content. The speech context refers to the part extracted from the speech transcription text that is associated with the current single-page speech content, aiming to extract the speech segments directly related to the current page content, ensuring the pertinence and accuracy of subsequent processing.

[0118] The speech context is input to a language large model for polishing processing. The language large model, such as GPT, BERT, etc., has strong natural language processing capabilities and can polish and optimize the text.

[0119] In this embodiment, the speech transcription text is polished by extracting the speech context associated with the single-page speech content and inputting it to the language large model, effectively correcting redundant words and supplementing omitted information, making the text more fluent, natural and complete. This process significantly improves the readability and accuracy of the speech transcription text and enhances the expression effect of the text content, providing a high-quality text basis for recording, organizing and disseminating the speech content.

[0120] In one exemplary embodiment, the modal weights of the polished speech transcription text and the structured semantic information are calculated respectively, including:

[0121] The polished speech transcription text and the structured semantic information are input into a multi-modal semantic fusion model;

[0122] The key nodes of the structured semantic information are used as query requests, and the word vectors of the polished speech transcription text are used as key values. Through the cross-attention mechanism, the modal weights of the polished speech transcription text and the structured semantic information are calculated respectively.

[0123] Specifically, the multi-modal semantic fusion model is a deep learning model that can handle multiple modal data (such as text and structured information), and generates more comprehensive semantic representations by fusing information from different modalities. The polished speech transcription text and the structured semantic information are input into the multi-modal semantic fusion model, and these data will be used by the model for subsequent modal weight calculation.

[0124] The cross-attention mechanism is a variant of the attention mechanism, which is used to handle the interaction between different modalities. Specifically, it aligns and weights the information from different modalities through query (Query) and key value (Key). Among them, the query request (Query) is the key node of the structured semantic information, which is the most important part of the structured information, used to guide the attention mechanism. The key value (Key) is the word vector of the polished speech transcription text, which is a high-dimensional representation of the speech transcription text, used to match the query.

[0125] Through the cross-attention mechanism, the modal weights of the polished speech transcription text and the structured semantic information are calculated. Cosine similarity is usually used to measure the similarity between the query and the key value. The higher the similarity, the more relevant the two modal information is in terms of semantics.

[0126] According to the calculated similarity, a weight is assigned to each modality. The modality with a higher weight will play a greater role in generating the final content. For example, the PPT title has a weight of 0.6, and the speech emphasized part has a weight of 0.4. The weighted fusion generates a single-page core content (including the main points, explanations, and logical relationships).

[0127] In this embodiment, the precise fusion of multi-modal information is achieved by calculating the modal weights of the polished speech transcription text and the structured semantic information. Using the cross-attention mechanism, the key nodes of the structured semantic information are used as query requests, and the word vectors of the speech transcription text are used as key values. The semantic similarity between the two is dynamically calculated, and a reasonable weight is assigned to each modality. This process not only improves the accuracy and completeness of the text summary, but also enhances the logicality and readability of the content.

[0128] In one exemplary embodiment, according to the modal weights, a text summary of the target speech video is generated, including:

[0129] According to the respective modal weights of the polished speech transcription text and the structured semantic information, the core text of the single-page speech content is generated.

[0130] The core text of each page of speech content is input into a language large model, and is integrated into a hierarchical text summary report of the target speech video.

[0131] Specifically, according to the modal weights, the polished speech transcription text and the structured semantic information are weighted and fused. The modality information with a higher weight will play a greater role in generating the core text. Through the fused information, the core text of each page of speech content is generated. These core texts not only retain the fluency of the speech transcription text, but also integrate the accuracy of the structured semantic information.

[0132] The core text of each page of speech content is input into a language large model. The language large model, such as T5-3B, generates a hierarchical text summary report based on the input core text. This report not only retains the detailed content of each page, but also integrates these contents to form a logically structured overall report.

[0133] In this embodiment, the core text of each page of the speech content is generated by fusing the polished speech transcription text and the structured semantic information, and then the core text is input into the language large model to generate a hierarchical text summary report. This process not only retains the detailed content of each page, but also forms a logically structured overall report by integrating the content, significantly improving the quality and readability of the text summary.

[0134] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0135] Based on the same inventive concept, the embodiments of the present application also provide a video content extraction device for implementing the above-mentioned video content extraction method. The problem-solving implementation scheme provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more video content extraction device embodiments provided below can refer to the limitations of the video content extraction method in the above text, which will not be repeated here.

[0136] In one exemplary embodiment, as shown in Figure 5 a video content extraction device is provided, comprising:

[0137] The video image processing module 502 is configured to perform frame extraction processing on the downloaded target speech video to generate a continuous image sequence.

[0138] The video image processing module 502 is further configured to filter the continuous image sequence after frame extraction to identify picture frames containing valid speech content.

[0139] The video speech processing module 504 is configured to input the picture frames containing valid speech content into a multi-modal large model for semantic understanding to generate structured semantic information.

[0140] The video speech processing module 504 is further configured to perform speech transcription and polish the speech transcription.

[0141] The timing alignment module 506 is configured to perform timing alignment processing on the timestamp corresponding to the picture frame containing the valid speech content and the timestamp corresponding to the speech transcription.

[0142] The video content generation module 508 is configured to calculate the modal weight of the polished speech transcription text and the structured semantic information respectively, and generate a text summary of the target speech video according to the modal weight.

[0143] In an exemplary embodiment, the picture frame containing the valid speech content is input into a multi-modal large model to extract image region features, which include title areas, list items, and chart areas.

[0144] According to the image region features, text features are extracted through a text encoder and a predefined text extraction instruction.

[0145] The image region features and the text features are aligned using a cross-modal attention mechanism to generate structured semantic information.

[0146] In an exemplary embodiment, the timestamp of each picture frame containing valid speech content is extended to a time window.

[0147] Within the time window, a first speech transcription text segment with a timestamp overlap in the speech transcription is matched.

[0148] The multi-modal large model is used to extract semantic vectors of the picture frame containing the valid speech content and the speech transcription text respectively, and calculate the cosine similarity between them.

[0149] According to the cosine similarity, the second speech transcription text segment with the highest similarity is selected from the first speech transcription text segment.

[0150] In an exemplary embodiment, the speech context associated with the single-page speech content is intercepted, and the speech context is input into a language large model for polishing processing to correct redundant words and supplement omitted information.

[0151] In an exemplary embodiment, the polished speech transcription text and the structured semantic information are input into a multi-modal semantic fusion model.

[0152] The key nodes of the structured semantic information are used as query requests, and the word vectors of the polished speech transcription text are used as key values. Through cross-attention mechanism, the modal weight of the polished speech transcription text and the structured semantic information is calculated.

[0153] In an exemplary embodiment, according to the modal weight of the polished speech transcription text and the structured semantic information, a core text of the single-page speech content is generated.

[0154] The core text of each page of the speech content is input into the language large model, and is integrated into a hierarchical text summary report of the target speech video.

[0155] Each module in the video content extraction device can be implemented wholly or partially by software, hardware, and combinations thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so that the processor invokes and executes the operations corresponding to each module.

[0156] In an exemplary embodiment, a computer device, which can be a server, has an internal structure diagram as shown in Figure 6 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store target speech video data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a video content extraction method.

[0157] Those skilled in the art can understand that Figure 6 The structure shown in the above

[0158] In an exemplary embodiment, a computer device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps:

[0159] Frame extraction processing is performed on the downloaded target speech video to generate a continuous image sequence.

[0160] The continuous image sequence after frame extraction is filtered to identify picture frames containing valid speech content.

[0161] inputting the picture frame containing the effective speech content into the multi-modal large model for semantic understanding to generate structured semantic information;

[0162] separating the audio track in the target speech video, performing speech transcription with timestamps, generating speech transcription text, and polishing the speech transcription text;

[0163] aligning the timestamps corresponding to the picture frame containing the effective speech content with the timestamps corresponding to the speech transcription in time sequence;

[0164] calculating the modal weights of the polished speech transcription text and the structured semantic information respectively, and generating a text summary of the target speech video according to the modal weights.

[0165] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0166] inputting the picture frame containing the effective speech content into the multi-modal large model to extract image region features, the image region features including title area, list items and chart area;

[0167] extracting text features from the image region features through a text encoder and a predefined text extraction instruction;

[0168] aligning the image region features and the text features using a cross-modal attention mechanism to generate structured semantic information.

[0169] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0170] extending the timestamp of each picture frame containing effective speech content into a time window;

[0171] matching a first speech transcription text segment with overlapping timestamps in the speech transcription within the time window;

[0172] extracting semantic vectors of the picture frame containing the effective speech content and the speech transcription text using the multi-modal large model, and calculating the cosine similarity between them;

[0173] According to the cosine similarity, the second speech transcription text segment with the highest similarity is selected from the first speech transcription text segment.

[0174] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0175] cutting the speech context associated with the single-page speech content, inputting the speech context into the language large model for polishing, and correcting redundant words and supplementing omitted information.

[0176] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0177] The polished speech transcription text and structured semantic information are input into a multi-modal semantic fusion model.

[0178] The key nodes of the structured semantic information are taken as query requests, and the word vectors of the polished speech transcription text are taken as key values. Through a cross-attention mechanism, the modal weights of the polished speech transcription text and the structured semantic information are calculated.

[0179] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0180] According to the modal weights of the polished speech transcription text and the structured semantic information, the core text of the single-page speech content is generated.

[0181] The core text of each page of the speech content is input into a language large model to integrate into a hierarchical text summary report of the target speech video.

[0182] In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the following steps:

[0183] The downloaded target speech video is subjected to frame extraction processing to generate a continuous image sequence.

[0184] The continuous image sequence after frame extraction is screened to identify picture frames containing valid speech content.

[0185] The picture frames containing valid speech content are input into a multi-modal large model for semantic understanding to generate structured semantic information.

[0186] The audio track in the target speech video is separated, and the speech transcription is performed with a time stamp to generate a speech transcription text, and the speech transcription text is polished.

[0187] The time stamps corresponding to the picture frames containing valid speech content are time-aligned with the time stamps corresponding to the speech transcription.

[0188] The modal weights of the polished speech transcription text and the structured semantic information are calculated, and according to the modal weights, a text summary of the target speech video is generated.

[0189] In one embodiment, the computer program, when executed by the processor, also implements the following steps:

[0190] The picture frames containing valid speech content are input into a multi-modal large model to extract image region features, including title areas, list items, and chart areas.

[0191] According to the image region features, text features are extracted through a text encoder and predefined text extraction instructions;

[0192] The image region features and the text features are aligned through a cross-modal attention mechanism to generate structured semantic information.

[0193] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0194] The timestamp of each picture frame containing valid speech content is extended to a time window;

[0195] In the time window, the first speech transcription text segment with overlapping timestamps in the speech transcription is matched;

[0196] The semantic vectors of the picture frames and the speech transcription text containing valid speech content are extracted respectively using a multi-modal large model, and the cosine similarity between them is calculated;

[0197] According to the cosine similarity, the second speech transcription text segment with the highest similarity is selected from the first speech transcription text segment.

[0198] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0199] The speech context associated with the single-page speech content is intercepted, and the speech context is input into a language large model for polishing processing to correct redundant words and supplement omitted information.

[0200] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0201] The polished speech transcription text and the structured semantic information are input into a multi-modal semantic fusion model;

[0202] The key nodes of the structured semantic information are used as query requests, and the word vectors of the polished speech transcription text are used as key values. Through cross-attention mechanism, the modal weights of the polished speech transcription text and the structured semantic information are calculated.

[0203] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0204] According to the modal weights of the polished speech transcription text and the structured semantic information, the core text of the single-page speech content is generated;

[0205] The core texts of the speech content of each page are input into a language large model and integrated into a hierarchical text summary report of the target speech video.

[0206] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps:

[0207] Frame extraction processing is performed on the downloaded target speech video to generate a continuous image sequence;

[0208] The continuous image sequence after frame extraction is screened to identify picture frames containing valid speech content;

[0209] The picture frames containing valid speech content are input to a multi-modal large model for semantic understanding to generate structured semantic information;

[0210] The audio track in the target speech video is separated, and the timestamp is attached for speech transcription to generate speech transcription text, and the speech transcription text is polished;

[0211] The timestamps corresponding to the picture frames containing valid speech content are time-aligned with the timestamps corresponding to the speech transcription;

[0212] The modal weights of the polished speech transcription text and the structured semantic information are calculated, and according to the modal weights, a text summary of the target speech video is generated.

[0213] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0214] The picture frames containing valid speech content are input to the multi-modal large model to extract image region features, including title area, list items, and chart area;

[0215] According to the image region features, text features are extracted through a text encoder and a predefined text extraction instruction;

[0216] Using a cross-modal attention mechanism, the image region features and the text features are aligned to generate structured semantic information.

[0217] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0218] The timestamp of each picture frame containing valid speech content is expanded into a time window;

[0219] In the time window, a first speech transcription text segment with overlapping timestamps in the speech transcription is matched;

[0220] Using a multi-modal large model, semantic vectors of the picture frames containing valid speech content and the speech transcription text are extracted respectively, and the cosine similarity between them is calculated;

[0221] According to the cosine similarity, the second speech transcription text segment with the highest similarity is screened out from the first speech transcription text segment.

[0222] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0223] The speech context associated with the single-page speech content is intercepted, and the speech context is input to the language large model for polishing processing to correct redundant words and supplement omitted information.

[0224] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0225] The polished speech transcription text and the structured semantic information are input into a multi-modal semantic fusion model.

[0226] The key nodes of the structured semantic information are taken as query requests, and the word vectors of the polished speech transcription text are taken as key values, and through a cross-attention mechanism, the modal weights of the polished speech transcription text and the structured semantic information are calculated.

[0227] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0228] According to the modal weights of the polished speech transcription text and the structured semantic information, the core text of the single-page speech content is generated.

[0229] The core texts of the speech contents of each page are input into the language large model, and are integrated into a hierarchical text summary report of the target speech video.

[0230] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0231] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0232] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0233] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method of video content extraction, characterized by, The method comprises: frame extraction processing is performed on the downloaded target speech video to generate a continuous image sequence; screening is performed on the continuous image sequence after frame extraction to identify picture frames containing effective speech content; the picture frames containing effective speech content are input into a multi-modal large model for semantic understanding to generate structured semantic information; the audio track in the target speech video is separated, speech transcription is performed with timestamps, speech transcription text is generated, and polishing processing is performed on the speech transcription text; time sequence alignment processing is performed on the timestamps corresponding to the picture frames containing effective speech content and the timestamps corresponding to speech transcription; the modal weights of the polished speech transcription text and the structured semantic information are calculated, and a text summary of the target speech video is generated according to the modal weights.

2. The method of claim 1, wherein, The structured semantic information is generated by inputting the picture frames containing effective speech content into a multi-modal large model for semantic understanding, comprising: the picture frames containing effective speech content are input into a multi-modal large model to extract image region features, the image region features including title areas, list items and chart areas; text features are extracted through a text encoder and a pre-defined text extraction instruction according to the image region features; the image region features and the text features are aligned using a cross-modal attention mechanism to generate structured semantic information.

3. The method of claim 1, wherein, The time sequence alignment processing on the timestamps corresponding to the picture frames containing effective speech content and the timestamps corresponding to speech transcription comprises: the timestamps of each picture frame containing effective speech content are expanded into a time window; in the time window, a first speech transcription text segment with overlapping timestamps in speech transcription is matched; the semantic vectors of the picture frames containing effective speech content and the speech transcription text are extracted respectively using a multi-modal large model, and the cosine similarity between them is calculated; according to the cosine similarity, the second speech transcription text segment with the highest similarity is selected from the first speech transcription text segment.

4. The method of claim 1, wherein, The polishing processing on the speech transcription text comprises: the speech context associated with the single-page speech content is intercepted, and the speech context is input into a language large model for polishing processing to correct redundant words and supplement omitted information.

5. The method of claim 1, wherein, The calculation of the modal weights of the polished speech transcription text and the structured semantic information comprises: the polished speech transcription text and the structured semantic information are input into a multi-modal semantic fusion model; the key nodes of the structured semantic information are used as query requests, and the word vectors of the polished speech transcription text are used as key values, and the modal weights of the polished speech transcription text and the structured semantic information are calculated through cross-attention mechanism.

6. The method of claim 1, wherein, The generation of the text summary of the target speech video according to the modal weights comprises: the core texts of single-page speech content are generated according to the modal weights of the polished speech transcription text and the structured semantic information; the core texts of each page of speech content are input into a language large model to integrate into a hierarchical text summary report of the target speech video.

7. A video content extraction apparatus characterized by comprising: The device comprises: a video image processing module, configured to perform frame extraction processing on a downloaded target speech video to generate a continuous image sequence; the video image processing module is further configured to perform screening on the continuous image sequence after frame extraction to identify picture frames containing valid speech content; a video and voice processing module, configured to input the picture frames containing valid speech content into a multi-modal large model to perform semantic understanding and generate structured semantic information; the video and voice processing module is further configured to perform voice-to-text conversion and polishing processing on the voice-to-text converted text; a time sequence alignment module, configured to perform time sequence alignment processing on timestamps corresponding to the picture frames containing valid speech content and timestamps corresponding to voice-to-text conversion; a video content generation module, configured to calculate modal weights of the polished voice-to-text converted text and the structured semantic information respectively, and generate a text summary of the target speech video according to the modal weights. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to realize the steps of the method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 6.

Images