Dynamic news abstract generation method based on voice and visual perception

Abstract

The invention discloses a dynamic news abstract generation method based on voice and visual perception, which comprises the steps of voice recognition text extraction, audio-text alignment, prompt word generation and final news abstract generation. The objective is achieved through an innovative process comprising a plurality of key optimization links. A robust long audio processing and recognition mechanism, an optimized audio-text alignment method, an elaborately designed cue word framework and an efficient fine-tuning multi-modal large language model are integrated. According to the method, deep understanding of multi-modal information and high-quality text generation are jointly realized, and the quality of four core dimensions including content coverage, semantic accuracy, logic continuity and language fluency of the abstract is comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for generating dynamic news summaries based on voice and visual perception. Background Technology

[0002] In recent years, with the explosive growth of multimodal information, the demand for news content from businesses and individual users is gradually shifting from "comprehensive acquisition" to "efficient understanding," meaning a greater focus on whether information is accurate, concise, and logically clear. This demand for high-quality information is particularly prominent in specific applications such as content moderation on self-media platforms and rapid screening of financial information that requires quick responses to market changes. For example, in content moderation scenarios, moderators not only need to quickly understand the core content of a video but also need to ensure that the summary accurately reflects the core meaning and key information of the original content. This places high demands on automatically generated summaries in terms of semantic accuracy and content coverage. Therefore, the compression and reconstruction of audio and video news content, especially the generation of high-quality summaries, has become an important research topic in the field of news dissemination and information processing.

[0003] Currently, most news summarization methods focus on processing single modalities such as images or text, neglecting the rich information contained in multimodal data. This makes it difficult to meet the requirements of comprehensive and accurate information delivery in the aforementioned application scenarios, resulting in summaries that are deficient in logical coherence, content coverage (i.e., semantic integrity), and key information extraction. For example, existing keyframe extraction methods mostly rely on image features and struggle to combine them with auxiliary information such as subtitles, speech semantics, and charts, easily leading to summaries omitting core news elements, thus affecting coverage, representativeness, and accuracy. Furthermore, weak semantic fusion capabilities between multimodal data also limit further improvements in the quality of summaries, such as semantic accuracy and logical coherence.

[0004] Multimodal summarization methods can be categorized into three types: traditional methods, deep learning methods, and large model methods.

[0005] Traditional methods for multimodal summarization:

[0006] Traditional multimodal summarization methods largely rely on statistical features and rule-based matching techniques, typically compressing text and visual data through key feature extraction, cluster analysis, and similarity calculation. In text summarization, TF-IDF, Latent Semantic Analysis (LSA), and graph-based TextRank algorithms are widely used to extract key information from massive amounts of text. In visual summarization, traditional methods analyze images using local features (such as SIFT and HOG) or global features (such as BoVW) to filter out the visual information most relevant to the text content. While these methods are effective to some extent, they often ignore the deep semantic connections between different modalities and have poor adaptability when dealing with complex scenarios.

[0007] Deep learning methods for multimodal summarization generation:

[0008] With the introduction of deep learning technology, especially breakthroughs in feature learning, semantic alignment, and information fusion, the performance of multimodal summarization has been significantly improved. Text-image summarization methods based on CNNs and RNNs extract visual features using CNNs, process textual information using LSTMs, and fuse information through attention mechanisms to generate more accurate summaries. For example, visual features can be transformed into sequences using LSTM structures, thereby generating fluent natural language summaries. This advancement has enabled multimodal summarization to handle more complex cross-modal information, driving the rapid development of this field.

[0009] Methods for generating large models through multimodal summarization:

[0010] In recent years, multimodal summarization methods based on large-scale pre-trained models have gradually become mainstream. These models acquire stronger generalization capabilities through large-scale unsupervised learning and can efficiently fuse multimodal information such as text, vision, and audio to generate high-quality summaries. Vision-Language pre-trained models (VLPs) employ a multimodal Transformer structure for cross-modal information fusion, demonstrating powerful performance in both text and visual summarization tasks. For example, the BLIP model, combined with text-image contrast learning, further improves the performance of multimodal summarization tasks; and the GPT-3 pre-training framework demonstrates outstanding performance in speech-text joint summarization tasks. Meanwhile, end-to-end large-model methods, such as GPT-3, GPT-4, and Gemini, have also made significant progress in video, audio, and text summarization.

[0011] In summary, with the rapid growth of multimodal news content, addressing issues such as incomplete information coverage, semantic bias, logical breaks, and awkward expression in dynamic scenarios, and generating high-quality summaries that are comprehensive in content coverage, accurate in semantic expression, clear in logical structure, and natural and fluent in language, has become a critical challenge that urgently needs to be solved. Existing audio and video summarization methods often exhibit limitations such as insufficient multimodal collaboration, poor contextual understanding, and poor logical coherence when processing dynamic and complex news content, making it difficult to meet the needs of high-quality summary generation. Summary of the Invention

[0012] To address the problems existing in the prior art, this invention provides a dynamic news summary generation method based on speech and visual perception. This invention is designed for audio and video news scenarios and combines advanced multimodal large-scale pre-trained models, cue word framework design, and model fine-tuning methods.

[0013] The objective of this invention is achieved through the following technical solutions.

[0014] A method for generating dynamic news summaries based on speech and visual perception, comprising the following steps:

[0015] 1) Speech recognition and text extraction: Extract speech content from input news videos. Apply a long audio processing algorithm that is robust to fragmented noise and adapts to low whispers. Combine the inherent structural characteristics of news broadcasts with intelligent speech segmentation and scene-adaptive noise reduction mechanisms.

[0016] 2) Audio-text alignment: Construct a timestamp synchronization relationship between audio segments and the identified text content. By integrating advanced semantic matching analysis and speaker role recognition technology, ensure a high degree of consistency of content across the multimodal timeline.

[0017] 3) Prompt word generation: Dynamically construct structured, semantically driven prompt templates based on different news topics and types;

[0018] 4) News summary generation: The Qwen2.5-VL multimodal large language model, which has been efficiently fine-tuned by the LoRA method, is used to comprehensively process text information from the preceding module, speech features with implicit associations through alignment, and key visual frame information.

[0019] Furthermore, in step 1), the speech recognition uses a noise reduction library based on spectrum threshold technology for audio preprocessing to separate the speech signal from the background noise. The noise reduction process is optimized in a scene-aware manner based on the structural characteristics of the news scene.

[0020] Furthermore, the long audio processing algorithm in step 1) includes: the intelligent structure analysis module detects key structural points in the news video, adopts an intelligent segmentation strategy, and performs adaptive segmentation based on the position of the structural points, so that the segment size is both in line with the model's processing capacity and does not disrupt the logical flow of the news. After segmentation, the audio segments are fed into the Whisper-Large-v3 model for ASR speech recognition. At the same time, the model's temperature parameters are adjusted in conjunction with scene adaptive parameters. After ASR recognition, FNR-WA also performs news-specific post-processing, standardizes the expression of numbers and dates, and automatically identifies unique elements such as news titles and quotations to optimize the final text output format.

[0021] Furthermore, step 2) includes precise audio-text alignment: starting from audio loading and processing, the WebRTC speech activity detection module is introduced, and an intelligent segment merging strategy is introduced to reasonably splice continuous speech segments to maintain semantic coherence. On the text side, semantic vectors are extracted through the BERT model to divide the text into sentence-level semantic units.

[0022] Furthermore, during the alignment stage, BERT vectors are used to enhance the matching logic between speech and text. After alignment, the algorithm also incorporates a fine-grained post-processing mechanism for news content, accurately annotating key information and performing personalized optimization based on different speaking roles.

[0023] Furthermore, step 3) includes the design of a prompt word framework: a news summary generation process based on the CoT thinking chain mechanism and the CO-STAR prompt word framework, including news data input, thinking chain reasoning guidance, CO-STAR framework construction and summary generation output.

[0024] Furthermore, step 3) includes prompt word compression: the metadata of the news input is structurally simplified, retaining only the most critical elements and eliminating redundant fields to significantly compress the input size of the model. The algorithm performs semantic reorganization on the compressed information to generate optimized prompt word templates adapted to different news types.

[0025] Furthermore, step 4) includes fine-tuning the Qwen2.5-VL model: combining task characteristics and hardware resources, the Qwen2.5-VL is efficiently fine-tuned based on the LoRA method, and a systematic design is carried out in terms of parameter configuration, training strategy and model adaptation.

[0026] Furthermore, step 4) includes multimodal fusion and news summary generation: the prepared multimodal information is input into the fine-tuned Qwen2.5-VL model. The input includes: news text processed and optimized by FNR-WA and its timestamp information, several key visual frames representing the core content of the news video, and structured cue words that combine logical context and expression norms and are compressed. The Qwen2.5-VL model uses its powerful visual language understanding capabilities to process these inputs, converting them into semantic representations through a text encoder. The key frames extract visual features through a visual encoder. The multimodal fusion module and attention mechanism, fine-tuned by LoRA, combine text semantics with key visual information to form a deep, cross-modal understanding of the news event. The structured cue words play a key guiding role in this process, constraining the model to think and generate according to the preset logical context and expression norms. The model generates a preliminary news summary text.

[0027] Furthermore, the model generates preliminary news summary text which enters the post-processing and optimization process. The steps include: content verification and completion: automatically checking whether the summary covers the core elements in the source material and supplementing it based on contextual logic and timestamp information; structure and format standardization: adjusting the chapter structure and layout of the summary; language polishing: refining the summary text and optimizing the wording and sentence structure.

[0028] Compared with the prior art, the advantages of the present invention are: (1) The present invention proposes an FNR-WA algorithm optimized for news scenarios, which significantly improves the quality and robustness of speech recognition in complex acoustic environments;

[0029] (2) This invention constructs a structured prompt generation strategy that integrates the CoT and CO-STAR frameworks, which effectively enhances the logicality and contextual relevance of the generated summary;

[0030] (3) Through the collaborative design of each optimization step and the deep integration of multimodal information, this invention significantly improves the professional level and practical application value of the final generated summary. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, the dynamic news summary generation method based on speech and visual perception includes four modules: speech recognition text extraction, audio text alignment, prompt word generation, and news summary generation, as follows:

[0034] (1) Speech Recognition Text Extraction Module. The core task of the speech recognition text extraction module is to extract speech content from the input news video in a preliminary but accurate manner. Considering that news corpora often contain long audio segments, varied broadcasting scenarios, and high-density information distribution, this invention has specially developed and applied a fragmented noise-robust low-voice adaptive long audio processing algorithm. It cleverly combines the inherent structural characteristics of news broadcasting and integrates intelligent speech segmentation and scene-adaptive noise reduction mechanisms. This not only significantly improves the quality of the input speech signal, but also effectively enhances the accuracy and stability of subsequent speech recognition, laying a solid text foundation for generating high-quality summaries.

[0035] High-quality speech recognition. In dynamic news summary generation tasks, high-quality speech recognition is a crucial foundation for ensuring the accuracy of subsequent information processing. However, news audio is often accompanied by complex environmental noise, such as wind noise in outdoor reporting and background music at interview sites. These factors significantly interfere with the robustness and accuracy of speech recognition systems. To address these issues, this invention employs a noise reduction library based on spectrum thresholding technology for audio preprocessing, effectively separating the speech signal from background noise and significantly improving speech clarity. Unlike general audio processing methods, this invention optimizes the noise reduction process in a scene-aware manner, taking into account the structural characteristics of news scenarios.

[0036] Long audio processing. In the task of automatic news video summarization, long audio processing has always been a key technical bottleneck. News videos often consist of multiple heterogeneous segments, such as introductory sequences, anchor broadcasts, live reporter interviews, and guest interviews. Directly inputting the entire audio segment into an automatic speech recognition (ASR) model can not only cause memory overflow (OOM) issues but also significantly reduce the accuracy and stability of recognition. The intelligent structure analysis module detects key structural points in the news video and employs an intelligent segmentation strategy. It adaptively segments the audio based on the location of these structural points, ensuring that the segment size matches the model's processing capacity without disrupting the logical flow of the news. After segmentation, the audio segments are fed into the Whisper-Large-v3 model for ASR speech recognition. Simultaneously, the model's temperature parameters are adjusted using scene-adaptive parameters. After ASR recognition, FNR-WA also performs news-specific post-processing, standardizing numerical and date expressions and automatically recognizing unique elements such as news titles and quotations to optimize the final text output format.

[0037] (2) Audio-Text Alignment Module. This module aims to accurately establish the timestamp synchronization relationship between audio segments and the identified text content. By integrating advanced semantic matching analysis and speaker role recognition technology, this step ensures a high degree of consistency of content across the multimodal timeline, providing crucial support for the effective integration of subsequent cross-modal information.

[0038] Accurate audio-text alignment. In news video processing, accurate audio-text alignment is not only crucial for the accuracy of speech recognition but also a core prerequisite for achieving multimodal information fusion. Addressing this critical aspect, this invention proposes an audio-text alignment optimization method that incorporates news scene characteristics, building upon the traditional Montreal Forced Alignment (MFA) framework. While MFA performs well in general scenarios, it suffers from insufficient adaptability when dealing with the unique speech pace, semantic pauses, and complex background noise in news broadcasts. Therefore, this invention systematically enhances audio detection, semantic modeling, and post-alignment processing to improve the overall accuracy and robustness of the alignment. Starting with audio loading and processing, a WebRTC speech activity detection module is introduced, employing an intelligent segment merging strategy to rationally splice continuous speech segments while maintaining semantic coherence. On the text side, a BERT model is used to extract semantic vectors, dividing the text into sentence-level semantic units. During the alignment phase, BERT vectors are used to enhance the matching logic between speech and text. After alignment, the algorithm also incorporates a fine-grained post-processing mechanism for news content, accurately annotating key information and performing personalized optimization based on different speaking roles.

[0039] (3) Prompt word generation module. Based on different news topics and types, this module can dynamically construct structured, semantically driven prompt templates. The innovation lies in the introduction of a combination strategy of thought chain and CO-STAR framework, combined with prompt compression technology. This significantly enhances the model's ability to deeply understand complex contexts and effectively guide the logic of news events during the summary generation process, thus making the generated summary more in line with the professional expression norms of news reporting.

[0040] Cue word framework design. In multimodal news summarization tasks, thought chain reasoning, as a logically progressive cue word construction mechanism, can effectively guide the model to generate content according to a clear reasoning path, thereby significantly improving the accuracy and information completeness of the summary. The CO-STAR framework, on the other hand, imposes structural constraints on the output text from six dimensions—context, target, style, tone, audience, and response format—providing stable style and semantic control capabilities for the generated content. The news summarization generation process based on the CoT thought chain mechanism and the CO-STAR cue word framework includes news data input, thought chain reasoning guidance, CO-STAR framework construction, and summary generation output. Cue word compression: The metadata of the news input is structurally simplified, retaining only the most critical elements and eliminating redundant fields to significantly compress the model's input size. The algorithm semantically reorganizes the compressed information to generate optimized cue word templates adapted to different news types.

[0041] (4) News Summary Generation Module. As the core output of the entire process, the Qwen2.5-VL multimodal large language model, which has been efficiently fine-tuned using the LoRA method, is adopted. This model can comprehensively process text information from previous modules, implicitly correlated speech features through alignment, and key visual frame information. Based on these rich and aligned multimodal inputs, the model generates a preliminary news summary. The generated result will also undergo a post-processing mechanism for further polishing and optimization, aiming to ensure that the final output summary meets high standards in terms of content completeness, logical coherence, and language readability.

[0042] Fine-tuning of the Qwen2.5-VL model. In news summarization tasks, Qwen2.5-VL, as an advanced visual-language model, possesses the ability to simultaneously understand text and images. However, its pre-training phase primarily targets general domains, lacking task optimization for news contexts. Direct application can lead to summarization lacking specificity, disorganized information, and even omission of core elements. To improve its performance in multimodal news summarization tasks, this invention combines task characteristics and hardware resources, efficiently fine-tuning Qwen2.5-VL based on the LoRA method, and systematically designing aspects such as parameter configuration, training strategies, and model adaptation.

[0043] Multimodal Fusion and News Summary Generation. As the final step in the dynamic news summary generation method based on speech and visual perception, this section elaborates on the news summary generation process. This stage integrates the processing results of the preceding modules and uses a fine-tuned Qwen2.5-VL model to generate high-quality news summaries. The prepared multimodal information is input into the fine-tuned Qwen2.5-VL model. The input includes: news text processed and optimized by FNR-WA and its timestamp information, several key visual frames representing the core content of the news video, and structured cue words that are compressed and combine logical context and expression norms. The Qwen2.5-VL model uses its powerful visual language understanding capabilities to process these inputs, transforming them into semantic representations through a text encoder. Keyframes extract visual features through a visual encoder. The multimodal fusion module and attention mechanism, fine-tuned by LoRA, combine text semantics with key visual information to form a deep, cross-modal understanding of the news event. The structured cue words play a key guiding role in this process, constraining the model to think and generate according to the preset logical context and expression norms. The model generates a preliminary news summary text. The model generates a preliminary news summary text, which enters the post-processing and optimization process. The steps include: content verification and completion: automatically checking whether the summary covers the core elements of the source material and supplementing it based on contextual logic and timestamp information; structure and format standardization: adjusting the chapter structure and layout of the summary; language polishing: refining the summary text and optimizing the wording and sentence structure.

[0044] The design philosophy of this method aims to overcome the limitations of traditional static summary generation methods that rely solely on textual information. By introducing targeted audio preprocessing, precise cross-modal information alignment, and intelligent structured cue word engineering, the dynamic news summary generation method based on speech and visual perception can achieve a deeper and more comprehensive understanding of news content, thereby generating high-quality summaries that are rich in information and accurately expressed.

[0045] This invention integrates a series of key optimization steps and advanced technologies, including long audio processing optimized for news scenarios, enhanced audio-text alignment, efficient fine-tuning of large visual language models using LoRA, and structured cue word design, aiming to comprehensively improve the overall quality of automatically generated audio and video news summaries. Tests show that, compared to traditional methods, the dynamic news summary generation method based on speech and visual perception not only performs better on automatic evaluation metrics such as ROUGE-L, but also demonstrates significant advantages in the four core quality dimensions of content coverage, semantic accuracy, language fluency, and logical coherence in crucial human evaluation.

[0046] Experiments have demonstrated the robustness and generalization ability of this method across different news categories.

[0047] The results, based on the ROUGE-L metric scores on TVSum, SumMe, and a self-built news dataset, are as follows:

[0048] Compared to the traditional plain text summarization method TextRank (score 0.295), the method presented in this invention achieves a 61% improvement.

[0049] Compared to BART-base (score 0.370) based on a pre-trained model, the method in this paper achieves an improvement of approximately 28%.

[0050] Compared to the large language model ChatGLM3-6B (score 0.390), the method presented in this invention achieves a 22% improvement.

[0051] Through detailed ablation experiments, the effectiveness of each key optimization step of the model was verified. LoRA fine-tuning and long audio processing strategies were identified as the most critical factors contributing to overall performance improvement. Effective utilization of visual information and optimized alignment methods also made significant contributions. The dynamic news summarization method based on speech and visual perception demonstrated its effectiveness in addressing the challenges of dynamic, multimodal news summarization, and its application potential and feasibility in generating high-quality news summaries (characterized by high coverage, high accuracy, high fluency, and high coherence).

Claims

1. A method for generating dynamic news summaries based on speech and visual perception, characterized by the following steps: include: 1) Speech recognition and text extraction: Extract speech content from input news videos. Apply a long audio processing algorithm that is robust to fragmented noise and adapts to low whispers. Combine the inherent structural characteristics of news broadcasts with intelligent speech segmentation and scene-adaptive noise reduction mechanisms. 2) Audio-text alignment: Construct a timestamp synchronization relationship between audio segments and the identified text content. By integrating advanced semantic matching analysis and speaker role recognition technology, ensure a high degree of consistency of content across the multimodal timeline. 3) Prompt word generation: Dynamically construct structured, semantically driven prompt templates based on different news topics and types; 4) News summary generation: The Qwen2.5-VL multimodal large language model, which has been efficiently fine-tuned by the LoRA method, is used to comprehensively process text information from the preceding module, speech features with implicit associations through alignment, and key visual frame information.

2. The dynamic news summary generation method based on speech and visual perception according to claim 1, characterized in that... In step 1), the speech recognition uses a noise reduction library based on spectrum threshold technology for audio preprocessing to separate the speech signal from the background noise. The noise reduction process is optimized in a scene-aware manner based on the structural characteristics of the news scene.

3. The method for generating dynamic news summaries based on speech and visual perception according to claim 1, characterized in that... The long audio processing algorithm in step 1) includes: the intelligent structure analysis module detects key structural points in the news video, adopts an intelligent segmentation strategy, and performs adaptive segmentation based on the position of the structural points, so that the segment size is both in line with the model's processing capacity and does not disrupt the logical flow of the news. After segmentation, the audio segments are fed into the Whisper-Large-v3 model for ASR speech recognition. At the same time, the temperature parameters of the model are adjusted in combination with scene adaptive parameters. After ASR recognition, FNR-WA also performs news-specific post-processing, standardizes the expression of numbers and dates, and automatically recognizes unique elements such as news titles and quotations to optimize the final text output format.

4. The method for generating dynamic news summaries based on speech and visual perception according to claim 1, characterized in that... Step 2) includes precise audio-text alignment: starting from audio loading and processing, the WebRTC speech activity detection module is introduced, and an intelligent segment merging strategy is introduced to reasonably splice continuous speech segments to maintain semantic coherence. On the text side, semantic vectors are extracted through the BERT model to divide the text into sentence-level semantic units.

5. The dynamic news summary generation method based on speech and visual perception according to claim 4, characterized in that... During the alignment phase, BERT vectors are used to enhance the matching logic between speech and text. After alignment, the algorithm also incorporates a fine-grained post-processing mechanism for news content, accurately annotating key information and performing personalized optimization based on different speaking roles.

6. The method for generating dynamic news summaries based on speech and visual perception according to claim 1, characterized in that... Step 3) includes the design of the prompt word framework: a news summary generation process based on the CoT thinking chain mechanism and the CO-STAR prompt word framework, including news data input, thinking chain reasoning guidance, CO-STAR framework construction and summary generation output.

7. The method for generating dynamic news summaries based on speech and visual perception according to claim 1, characterized in that... Step 3) includes prompt word compression: the metadata of the news input is structurally simplified, retaining only the most critical elements and eliminating redundant fields to significantly compress the input size of the model. The algorithm performs semantic recombination on the compressed information to generate optimized prompt word templates adapted to different news types.

8. The method for generating dynamic news summaries based on speech and visual perception according to claim 1, characterized in that... Step 4) includes fine-tuning the Qwen2.5-VL model: combining task characteristics and hardware resources, the Qwen2.5-VL model is fine-tuned efficiently based on the LoRA method, and a systematic design is carried out in terms of parameter configuration, training strategy and model adaptation.

9. A method for generating dynamic news summaries based on speech and visual perception according to claim 1, characterized in that... Step 4) includes multimodal fusion and news summary generation: The prepared multimodal information is input into the fine-tuned Qwen2.5-VL model. The input includes: news text processed and optimized by FNR-WA and its timestamp information, several key visual frames representing the core content of the news video, and structured cue words that combine logical context and expression norms and are compressed. The Qwen2.5-VL model uses its powerful visual language understanding ability to process these inputs, converting them into semantic representations through a text encoder. The key frames extract visual features through a visual encoder. The multimodal fusion module and attention mechanism, fine-tuned by LoRA, combine text semantics with key visual information to form a deep, cross-modal understanding of the news event. The structured cue words play a key guiding role in this process, constraining the model to think and generate according to the preset logical context and expression norms. The model generates a preliminary news summary text.

10. A method for generating dynamic news summaries based on speech and visual perception according to claim 9, characterized in that... The model generates a preliminary news summary text, which enters the post-processing and optimization process. The steps include: content verification and completion: automatically checking whether the summary covers the core elements of the source material and supplementing it based on contextual logic and timestamp information; structure and format standardization: adjusting the chapter structure and layout of the summary; language polishing: refining the summary text and optimizing the wording and sentence structure.

Images