A method and system for news video topic search and segmentation based on a large model
By analyzing the semantic coherence of speech and video through multimodal large model analysis, the system automatically identifies the boundaries of news video segments and performs deep semantic matching, solving the problems of poor system generalization ability and missed detection and segmentation in existing technologies. This enables fast and accurate news video topic search and segmentation, and improves the robustness of title recognition.
Patent Information
- Application Number
- CN202610745463.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-27
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies rely on fixed feature libraries and shallow keyword matching in news video processing, resulting in poor system generalization ability and serious omissions and missed detections. They are unable to quickly respond to the needs of hot news, and the title recognition is easily interfered with, with large boundary positioning errors.
Multimodal large model analysis is used to analyze the semantic coherence of speech-to-text sequences and video sequences, automatically identify information fragment boundaries, and perform topic search and segmentation through deep semantic understanding. Semantic matching is performed in combination with natural language commands, and title correction is achieved by using image-text association.
The system has improved its adaptability to information data from different pages, enhanced the comprehensiveness of searches and processing accuracy, reduced the workload of manual review, and improved the response speed to hot news and the accuracy of headline extraction.
Smart Images

Figure CN123115459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimedia data processing technology, and in particular to a method and system for searching and segmenting news video topics based on a large model. Background Technology
[0002] With the rapid development of multimedia technology and the broadcast television information industry, massive amounts of news video data are constantly emerging. Searching for specific news topics and segmenting video clips have become crucial foundational tasks within the industry. How to quickly extract and locate the core reporting content that users are interested in from lengthy and continuous streams of news data is an application requirement that urgently needs to be met in the field of news video processing.
[0003] Traditional news video processing solutions typically employ a step-by-step, sequential approach. The system first relies on a pre-maintained database of presenter facial images and a fixed-position title bar feature database. It then uses facial feature extraction, subtitle text recognition, and scene transition analysis to segment the overall news video into multiple news items. Next, based on a manually set list of keywords, the system performs a shallow keyword text search within the segmented news items using speech recognition technology. Finally, a large amount of manual review and verification of the search results is conducted to generate the target-themed news segment.
[0004] However, the aforementioned existing technologies have significant shortcomings in practical applications. Because their segmentation process relies heavily on manually preset feature libraries that are prone to failure, and the retrieval process can only rely on fixed keywords for shallow comparison, the system not only has poor generalization ability, but also suffers from serious omissions in segmentation and detection due to a lack of understanding of the contextual logic of audiovisual content. Summary of the Invention
[0005] This invention provides a news video topic search and segmentation method and system based on a large model, which solves the defects of existing technologies that rely on fixed feature libraries and shallow keyword matching, resulting in poor system generalization ability and serious omissions in segmentation and detection. It realizes accurate video segmentation and intelligent retrieval of target segments based on deep semantic understanding.
[0006] This invention provides a method for news video topic search and segmentation based on a large model, including: Acquire news and video data to be processed, as well as natural language search commands; Extract the speech-to-text sequence and video frame sequence from the aforementioned information video data; The speech-to-text sequence and the video sequence are input into a multimodal large model. The semantic coherence of the speech-to-text sequence and the video sequence is analyzed by the multimodal large model to determine the boundary time points of each information segment in the information video data, so as to split the information video data into multiple information segments. The target information video segment is obtained by semantically matching the natural language search command with the information segment using the multimodal large model.
[0007] According to the news video topic search and segmentation method based on a large model provided by the present invention, the step of analyzing the semantic coherence of the speech-to-text sequence and the video frame sequence through the multimodal large model includes: The timing information contained in the speech-to-text sequence and the video sequence are analyzed synchronously based on the multimodal large model. Based on the multimodal joint features of the time sequence information and the video frame sequence, the topic transition logic representing the beginning, development, transition and conclusion of information events is extracted through the multimodal big model; Candidate segmentation points are set along the time axis. According to the topic switching logic, the multimodal big model fuses the speech-to-text sequence and the video sequence before and after the candidate segmentation points to obtain the multimodal context of the previous time period and the multimodal context of the next time period. The comprehensive semantic correlation between the previous multimodal context and the subsequent multimodal context is evaluated as the result of the semantic coherence analysis.
[0008] According to the news video topic search and segmentation method based on a large model provided by the present invention, after splitting the news video data into multiple news segments, the method further includes: The video frame of the news segment is identified using the multimodal large model, and image information of the title area is extracted. Local text content corresponding to the timeline of the news segment is obtained from the speech-to-text sequence. The image information and the local text content are feature-fused using the multimodal large model. Based on the semantic information of the local text content, the image interference in the image information is compared and corrected to construct the corrected image-text association information. The news title of the news segment is extracted or generated based on the image-text association information.
[0009] According to the news video topic search and segmentation method based on a large model provided by the present invention, the step of semantically matching the natural language search command and the news segment using the multimodal large model to obtain the target news video segment includes: The natural language search command is subjected to intent parsing to extract target semantic features; Based on the boundary time points, local data segments belonging to the corresponding information segments are extracted from the speech-to-text sequence and the video frame sequence, respectively, to construct a multimodal feature representation of the information segment; The target semantic features and the multimodal feature representation are jointly semantically compared using the multimodal large model to obtain the corresponding semantic matching degree. Based on the semantic matching degree, segments whose matching degree meets the preset threshold condition are selected from multiple information segments and used as the target information video segments.
[0010] According to the news video topic search and segmentation method based on a large model provided by the present invention, the step of extracting the speech-to-text sequence and video frame sequence of the news video data includes: Extract keyframes from the information video data to obtain the corresponding previous timestamps; Extract the speech-to-text sentences from the news video data and obtain the corresponding post-timestamps; Using the time axis of the post-timestamp as the main axis, the video keyframes with overlapping time intervals and the speech-to-text sentences are reconstructed into multimodal data blocks. The multiple multimodal data blocks are arranged in chronological order and transformed into an interleaved sequence with explicit timestamps, which serves as the standard data format input to the multimodal large model.
[0011] According to the news video topic search and segmentation method based on a large model provided by the present invention, determining the boundary time points of each information segment in the news video data includes: A dynamic sliding window with a fixed duration and overlapping data area is set, and the speech-to-text sequence and the video sequence are read in blocks according to the dynamic sliding window; When the multimodal large model identifies the starting logical point of the information fragment within the current sliding window, and does not identify the ending logical point at the end of the current sliding window, it generates an unclosed state feature vector and a semantic summary of the current topic. When reading the next adjacent sliding window, the unclosed state feature vector and the current topic semantic summary are input into the multimodal large model as prior conditions, and cross-window feature fusion is performed in combination with the sequence content of the data overlap area to find the termination logic point of the information segment. The time positions of the starting logic point and the ending logic point are combined to determine the boundary time point.
[0012] According to the news video topic search and segmentation method based on a large model provided by the present invention, the step of evaluating the comprehensive semantic correlation between the multimodal context of the previous time period and the multimodal context of the subsequent time period as the analysis result of the semantic coherence includes: The multimodal context of the previous time period and the multimodal context of the next time period are analyzed to extract deep content semantic features. Using the deep content semantic features as the calculation and judgment benchmark, the multimodal big model calculates the similarity between the multimodal context of the previous time period and the multimodal context of the next time period, and obtains the comprehensive semantic relevance as the analysis result of the semantic coherence.
[0013] According to the news video topic search and segmentation method based on a large model provided by the present invention, the step of parsing the natural language search command to extract the target semantic features includes: Extract basic query terms from the natural language search instructions; mine implicit related concepts with synonymous and derived attributes to the basic query terms based on the multimodal big model; fuse and expand the basic query terms and the implicit related concepts to construct the target semantic features for joint semantic comparison.
[0014] This invention also provides a news video topic search and segmentation system based on a large model, comprising: The acquisition module is used to acquire the news video data to be processed and natural language search instructions; The extraction module is used to extract the speech-to-text sequence and video frame sequence of the information video data; The analysis module is used to input the speech-to-text sequence and the video sequence into a multimodal large model, analyze the semantic coherence of the speech-to-text sequence and the video sequence through the multimodal large model, determine the boundary time points of each information segment in the information video data, and split the information video data into multiple information segments. The matching module is used to perform semantic matching between the natural language search command and the information fragment using the multimodal big data model to obtain the target information video fragment.
[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the news video topic search and segmentation method based on the large model as described above.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the news video topic search and segmentation method based on a large model as described above.
[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the news video topic search and segmentation method based on a large model as described above.
[0018] This invention provides a news video topic search and segmentation method and system based on a large-scale model. It acquires news video data to be processed and natural language search commands, extracts the speech-to-text sequence and video frame sequence of the news video data, and inputs them into a multimodal large-scale model. The multimodal large-scale model analyzes the semantic coherence of the speech and images to determine the boundary time points of each news segment to complete video segmentation. This process transforms the judgment criteria from manually preset visual features to the coherent logic of audiovisual content, thereby improving the system's adaptability to news data with different layouts or formats. After segmenting multiple news segments, the multimodal large-scale model performs semantic matching between the natural language search commands and the news segments to obtain the target news video segment. This transforms fixed keyword retrieval into a semantic comparison of intent and content, thereby improving the recall and processing accuracy of news video data. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 This is one of the flowcharts of the news video topic search and segmentation method based on a large model provided by the present invention.
[0021] Figure 2 This is the second flowchart of the news video topic search and segmentation method based on a large model provided by the present invention.
[0022] Figure 3 This is a schematic diagram of the structure of the news video topic search and segmentation system based on a large model provided by the present invention.
[0023] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0025] In existing broadcast television news topic search and segmentation solutions, the common implementation involves two independent stages. The first stage is news item segmentation. The system pre-builds a database of presenter faces and a database of title board features. The title board feature database records the fixed coordinates of the title board within the frame. During processing, facial recognition technology is used to detect the presenter's image, subtitle recognition technology is used to locate the title board area, and scene transition detection algorithms are combined to analyze shot transitions, thereby marking the start and end times of each news item in a continuous news program, completing automatic segmentation. The second stage is topic search. Based on a manually set keyword list, the system performs keyword matching on the speech recognition text of each segmented news item, filtering out items relevant to the topic. Both stages require manual intervention for result review and correction, ultimately forming a list of topic news items.
[0026] In practical applications, the aforementioned solution, besides known issues such as reliance on manually pre-set libraries and limitations in keyword semantics, also suffers from inherent constraints in its processing flow. Since the segmentation stage requires waiting for the complete program stream input before analysis can begin, and the search stage depends on the output results after segmentation, the entire process exhibits a strictly sequential relationship. This means that from the original video input to the production of thematic entries, no intermediate results can be output in advance, and the processing delay is directly related to the video length, making it difficult to meet the needs of rapid response to trending news. Furthermore, traditional OCR technology is easily affected by factors such as font style, background texture, and lighting changes when recognizing title board text, resulting in a high error rate. Relying solely on camera transitions or face detection to determine entry boundaries significantly increases boundary localization errors in complex reporting scenarios such as host interviews and multi-guest dialogues.
[0027] To address the aforementioned technical limitations, this invention proposes a method and system for news video topic search and segmentation based on a large-scale model. This method no longer relies on any manually pre-set feature library. Instead, it synchronously inputs the speech-to-text sequence and video sequence of news programs into a multimodal large-scale model. Utilizing the model's ability to analyze semantic coherence, it automatically identifies the start and end boundaries of each news item, thus segmenting it into segments. Simultaneously, users can directly describe their topic requirements using natural language. The large-scale model performs deep semantic matching between the description and the segmented news fragments to directly locate the target topic segment. Furthermore, for individual news items after segmentation, this method also leverages the visual cognitive capabilities of the multimodal large-scale model to fuse the title area image with the speech-to-text of the corresponding time period, accurately extracting or generating the news title. The technical solution of this invention will be described in detail below with reference to the accompanying drawings.
[0028] Before describing the technical solutions of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained illustratively.
[0029] News video data: refers to continuously recorded broadcast television news programs or similar news video streams, which include various types of information such as images, audio, and subtitles, and is the main processing object of the method of this invention.
[0030] Natural language search commands: User-inputted topic queries in everyday natural language, rather than a fixed list of keywords.
[0031] Speech-to-text sequence: A series of text sentences ordered by time, obtained by automatically recognizing speech (ASR) the speech content in news video data, with each sentence associated with a corresponding timestamp.
[0032] Video frame sequence: A sequence of image frames extracted from news video data in chronological order, usually including keyframes or consecutive frames, used to present visual content.
[0033] Multimodal large model: A deep learning model capable of processing multiple modal information such as text, image, and speech simultaneously, and able to understand cross-modal semantic relationships. In this invention, it is used to analyze semantic coherence, perform semantic matching, and generate titles.
[0034] Boundary time points: The start and end times of each independent news segment in the news video data, used to split a continuous program stream into multiple news segments.
[0035] Topic transition logic: The multimodal big data model identifies the semantic transition pattern of one report ending and another beginning through semantic analysis of the content before and after the current period, which is reflected in the development and transition of the content theme.
[0036] Headline area: A fixed or semi-fixed image area in a news video frame used to display the headline of the news story. It is usually located at the bottom or side of the screen and includes the headline text and background.
[0037] Image information: Visual content extracted from the title panel area, including raw pixel-level data such as text shape, background texture, and layout.
[0038] Local text content: A subset of speech-transcribed text corresponding to the timeline of a certain information segment, used to assist in title recognition or semantic understanding.
[0039] Image-text association information: The enhanced representation formed by feature fusion, comparison and error correction of the image information in the title area and the speech-to-text content of the news is used to accurately extract or generate news titles.
[0040] Local data segment: A subset of data that is extracted from a complete speech-to-text sequence or video sequence based on boundary time points and belongs to a single information segment.
[0041] Target news video clips: Thematic news video clips that meet the matching degree requirements are selected after the natural language search commands are semantically matched with various news clips by a multimodal large model.
[0042] Multimodal data block: The basic data structure obtained by recombining video keyframes with overlapping time intervals and speech-to-text sentences with features, with explicit timestamps, serving as the standard data format for inputting large multimodal models.
[0043] Deep content semantic features: After the multimodal big model parses the multimodal context of the preceding and following time periods, it extracts abstract and high-order content meaning representations, which are used to calculate the comprehensive semantic relevance.
[0044] Basic query vocabulary: The raw words directly parsed from the natural language search commands entered by the user are the basis for further expanding implicit related concepts.
[0045] The execution entity of the news video topic search and segmentation method based on a large model provided in this invention can be a system with data storage and computing capabilities, such as a personal computer, server, workstation, cloud server, or dedicated processing equipment in a broadcast television production and broadcasting system. This system includes at least a processor, memory, and an input interface for reading video data. In practical deployment, this method can be integrated into a television station's news production management system or converged media processing platform, operating as a functional module for automated topic segmentation and search. For ease of description, the execution entity will be uniformly referred to as the processing system in the following embodiments.
[0046] Figure 1This is a flowchart illustrating the news video topic search and segmentation method based on a large model provided in an embodiment of the present invention. The method includes: Step 101: Obtain the information video data to be processed and the natural language search instructions.
[0047] The processing system first acquires raw news video data and natural language search commands input by the user. The news video data can be a continuous program file from a broadcast television signal acquisition card, local storage media, or network stream, such as a 30-minute news broadcast.
[0048] Natural language search commands are topic-specific queries entered by users in everyday expressions, such as "recent news reports about the application of artificial intelligence in the medical field." The system receives the command through a graphical user interface or API and saves it as a string format required for subsequent processing.
[0049] Step 102: Extract the speech-to-text sequence and video frame sequence from the information video data.
[0050] The processing system performs parallel analysis on the acquired video data. On one hand, it uses an Automatic Speech Recognition (ASR) engine to extract the audio stream from the video and convert it into a speech-to-text sequence with timestamps, where each text sentence corresponds to its start and end times in the video. On the other hand, it decodes the video stream and extracts keyframes at fixed intervals to form a sequence of video frames, each keyframe also associated with a corresponding timestamp. The system can optionally maintain the original timeline correspondence between the two sequences for subsequent synchronous processing.
[0051] Step 103: Input the speech-to-text sequence and the video sequence into a multimodal large model. Analyze the semantic coherence of the speech-to-text sequence and the video sequence using the multimodal large model to determine the boundary time points of each information segment in the information video data, so as to split the information video data into multiple information segments.
[0052] The processing system inputs both the speech-to-text sequence and the video sequence into a pre-deployed multimodal large-scale model. This model can simultaneously understand information from both text and image modalities. The model analyzes the input content segment by segment along the timeline, assessing semantic coherence between consecutive time periods: if there is a significant shift in theme, event, or logic between a preceding segment (e.g., a news report) and a subsequent segment (e.g., the next news report), it indicates that this is the boundary of a news segment. Based on this, the model marks the specific time corresponding to each boundary point on the timeline, including the start and end points. Finally, the processing system segments the original news video data into multiple independent news segments based on these boundary time points.
[0053] Step 104: Use the multimodal large model to perform semantic matching between the natural language search command and the information fragment to obtain the target information video fragment.
[0054] For each segment of information extracted, the processing system inputs its corresponding speech-to-text and video segments back into the multimodal big data model, along with the acquired natural language search commands. The multimodal big data model extracts the semantic features of the commands and the multimodal semantic features of each information segment, and then calculates the semantic matching degree between the two. The higher the matching degree, the more the content of the information segment matches the user's described topic requirements.
[0055] The system filters out eligible news clips based on a preset matching threshold, outputting them as the final target news video clips. The output format can be a list of clip time ranges or a directly edited video file.
[0056] The news video topic search and segmentation method based on a large model provided in this invention acquires the news video data to be processed and natural language search instructions, extracts the speech-to-text sequence and video frame sequence of the news video data, and inputs them into a multimodal large model. The multimodal large model is used to analyze the semantic coherence of speech and images to determine the boundary time points of each news segment to complete the video segmentation. This process changes the judgment criteria from manually preset visual features to the coherent logic of audiovisual content, thereby improving the system's adaptability to news data of different layouts or formats. After segmenting multiple news segments, the multimodal large model performs semantic matching between natural language search instructions and news segments to obtain target news video segments. Fixed keyword retrieval is transformed into semantic comparison of intent and content, thereby improving the recall and processing accuracy of news video data.
[0057] In some implementations, the operation of extracting the speech-to-text sequence and the video sequence in step 102 above can be implemented in the following ways.
[0058] The processing system first decodes the video stream of the information video data, extracting keyframes at a rate of 1 frame per second or based on scene switching detection results, and recording the preceding timestamp of each keyframe. For example, in a 30-second video, the system extracts keyframes at positions such as 1.0 second, 1.5 seconds, and 2.0 seconds, with each keyframe accompanied by its own time position information.
[0059] Meanwhile, the processing system uses an automatic speech recognition engine to process the audio stream and generate speech-to-text sentences with post-timestamps. For example, if the engine outputs the sentence "This morning, the company held a press conference," with a start time of 0.8 seconds and an end time of 3.2 seconds, then the post-timestamp of the sentence is recorded as [0.8, 3.2].
[0060] Then, the system processes each text sentence sequentially, using the time axis of the subsequent timestamp as the main axis. For the time interval [0.8, 3.2] of the sentence above, the system searches for all keyframes whose preceding timestamps fall within this interval, i.e., keyframes at positions such as 1.0 second, 1.5 seconds, and 2.0 seconds. The system reassembles the visual features of these keyframes with the semantic features of the text sentence to construct a multimodal data block. This data block simultaneously contains the text information "the company is holding a press conference" and continuous visual content within the corresponding time period.
[0061] For the next text sentence, such as "The meeting focused on discussing traffic congestion management solutions", with a time interval of [3.2, 6.5], the system also searches for keyframes with timestamps between 3.2 seconds and 6.5 seconds to construct a second multimodal data block.
[0062] Finally, the system arranges all these multimodal data blocks in chronological order according to the text sentences, and adds an explicit timestamp to each data block to form an interleaved sequence. This interleaved sequence is the standard data format for subsequent input into the multimodal large model.
[0063] Through the above methods, textual and visual information are precisely aligned in the temporal dimension. For example, in the above example, the text "news conference" is bound to the keyframes of the corresponding time period, avoiding misalignment between text and visuals during model processing. The standardized interleaved sequence enables large multimodal models to read and process data block by block in chronological order without requiring additional data alignment operations within the model. This improves the model's processing efficiency and the accuracy of cross-modal semantic understanding, laying a high-quality data foundation for subsequent semantic coherence analysis and boundary recognition.
[0064] In some implementations, step 103 above analyzes the semantic coherence of the speech-to-text sequence and the video sequence using a multimodal large model. This can be achieved in the following ways: Upon receiving a speech-to-text sequence and a video sequence, the multimodal large model first analyzes both modalities simultaneously. The model not only reads the semantic content of each sentence in the text sequence and its accompanying temporal information, but also simultaneously parses the visual semantic content of each frame in the video sequence. For example, when processing a news program, the model simultaneously focuses on the text describing an earthquake in a certain location and the visual elements shown in the footage, such as collapsed buildings and rescue workers.
[0065] Based on the above synchronous analysis, the model utilizes the multimodal joint features of temporal information and video footage to automatically extract the topic transition logic that represents the beginning, development, transition, and conclusion of a news event. This logic reflects the inherent pattern of the beginning, development, conclusion, and transition to the next event in a complete news report. For example, by learning from a large number of news programs, the model can identify the typical topic transition pattern of "anchor's closing remarks combined with scene transitions."
[0066] After extracting the topic transition logic, the model sets multiple candidate segmentation points along the timeline at certain intervals. These candidate segmentation points can be every second or every two seconds. For each candidate segmentation point, the model fuses the speech-to-text and video footage from a continuous time period preceding that point to generate the multimodal context for the previous time period; simultaneously, it fuses the speech-to-text and video footage from a continuous time period following that point to generate the multimodal context for the next time period. For example, at the 15-second candidate segmentation point, the model fuses all text and footage from seconds 0 to 15 into the previous context, and fuses all text and footage from seconds 15 to 30 into the next context.
[0067] The model then evaluates the overall semantic relevance between the multimodal contexts of the preceding and following time periods, using this relevance as the result of the semantic coherence analysis. If the two sections describe the same news event—for example, the preceding section recounts the earthquake, while the following section continues with disaster statistics and rescue progress—the model outputs a high degree of relevance. Conversely, if the preceding section describes the earthquake, while the following section abruptly switches to a sports event, the model outputs a low degree of relevance. The model typically measures relevance as a numerical value between 0 and 1.
[0068] Specifically, the model performs deep analysis of the multimodal context of the preceding and following time periods, extracting high-order, deep semantic features. These features are not shallow text keywords or color distributions in images, but rather abstract semantic representations of event themes, interpersonal relationships, scene types, etc. Taking the aforementioned earthquake report as an example, the deep features of the preceding context might include topic tags such as "natural disaster," "earthquake," and "casualties," while the deep features of the following context might be "sports event" or "football score," resulting in a significant difference between the two.
[0069] The model uses extracted deep semantic features as the calculation criterion to measure the similarity between the multimodal contexts of the preceding and following time periods. Similarity can be calculated using vector cosine distance, Euclidean distance, or other metrics, ultimately yielding a comprehensive semantic relevance score. This score is typically quantified as a value between 0 and 1; a higher value indicates greater semantic coherence between the two segments, while a lower value suggests a greater likelihood of semantic breakup. The model outputs this score as the result of its semantic coherence analysis.
[0070] Through the above methods, the multimodal large model does not simply rely on shallow features such as camera switching or silence detection, but rather uses deep semantic features to determine whether a topic has shifted based on similarity judgment. This makes boundary recognition more accurate, especially in complex scenarios such as live broadcasts, multi-host dialogues, and breaking news interruptions. It can still correctly distinguish the start and end of different news items, effectively reducing the probability of missed or incorrect segmentation, increasing the proportion of directly usable segmentation results, and thus reducing the workload of subsequent manual review.
[0071] In some implementations, the operation of determining the boundary time points of each information segment in step 103 above can be implemented using a cross-window analysis method based on a dynamic sliding window to address the context length limitation problem in long video processing.
[0072] The processing system first sets a dynamic sliding window with a fixed duration, for example, 30 seconds, and sets an overlap area between adjacent windows, for example, an overlap of 5 seconds. Following this window setting, the system reads the speech-to-text sequence and video sequence in blocks, sending the data block within the current window into a multimodal large model for analysis each time.
[0073] When a multimodal large model identifies the starting point of a news segment within the current sliding window—for example, when the model detects that the host has said, "Let's look at a news story about..." and the corresponding title appears on the screen—the model determines that this is the beginning of a new segment. However, if the content of the news segment is long, and the model has not detected the ending point of the segment before the current window closes (i.e., no topic transition or end marker is found), the model generates an unclosed state feature vector and a semantic summary of the current topic. This feature vector and the summary together describe the core content of the current unfinished segment, such as "Topic: New traffic regulations implemented in a certain area; Current state: Introducing specific traffic restriction measures."
[0074] When the system reads the next adjacent sliding window, it no longer starts analyzing from scratch. Instead, it uses the unclosed state feature vector generated by the previous window and the semantic summary of the current topic as prior conditions, inputting them into the multimodal large model. Simultaneously, because there are overlapping regions between adjacent windows, the model can perform cross-window feature fusion by combining the sequence content within the overlapping regions (e.g., the last 5 seconds of the previous window and the first 5 seconds of the next window). In this way, the model can continue the recognition state of the previous window and continue searching for the termination logic point of the information segment in subsequent windows. For example, the model continues analyzing in a new window until it detects the host saying "That concludes the report of this news" or the screen switches to the headline of the next news item, at which point the model marks the termination logic point.
[0075] Finally, the system combines the time position of the starting logical point identified in the first window with the time position of the ending logical point identified in the subsequent windows to form a complete boundary time point pair, which is the start and end time of the information segment.
[0076] Through the aforementioned sliding window and cross-window state transfer methods, the system can process news videos of arbitrary length without being limited by the input length of the multimodal large model. Meanwhile, the overlapping region ensures a smooth transition of information between adjacent windows, and the transfer of unclosed state feature vectors avoids information loss due to window truncation, ensuring complete recognition of long news clips and thus improving the accuracy and robustness of boundary time point determination.
[0077] In some implementations, after step 103 above splits the news video data into multiple news segments, the processing system further performs a news headline extraction operation on each independent news segment.
[0078] The processing system first inputs the video footage corresponding to the news clip into a multimodal large model, which then identifies the title bar area in the video footage. The title bar area is typically located at the bottom or side of the screen and has a specific background color or border. The model automatically frames this area using its visual positioning capabilities and extracts the original image information from it. For example, in a news clip, the title bar area might display the text "Artificial Intelligence Conference held in this city," but due to the fancy font or slight background interference, direct reading might result in errors.
[0079] At the same time, the processing system extracts the local text content corresponding to the timeline of the information segment from the previously extracted speech-to-text sequence. For example, if the segment is 25 seconds long, the system extracts all the text sentences recognized within those 25 seconds, which may include: "Yesterday morning, the city's Artificial Intelligence Conference opened at the convention center, with more than 100 companies participating."
[0080] Then, the system inputs the image information of the caption board area together with the local text content corresponding to the segment into the multimodal large model. The model performs feature fusion on the two, and uses the complete semantic information in the local text to compare and correct the caption text in the image information. For example, the model may recognize through OCR that the text in the image is "This city holds an artificial intelligence conference", where the character "zhi" (wisdom) is misrecognized as "hui" (wisdom); at this time, the model compares with "artificial intelligence conference" in the local text, and infers that the correct caption on the caption board should be "This city holds an artificial intelligence conference". Through such comparison and correction, the model constructs corrected image-text association information, which integrates the visual recognition result and the text semantic verification result.
[0081] Finally, the processing system extracts an accurate news caption based on the above image-text association information, or directly generates a caption that summarizes the core content of the news item. The output caption can be used as metadata of the news segment for subsequent retrieval, archiving or display.
[0082] Through the above method, the conventional pure OCR recognition method is prone to errors due to interferences such as fonts, light, and backgrounds, while the present application uses the speech-to-text text in the same segment as a semantic reference, and performs joint reasoning and correction through the multimodal large model, which significantly improves the accuracy and robustness of caption extraction. Even if the text in the caption board area is partially occluded or deformed, the system can still obtain the correct caption in combination with the speech content, thereby reducing the workload of manual checking and caption correction.
[0083] In some implementations, the semantic matching of the natural language search instruction and the news segment performed by the multimodal large model in step 104 described above may specifically be implemented in the following manner: The processing system first performs intent parsing on the natural language search instruction input by the user, and extracts target semantic features. Taking the user input "recent reports on the application of artificial intelligence in the medical field" as an example, the system parses the instruction through the multimodal large model, extracts basic semantic elements such as "artificial intelligence", "medical care", "application", and "recent" therefrom, and further excavates related implicit concepts, such as "AI-assisted diagnosis", "smart medical care", and "intelligent consultation", and finally forms a rich target semantic feature vector oriented to semantic matching.
[0084] Simultaneously, based on the boundary time points determined in step 103, the processing system extracts local data segments belonging to each news fragment from the original speech-to-text sequence and video frame sequence. For example, if the boundary of a news fragment is from second 12 to second 45, the system extracts all text sentences within this time period from the speech-to-text sequence and all keyframes within this time period from the video frame sequence, combining the two to construct a multimodal feature representation of the fragment. This representation includes both the specific semantic information of the report content and visual information such as scenes and people in the video.
[0085] Then, the processing system inputs the target semantic features along with the multimodal feature representations of each information segment into a large multimodal model, which performs joint semantic comparison. The comparison process is not simply calculating the overlap of text keywords, but rather delving into whether the deep semantics of both sides are related. For example, if a user searches for "the application of artificial intelligence in the medical field," the model matches the multimodal features of one segment (containing the text "doctors use AI systems to analyze CT images" and a scene showing a doctor operating a computer) with it, resulting in a high semantic match. For another segment (the text "the application of artificial intelligence in Go games"), even though it contains the keyword "artificial intelligence," the model gives a lower match because the subject matter does not involve medicine.
[0086] Finally, the processing system selects segments from all information clips that meet a preset matching threshold, such as 0.7, and outputs these segments as the final target information video clips. The output results can be sorted from highest to lowest matching score, allowing users to prioritize the most relevant content.
[0087] By employing the above methods, the processing system no longer relies on manually exhaustively listing keyword lists. Instead, it utilizes a multimodal large model to perform deep semantic understanding and comparison of natural language instructions and information fragments. Even if the topic description contains synonyms, hypernyms, or implicit concepts, the processing system can still accurately identify and recall relevant content, thereby significantly improving the recall and precision of topic searches and reducing the risk of missed detections due to differences in semantic expression.
[0088] Furthermore, the aforementioned operations of parsing the intent of natural language search commands and extracting target semantic features can be implemented in the following ways: The processing system first receives a natural language search command input by the user, such as "recent news reports related to smart agriculture". The system extracts basic query terms from this command, including "recent", "smart agriculture", and "news". These basic query terms are the core keywords directly expressed by the user, but relying solely on these terms for matching can easily miss semantically similar or related news reports.
[0089] Based on these basic query terms, the system utilizes a multimodal large model for semantic mining, automatically expanding to include implicitly related concepts with synonymous and derived attributes. For example, for "smart agriculture," the model can uncover synonymous concepts such as "intelligent agriculture" and "precision agriculture," as well as derived concepts such as "drone plant protection," "agricultural IoT," "remote sensing monitoring," and "intelligent irrigation." For "recent," the model can assign a time range threshold, such as the last three days or the last week.
[0090] Subsequently, the system integrates and expands the original basic query terms with the mined implicit related concepts to construct rich and clearly hierarchical target semantic features. These features are no longer limited to the original words input by the user, but form a set of concepts covering multiple expressions. For example, the final target semantic feature vector includes multiple related semantic dimensions such as "smart agriculture," "intelligent agriculture," "precision agriculture," "drone plant protection," and "agricultural IoT."
[0091] The processing system uses the target semantic features as the benchmark for subsequent joint semantic comparison and performs matching calculations with the multimodal feature representations of each information fragment.
[0092] By employing the above methods, even if the user's search query is relatively short or uses a specific expression, the system can still cover related news content with different expressions but similar meanings through semantic expansion. This effectively overcomes the inherent semantic gap problem of keyword search, further improves the recall rate of topic search, and reduces the omission of related content due to differences in expression.
[0093] Furthermore, the aforementioned news video topic search and segmentation method based on large models can be executed based on real-time data streams. The processing system continuously acquires news video data input in the form of real-time data streams, such as live program streams from broadcast television signal acquisition cards or online live streams.
[0094] During the reception of real-time data streams, the system integrates the steps of determining boundary time points and splitting the information video data into multiple information segments in step 103 with the step of semantically matching natural language search commands with information segments using a multimodal large model in step 104 to obtain target information video segments. Specifically, the system does not need to wait for the entire program stream to be received before starting segmentation, nor does it need to wait for all segments to be completed before searching. While receiving the data stream, the system simultaneously identifies the boundary points that have been reached in the video content and cuts out the completed information segments, while immediately performing semantic matching of these generated information segments with the user's preset natural language search commands. The segmentation and search positioning actions are executed synchronously and in parallel.
[0095] Through the aforementioned streaming processing method, the system achieves end-to-end processing without the need for full segmentation and waiting. When the topic content that a user is interested in appears in the middle or later part of the program stream, the system can locate and output the content in a very short time after its arrival, without waiting for all the preceding programs to be processed. This significantly reduces processing latency and meets the business needs of rapid discovery of trending news and real-time topic search.
[0096] like Figure 2 As shown, this embodiment takes the real-time processing of broadcast television news programs as an example to provide a detailed description of the news topic search and segmentation method based on a multimodal large model provided by the present invention.
[0097] After the processing system starts, it first continuously acquires real-time video data in the form of a broadcast television signal acquisition card. The system performs audio extraction and video frame extraction operations in parallel: on the one hand, it separates the audio stream from the data stream, and on the other hand, it extracts video keyframes at fixed intervals to form the audio and video data to be processed.
[0098] The system feeds the extracted audio stream into an automatic speech recognition engine, generating a speech-to-text sequence with timestamps in real time. Simultaneously, the system arranges keyframes of the video in chronological order, forming a video frame sequence. The speech-to-text sequence and the video frame sequence together constitute the foundational data for the input multimodal large-scale model.
[0099] The system uses a dynamic sliding window approach to segment the above sequence. The window length is set to 30 seconds, with adjacent windows overlapping by 5 seconds. Each time, the data block within the current window is sent to the multimodal large model. During the reception of the real-time data stream, the system integrates the segmentation and search operations, eliminating the need to wait for the entire program stream to be received.
[0100] For the data block within the current window, the system first inputs it into a multimodal big data model and performs semantic matching against the user's pre-entered natural language search command. For example, if the user inputs a command like "recent news reports related to smart agriculture," the system performs intent parsing, extracts basic query terms, and mines synonyms and derived concepts to construct target semantic features. The multimodal big data model then performs a joint semantic comparison between the target semantic features and the multimodal features of the current data block to determine whether the data block contains news items related to the search command.
[0101] If the semantic matching degree is lower than a preset threshold, the system determines that the current data block does not contain relevant topic content and continues processing the next data block. If the semantic matching degree reaches or exceeds the threshold, the system determines that the current data block contains relevant topic content and triggers fine-grained segmentation processing of the data block and its associated data blocks.
[0102] During the segmentation phase, a multimodal large model analyzes the semantic coherence of the current data block and adjacent window data blocks. The model evaluates the thematic changes in the speech-to-text and video footage segment by segment along the timeline, determining the start and end boundary time points for each news segment. For long news articles spanning multiple windows, the system uses unclosed state feature vectors and thematic semantic summaries to achieve cross-window state transfer, ensuring the integrity of boundary recognition. Based on the identified boundary time points, the system segments the continuous program stream into multiple independent news segments.
[0103] For each news segment obtained after segmentation, the system further performs a title extraction operation. A multimodal large model identifies the title area in the segment's image and extracts image information; simultaneously, it extracts local text content corresponding to the segment's timeline from the speech-to-text sequence. The model fuses the image information and local text content, uses text semantics to compare and correct the image recognition results, constructs corrected image-text association information, and extracts or generates accurate news titles accordingly.
[0104] The system outputs the extracted news snippets, news titles, and semantic matching results. All outputs undergo a manual review process, with reviewers making final confirmations on the accuracy of boundaries and the correctness of titles. Once confirmed to be error-free, a list of special news items is generated.
[0105] The system continues to process subsequent data blocks until all data blocks have been processed, at which point the entire process ends.
[0106] Through the integrated streaming processing approach described above, the system simultaneously performs segmentation and search operations while receiving real-time data streams, achieving end-to-end processing without waiting for full segmentation. When the topic content that a user is interested in appears in the latter part of the program stream, the system can locate and output the content in a very short time after its arrival, without waiting for all preceding programs to be processed. This significantly reduces processing latency and meets the real-time requirements of broadcast television news topic search and segmentation services.
[0107] The following describes the news video topic search and segmentation system based on a large model provided by the embodiments of the present invention. The news video topic search and segmentation system based on a large model described below can be referred to in correspondence with the news video topic search and segmentation method based on a large model described above.
[0108] This invention provides a news video topic search and segmentation system based on a large model. See [link to relevant documentation]. Figure 3 ,include: The acquisition module 310 is used to acquire the information video data to be processed and natural language search instructions; Extraction module 320 is used to extract the speech-to-text sequence and video frame sequence of the information video data; Analysis module 330 is used to input the speech-to-text sequence and the video sequence into a multimodal large model, analyze the semantic coherence of the speech-to-text sequence and the video sequence through the multimodal large model, determine the boundary time points of each information segment in the information video data, and split the information video data into multiple information segments. The matching module 340 is used to perform semantic matching between the natural language search command and the information fragment using the multimodal large model to obtain the target information video fragment.
[0109] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a news video topic search and segmentation method based on a large model. This method includes: acquiring news video data to be processed and natural language search instructions; extracting speech-to-text sequences and video frame sequences from the news video data; inputting the speech-to-text sequences and video frame sequences into a multimodal large model, analyzing the semantic coherence of the speech-to-text sequences and video frame sequences through the multimodal large model, determining the boundary time points of each news segment in the news video data to segment the news video data into multiple news segments; and performing semantic matching between the natural language search instructions and the news segments through the multimodal large model to obtain the target news video segment.
[0110] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the news video topic search and segmentation method based on a large model provided by the above methods. The method includes: acquiring news video data to be processed and natural language search instructions; extracting speech-to-text sequences and video frame sequences from the news video data; inputting the speech-to-text sequences and video frame sequences into a multimodal large model, analyzing the semantic coherence of the speech-to-text sequences and video frame sequences through the multimodal large model, determining the boundary time points of each news segment in the news video data, so as to segment the news video data into multiple news segments; and performing semantic matching between the natural language search instructions and the news segments through the multimodal large model to obtain target news video segments.
[0112] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the news video topic search and segmentation method based on a large model provided by the above methods. The method includes: acquiring news video data to be processed and natural language search instructions; extracting speech-to-text sequences and video frame sequences from the news video data; inputting the speech-to-text sequences and video frame sequences into a multimodal large model, analyzing the semantic coherence of the speech-to-text sequences and video frame sequences through the multimodal large model, determining the boundary time points of each news segment in the news video data, so as to segment the news video data into multiple news segments; and performing semantic matching between the natural language search instructions and the news segments through the multimodal large model to obtain target news video segments.
[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for news video topic search and segmentation based on a large model, characterized in that, include: Acquire news and video data to be processed, as well as natural language search commands; Extract the speech-to-text sequence and video frame sequence from the aforementioned information video data; The speech-to-text sequence and the video sequence are input into a multimodal large model. The semantic coherence of the speech-to-text sequence and the video sequence is analyzed by the multimodal large model to determine the boundary time points of each information segment in the information video data, so as to split the information video data into multiple information segments. The target information video segment is obtained by semantically matching the natural language search command with the information segment using the multimodal large model.
2. The news video topic search and segmentation method based on a large model according to claim 1, characterized in that, The step of analyzing the semantic coherence of the speech-to-text sequence and the video sequence using the multimodal large model includes: The timing information contained in the speech-to-text sequence and the video sequence are analyzed synchronously based on the multimodal large model. Based on the multimodal joint features of the time sequence information and the video frame sequence, the topic transition logic representing the beginning, development, transition and conclusion of information events is extracted through the multimodal big model; Candidate segmentation points are set along the time axis. According to the topic switching logic, the multimodal big model fuses the speech-to-text sequence and the video sequence before and after the candidate segmentation points to obtain the multimodal context of the previous time period and the multimodal context of the next time period. The comprehensive semantic correlation between the previous multimodal context and the subsequent multimodal context is evaluated as the result of the semantic coherence analysis.
3. The news video topic search and segmentation method based on a large model according to claim 1, characterized in that, After splitting the information video data into multiple information segments, the method further includes: Based on the multimodal large model, the video frame of the information segment is identified, and image information of the title plate area is extracted; Extract local text content from the speech-to-text sequence that corresponds to the timeline of the information segment; The image information and the local text content are fused using the multimodal large model. Based on the semantic information of the local text content, the image interference in the image information is compared and corrected to construct the corrected image-text association information. The title of the news segment is extracted or generated based on the image and text association information.
4. The news video topic search and segmentation method based on a large model according to claim 1, characterized in that, The step of semantically matching the natural language search command with the information fragment using the multimodal large model to obtain the target information video fragment includes: The natural language search command is subjected to intent parsing to extract target semantic features; Based on the boundary time points, local data segments belonging to the corresponding information segments are extracted from the speech-to-text sequence and the video frame sequence, respectively, to construct a multimodal feature representation of the information segment; The target semantic features and the multimodal feature representation are jointly semantically compared using the multimodal large model to obtain the corresponding semantic matching degree. Based on the semantic matching degree, segments whose matching degree meets the preset threshold condition are selected from multiple information segments and used as the target information video segments.
5. The news video topic search and segmentation method based on a large model according to claim 1, characterized in that, The steps of extracting the speech-to-text sequence and video frame sequence of the information video data include: Extract keyframes from the information video data to obtain the corresponding previous timestamps; Extract the speech-to-text sentences from the news video data and obtain the corresponding post-timestamps; Using the time axis of the post-timestamp as the main axis, the video keyframes with overlapping time intervals and the speech-to-text sentences are reconstructed into multimodal data blocks. The multiple multimodal data blocks are arranged in chronological order and transformed into an interleaved sequence with explicit timestamps, which serves as the standard data format input to the multimodal large model.
6. The news video topic search and segmentation method based on a large model according to claim 1, characterized in that, Determining the boundary time points of each information segment in the information video data includes: A dynamic sliding window with a fixed duration and overlapping data area is set, and the speech-to-text sequence and the video sequence are read in blocks according to the dynamic sliding window; When the multimodal large model identifies the starting logical point of the information fragment within the current sliding window, and does not identify the ending logical point at the end of the current sliding window, it generates an unclosed state feature vector and a semantic summary of the current topic. When reading the next adjacent sliding window, the unclosed state feature vector and the current topic semantic summary are input into the multimodal large model as prior conditions, and cross-window feature fusion is performed in combination with the sequence content of the data overlap area to find the termination logic point of the information segment. The time positions of the starting logic point and the ending logic point are combined to determine the boundary time point.
7. The news video topic search and segmentation method based on a large model according to claim 2, characterized in that, The step of evaluating the comprehensive semantic correlation between the previous time period multimodal context and the subsequent time period multimodal context as the analysis result of semantic coherence includes: The multimodal context of the preceding time period and the multimodal context of the following time period are analyzed to extract deep content semantic features; Using the deep content semantic features as the calculation criterion, the multimodal big model calculates the similarity between the multimodal context of the previous time period and the multimodal context of the next time period, and obtains the comprehensive semantic relevance as the analysis result of the semantic coherence.
8. The news video topic search and segmentation method based on a large model according to claim 4, characterized in that, The step of parsing the natural language search command to extract target semantic features includes: Extract the basic query terms from the natural language search instructions; Based on the multimodal large model, implicit related concepts with synonymous and derived attributes to the basic query terms are mined; The basic query terms are fused and expanded with the implicit related concepts to construct the target semantic features for joint semantic comparison.
9. A news video topic search and segmentation system based on a large model, characterized in that, include: The acquisition module is used to acquire the news video data to be processed and natural language search instructions; The extraction module is used to extract the speech-to-text sequence and video frame sequence of the information video data; The analysis module is used to input the speech-to-text sequence and the video sequence into a multimodal large model, analyze the semantic coherence of the speech-to-text sequence and the video sequence through the multimodal large model, determine the boundary time points of each information segment in the information video data, and split the information video data into multiple information segments. The matching module is used to perform semantic matching between the natural language search command and the information fragment using the multimodal big data model to obtain the target information video fragment.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the news video topic search and segmentation method based on a large model as described in any one of claims 1 to 8.
