Video description generation method and system based on macro-window streaming direct preference optimization

Abstract

The application discloses a video description generation method and system based on macro window stream direct preference optimization, constructs a macro window stream direct preference optimization system for optimizing a stream commentary model, first automatically constructs stream commentary timing anchor data, then decouples factual accuracy and language fluency, constructs a pair of data triplets for preference optimization, and then performs macro window stream direct preference optimization, performs DPO training based on the constructed preference triplets under a macro window, guides a strategy model to align semantics on a macro event scale, and maintains stream reasoning capability on a micro step, so that an optimized stream commentary model is obtained. The application does not need manual fine labeling data, is optimized in an end-to-end self-supervised manner, adopts low-delay stream reasoning, effectively solves the supervision granularity mismatch problem in stream generation by using a macro window alignment strategy combined with direct preference optimization, and realizes accurate capture and fact landing of key events in real-time video streams.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to intelligent video processing, specifically a video description generation method and system based on macro-window streaming direct preference optimization. Background Technology

[0002] Existing streaming video understanding and generation technologies, especially for real-time commentary of sports events, primarily rely on multimodal large language models and employ strategies such as window attention mechanisms, key-value caching reuse, and timestamp alignment supervision to attempt continuous text generation from unbounded video streams under limited GPU memory and latency constraints. However, since key events in sports events like football often span macroscopic time windows of several seconds or even longer, existing fine-grained supervised training methods based on microsteps suffer from a severe "supervision granularity mismatch" problem. This disconnect between micro-training and macro-semantics makes it difficult for models to capture the complete picture of events, easily leading to temporal drift, content gaps, and factual illusions in long-term streaming inference, such as fabricated events or incorrect descriptions. Furthermore, the streaming commentary data upon which existing technologies rely often suffers from defects such as inaccurate timestamp alignment, high ASR transcription noise, or a lack of tactical context. This not only increases the difficulty of model training but also limits the practical application of models in real-world high-dynamic, high-precision live streaming scenarios.

[0003] The core technology in this field lies in resolving the contradiction between microscopic supervisory signals and macroscopic event semantics under strict streaming inference latency constraints, efficiently extracting key event information from sparse and transient visual streams, and transforming it into time-accurate and fact-based feature representations. To address this problem, existing researchers have proposed several solutions:

[0004] (1) Patent CN202511263018.5 "A knowledge-enhanced sports video understanding method based on dual-mode reasoning of a large model": This scheme adapts to the dynamic nature and diversity of sports videos by combining reactive and deliberate dual-mode reasoning with dynamic motion segmentation, key segment selection and fine-grained matching of sports knowledge graph. However, the multi-module collaboration of its deliberate reasoning agent needs to handle spatial, temporal and spatiotemporal three-dimensional distortion and matching calculation. In ultra-long sports video scenarios, multi-level feature processing and similarity calculation can easily lead to increased reasoning latency. Moreover, the construction and updating of the sports knowledge graph depends on the accumulation of domain data, which is not suitable for mass sports projects.

[0005] (2) Patent CN202511171100.5 "A large model alignment training method and system based on direct preference optimization": This scheme solves the problems of suboptimal reference model and fixed reward margin in traditional DPO methods by constructing a static reference model and an implicit reference model, dynamically optimizing preference data and adjusting the policy model. However, the hyperparameter α of its implicit reference model needs to be debugged in combination with specific models and datasets, lacking an adaptive adjustment mechanism. Moreover, the weight calculation when dynamically filtering preference data depends on the performance of the reference model. In specific datasets such as sports, it is easily affected by the distribution characteristics of domain data, resulting in fluctuations in alignment accuracy.

[0006] (3) Patent application CN202510597487.4 "A method and system for generating sports video commentary based on a multimodal large language model": This scheme achieves deep integration of video, audio and text through the construction of a multimodal embedding space, clustering memory units and retrieval enhancement context learning. However, the initial value of the K-means cluster center of its multimodal clustering memory unit depends on the preset event category, which is prone to getting trapped in local optima; and the sparse regularized distance metric of the retrieval enhancement mechanism is sensitive to the β parameter. In scenarios with fast-paced events and multiple concurrent events, the real-time adaptability of historical instance retrieval to the current scene needs to be improved.

[0007] (4) Patent application CN202411537440.0 “A method for generating sports commentary based on a large language model”: This scheme extracts action information through a video analysis module, combines retrieval enhancement generation technology and prompt word engineering, and uses the Zhipu GLM4 model to generate commentary. However, its video analysis module relies on PP-TSM and Attention-LSTM for action recognition, which has limited accuracy in recognizing high-dynamic and heavily occluded sports actions; and the vectorization and retrieval of the commentary knowledge base rely on the Langchain framework and Chroma database. In large-scale sports data scenarios, the storage and retrieval efficiency of the knowledge base is easily affected by the increase in data volume.

[0008] However, existing technologies related to sports video streaming understanding and generation, and large model preference alignment have the following shortcomings: First, the integration of multimodal information is superficial, and the high dynamism of sports videos and the macro-temporal characteristics of event events are not fully adapted, resulting in problems such as "mismatch of supervision granularity" and "insufficient modal interaction," leading to incomplete event capture and significant temporal drift. Second, preference alignment relies on fixed reference models or manually labeled data, lacking dynamic adaptation mechanisms and end-to-end self-supervised optimization, resulting in poor domain adaptability and high deployment costs. Third, streaming inference is prone to producing factual illusions and content gaps, and it is difficult to balance computational overhead and real-time performance, failing to meet the requirements of low-latency, high-precision scenarios such as live sports broadcasts.

[0009] For example, in the aforementioned prior art:

[0010] (1) In CN202511263018.5, the scheme improves the accuracy of understanding by combining reactive and thoughtful dual-mode reasoning with sports knowledge graphs, and is suitable for both simple and complex problems. However, its thoughtful reasoning agent needs to perform spatial, temporal and spatiotemporal distortion processing and multi-class cosine similarity calculation, resulting in high latency of multi-module collaborative reasoning and limited processing efficiency in ultra-long sports video scenarios; moreover, the construction and updating of sports knowledge graphs rely on the accumulation of a large amount of domain data, which is not suitable for niche sports projects and has limited generalization ability.

[0011] (2) In CN202511171100.5, this scheme combines a static reference model with an implicit reference model to solve the problems of suboptimal traditional DPO reference models and fixed reward margins. However, the hyperparameter α of its implicit reference model needs to be manually adjusted in combination with the specific model and dataset, and it lacks an adaptive adjustment mechanism; the weight calculation when dynamically filtering preferred data depends on the performance of the reference model, and in datasets with strong specificity in fields such as sports, it is easily affected by the data distribution characteristics, resulting in fluctuations in alignment accuracy.

[0012] (3) In CN202510597487.4, the scheme achieves deep integration of video, audio and text through multimodal embedding space construction, cluster memory unit and retrieval enhancement mechanism. However, the initial value of the K-means cluster center of its multimodal cluster memory unit depends on the event category preset and is prone to getting trapped in local optima; the sparse regularized distance metric of the retrieval enhancement mechanism is sensitive to the β parameter. In the scenario of fast-paced competition and multiple concurrent events, the real-time adaptability of historical instances and current scenarios is insufficient and the consistency of the narration time sequence is difficult to guarantee.

[0013] (4) In CN202411537440.0, the scheme extracts action information through the PP-TSM model and combines retrieval enhancement generation technology and prompt word engineering to optimize the narration quality. However, its video analysis module relies on PP-TSM and Attention-LSTM for action recognition, and the accuracy of recognition for sports actions with high dynamics and severe occlusion is limited; the vectorization and retrieval of the narration knowledge base rely on the Langchain framework and Chroma database. In the scenario of large-scale event data, the storage and retrieval efficiency of the knowledge base is prone to decrease with the increase of data volume, and the scalability is insufficient. Summary of the Invention

[0014] The technical problem this invention aims to solve is that existing streaming video understanding and generation technologies, especially in the application scenario of real-time sports video commentary, suffer from problems such as "mismatch of supervision granularity," resulting in incomplete event capture and significant temporal drift. The models used rely on fixed reference models or manually labeled data, lack dynamic adaptation mechanisms and end-to-end self-supervised optimization, have poor domain adaptability and high deployment costs, and are prone to factual illusions and content gaps during inference. In addition, existing models also have the problem of balancing computational overhead and real-time performance, and cannot meet the requirements of low-latency and high-precision scenarios such as live sports broadcasts.

[0015] The technical solution of this invention is as follows: a video description generation method based on macro-window streaming direct preference optimization, which constructs a macro-window streaming direct preference optimization system to optimize the streaming narration model and perform streaming video narration, including the following steps:

[0016] Step S1: Construction of time-series anchored data: Construct time-series anchored narration data and establish the time-series correspondence between video stream segments and video narration text;

[0017] Step S2: Macro Window Trajectory Definition: Based on the aforementioned time-series anchored interpretation data, define a length of... The macro window; for any macro window, the video stream segments that have arrived before the start time of the macro window, the generated or aligned historical narration text, and the historical state maintained by the key-value cache (KV Cache) are determined as the shared streaming prefix state. ; will be from this shared streaming prefix state Starting from the macro window, the set of narration increments generated at several consecutive micro time steps within the macro window and the corresponding video segments they arrive at are defined as the macro window trajectory.

[0018] Step S3: Constructing counterfactual strong negative samples: Based on the shared streaming prefix state Constructing preference triples from macro-window trajectories ,in The positive sample trajectory is the trajectory of the real event explanation that is consistent with the video evidence and real events within the current macro window; The counterfactual strong negative sample trajectory is an explanatory trajectory constructed from explanatory texts describing different events that contradict the video evidence within the current macro window. This counterfactual strong negative sample trajectory shares the same streaming prefix state as the positive sample trajectory. ;

[0019] Step S4: Macro-window streaming direct preference optimization training: based on the preference triples constructed in step S3 In the same shared streaming prefix state And, under the condition that video segments arrive in causal order within the same macro window, calculate the positive sample trajectory respectively. and negative sample trajectory The cumulative log-likelihood difference is calculated; and a macro-window streaming direct preference optimization loss function is constructed based on the cumulative log-likelihood difference. By maximizing the preference probability of positive sample trajectories relative to negative sample trajectories, the streaming interpretation model is guided to align semantics at the macro-event scale, while maintaining streaming inference capability at the micro-step size, thus obtaining the optimized streaming interpretation model.

[0020] Furthermore, for the streaming narration model obtained through macro-window streaming direct preference optimization, a streaming causal masking mechanism and a key-value cache (KV Cache) mechanism are adopted during the deployment phase. The model receives continuously arriving video frame streams and maintains the generated historical narration text and the arrived video segments as historical context states. At each micro-time step, the model only accesses the video and text information of the current time step and before, does not access future video frames, and generates word-by-word or phrase-by-phrase based on the current and historical information, thereby achieving low-latency real-time narration output.

[0021] Furthermore, the video stream segments are sports event videos.

[0022] Furthermore, construct negative sample trajectories. At that time, the commentary trajectory is constructed by selecting commentary texts from the real commentary texts corresponding to different event types in the same game or the same video dataset.

[0023] Furthermore, in step S1, the original video and its corresponding audio are processed through an automated data processing pipeline. The automated data processing pipeline includes audio preprocessing, speech denoising, automatic speech recognition (ASR), speaker separation, and timestamp calibration. Among these, automatic speech recognition (ASR) is used to transcribe the narration in the audio into text and generate word-level or character-level timestamps. Speaker separation is used to distinguish between narrator speech, interview speech, background noise, and other non-narration speech. Timestamp calibration is used to map words or characters in the transcribed text to the video timeline, thereby establishing a precise temporal correspondence between video stream segments and narration text.

[0024] Furthermore, the automated data processing pipeline also includes quality gating filtering and entity normalization processing. The quality gating filtering includes filtering the transcribed commentary text fragments based on ASR confidence, fragment duration, word density, silence interval, repetition, speaker labels, competition relevance, and entity legitimacy, and deleting advertisements, interviews, audience noise, empty filler words, and text fragments unrelated to the current competition content; wherein, the competition relevance and entity legitimacy are determined through multi-round consistency judgment using a large language model;

[0025] The entity normalization process includes introducing a phoneme-aware entity calibration module. This module first extracts player, team, coach, referee, and event-related entities from the commentary text, converts the extracted text entities into phoneme sequences or pronunciation codes, and performs similarity matching with the standard names, aliases, and phoneme sequences of candidate entities in the match metadata. Combining edit distance, team roster, jersey numbers, match information, and contextual constraints, it determines the standard entities corresponding to the text entities. Subsequently, misidentified ASR names, misspellings, abbreviations, or aliases are replaced with standard entity names or preset entity placeholders, thus obtaining the entity-calibrated time-anchored commentary text.

[0026] This invention also provides a macro-window streaming direct preference optimization system, comprising:

[0027] Data processing pipeline module: used to perform time-series processing on video stream segments and narration text to obtain time-series anchored narration data;

[0028] Preference construction module: used to construct preference triples based on video stream segments, their narration text, and macro window trajectories;

[0029] Macro Window Optimization Training Module: This module is used to perform direct preference optimization training on the streaming interpretation model based on preference triples and macro windows, resulting in an optimized streaming interpretation model.

[0030] The video description generation method proposed in this invention, based on Macro Window Streaming Direct Preference Optimization (MWS-DPO), can construct large-scale time-series anchored data through an automated pipeline without requiring manual fine-grained annotation. It effectively solves the problem of mismatched supervision granularity in streaming generation using a macro window alignment strategy, achieving accurate capture and factualization of key events in real-time video streams. This invention has broad application prospects in streaming perception scenarios such as sports broadcasting, accessibility services, intelligent monitoring, and autonomous driving: the streaming commentary model optimized by this method can achieve automated real-time commentary for sports events, significantly reducing broadcasting costs and providing professional-grade, low-latency commentary services for niche events; it can provide high-fidelity accessibility narration for visually impaired individuals, generating accurate scene descriptions in real-time from the video stream, improving the immediacy of information acquisition; and it can also be extended to autonomous driving road condition broadcasting or real-time security monitoring alerts, accurately identifying and describing sudden key events under strict latency constraints, improving the real-time perception and interaction capabilities of intelligent systems. Its beneficial effects are:

[0031] 1) Automated construction of time-series anchored data: Through ASR transcription, phoneme-level calibration, quality-gated filtering and entity normalization, high-precision time-anchored data can be extracted from the original event broadcast without the need for manual fine annotation, solving the problems of scarce streaming data and inaccurate alignment;

[0032] 2) Macro window preference alignment mechanism: Define a macro time window of fixed length, extend single time step optimization to event-level trajectory optimization, effectively solve the problem of "supervision granularity mismatch", capture the complete semantics of sports events spanning several seconds, and reduce time drift;

[0033] 3) Construction of strong counterfactual negative samples: Through the "type mismatch mining" strategy, negative samples with fluent language but factual errors are constructed to decouple language fluency from factual accuracy, force the model to rely on visual evidence, and significantly suppress factual illusions;

[0034] 4) Low-latency streaming inference: The streaming video commentary model adopts KV cache reuse and sliding window attention mechanism, which maintains the advantages of macro window training while ensuring real-time generation of micro step size and constant memory usage, meeting the needs of low-latency scenarios such as sports live broadcast.

[0035] 5) End-to-end self-supervised optimization: Without relying on manual preference labeling, preference triples are built through an automated pipeline to achieve dynamic adaptation of the model to sports event scenarios, reduce deployment costs, and improve domain adaptability. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the method of the present invention.

[0037] Figure 2 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0038] This invention proposes a streaming video commentary generation method based on macro-window streaming direct preference optimization (MWS-DPO). A macro-window streaming direct preference optimization system is constructed to optimize the streaming commentary model for streaming video commentary, particularly suitable for sports videos. The streaming commentary model employs a multimodal large model. The video stream input is processed by a visual encoder, mapping it to high-dimensional visual semantic features. Commonly used visual encoders are typically based on the ViT architecture. Next, the visual features are mapped to the embedding space of a language model through a projection layer (MLP or Projector) to achieve modal alignment between visual stream features and language features. In the language processing part, the model uses a large language model, combining historical streaming states (KV Cache) with the current visual input to complete the understanding and commentary generation of real-time game situations.

[0039] As an example, the overall flow of the method of the present invention is as follows: Figure 1 As shown, the specific explanation is as follows.

[0040] Step S1: Time-anchored data construction

[0041] We construct a large-scale streaming commentary dataset with precise time anchors; we process the original competition broadcast video and audio through an automated pipeline, use automatic speech recognition (ASR) technology to obtain the text, and combine phoneme-level calibration technology to generate word-level timestamps, thereby establishing a precise temporal correspondence between video stream segments and commentary text, providing learnable and strong supervisory signals for low-latency commentary.

[0042] Specifically, the raw video and its corresponding audio are processed through an automated data processing pipeline. This automated data processing pipeline includes audio preprocessing, speech denoising, automatic speech recognition (ASR), speaker separation, and timestamp calibration. ASR is used to transcribe the narration in the audio into text and generate word-level or character-level timestamps. Speaker separation is used to distinguish between narrator speech, interview speech, background noise, and other non-narration speech. Timestamp calibration is used to map words or characters in the transcribed text to the video timeline, thereby establishing a precise temporal correspondence between video stream segments and the narration text.

[0043] Furthermore, to ensure data quality, the automated data processing pipeline also includes quality gating filtering and entity normalization. The quality gating filtering involves screening the transcribed commentary text fragments based on ASR confidence, fragment duration, word density, silence interval, repetition, speaker labels, competition relevance, and entity legitimacy, removing advertisements, interviews, audience noise, empty filler phrases, and text fragments unrelated to the current competition content. The competition relevance and entity legitimacy are determined through multi-round consistency checks using a large language model.

[0044] The entity normalization process includes introducing a phoneme-aware entity calibration module. This module first extracts player, team, coach, referee, and event-related entities from the commentary text, converts the extracted text entities into phoneme sequences or pronunciation codes, and performs similarity matching with the standard names, aliases, and phoneme sequences of candidate entities in the match metadata. Further, it combines edit distance, team roster, jersey numbers, information about both sides of the match, and contextual constraints to determine the standard entities corresponding to the text entities. Subsequently, misidentified ASR names, misspellings, abbreviations, or aliases are replaced with standard entity names or preset entity placeholders, thus obtaining the entity-calibrated time-anchored commentary text.

[0045] This invention proposes an automated pipeline-based mechanism for constructing time-anchored data. By integrating speech denoising, automatic speech recognition (ASR), and speaker separation technologies, a scalable data processing pipeline is established. Specifically, this invention introduces a phoneme-aware entity calibration module, which performs phoneme-level matching and normalization between identified text entities and match lists, thereby generating streaming commentary data (SoccerNarrate-Data) with accurate timestamps, contextual coherence, and entity grounding. This mechanism effectively overcomes the shortcomings of existing sports commentary data, such as high time alignment noise, numerous factual errors, and sparse supervision signals, providing high-quality time-anchored supervision for the training of streaming models.

[0046] Step S2: Define the macro window trajectory

[0047] This invention introduces a "macro-window" mechanism to address supervision noise at micro-step sizes. Based on time-series anchored interpretation data, a length of [missing information] is defined. The macro window; for any macro window, the video stream segments that have arrived before the start time of the macro window, the generated or aligned historical narration text, and the historical state maintained by the key-value cache (KV Cache) are determined as the shared streaming prefix state. ; will be from this shared streaming prefix state Starting from the macro window, the sequence of narration increments generated at several consecutive micro-time steps and the corresponding set of video segments reached are defined as the macro window trajectory. The design of the macro window extends the prediction of a single time step into a continuous narration trajectory. In subsequent direct preference optimization, all generated content and their corresponding set of video segments within the macro window are locked as the overall optimization unit, rather than optimizing only the words at a single moment.

[0048] Step S3: Constructing strong counterfactual negative samples

[0049] According to the shared streaming prefix state Constructing preference triples from macro-window trajectories ,in The positive sample trajectory is the trajectory of the real event explanation that is consistent with the video evidence and real events within the current macro window; The counterfactual strong negative sample trajectory is an explanatory trajectory constructed from explanatory texts describing different events that contradict the video evidence within the current macro window. This counterfactual strong negative sample trajectory shares the same streaming prefix state as the positive sample trajectory. Furthermore, for negative sample trajectories, it is preferable to select them from real commentary texts corresponding to different event types in the same match or the same video dataset, so that they are reasonable in terms of language style and text form, that is, they conform to the commentary style and form of the current video, but their description of events is inconsistent with the video evidence in the current macro window.

[0050] By decoupling factual accuracy from linguistic fluency, pairwise data triples are constructed for preference optimization. The construction of counterfactual strong samples aims to force the model to rely on visual evidence rather than linguistic priors for interpretive text generation.

[0051] This invention proposes a fact-based preference construction strategy based on counterfactual strong negative samples. To decouple factual accuracy from language style, this invention constructs a special preference triple. Under a shared streaming prefix, true explanations of the target event are selected as winning items (positive samples), while true explanations of the same length but with mismatched event types are selected as rejection items (negative samples). This strategy forces the model to focus on visual and textual factual consistency during preference learning, rather than simply imitating language style, significantly suppressing event illusions and temporal drift phenomena commonly found in streaming explanations.

[0052] Step S4: Macro-window streaming direct preference optimization (MWS-DPO)

[0053] Based on the macro-window trajectory definition in step S2 and the preference triplet in step S3 In the same shared streaming prefix state Given video segments arriving in causal order within the same macro window, a macro-window streaming direct preference optimization (MWS-DPO) loss function is constructed; where, Indicates the shared streaming prefix state. Represents the positive sample trajectory. This represents the trajectory of a negative sample. The loss function is:

[0054]

[0055] in, This represents the streaming interpretation model to be optimized, i.e., the policy model. Represents the reference model. Indicates the preference intensity coefficient. This represents the Sigmoid function. This represents the expected outcome. For any macro window trajectory... Its conditional log-likelihood is obtained by summing the log probabilities of the text generated at each micro-time step within the macro window, that is:

[0056]

[0057] in, For the length of the macro window, Indicates as of the date The video clip that has reached a microscopic time step. Indicates the first Historical explanatory texts prior to a micro-time step, Indicates the first The first micro-time step generated the first Each text unit. This indicates the starting micro-time step index of the current macro window. Streaming interpretation is generated step-by-step, such as generating a text increment every second. When a length of is selected When using a macro window, assuming this macro window starts from the first... Starting from a micro-time step, the time step covered by this macro window is: Therefore, in the formula This means taking from the first Step start, continuous The logarithmic probabilities of text generation within each micro-time step are summed up.

[0058] By minimizing the MWS-DPO loss function, the positive sample trajectory is made... Relative to negative sample trajectory By obtaining higher conditional generation probabilities, the streaming interpretation model can be guided to align with the semantics of real events at the macro-event scale, while maintaining causal streaming reasoning and low-latency generation capabilities at the micro-time step, resulting in an optimized streaming interpretation model.

[0059] The macro-window streaming direct preference optimization (MWS-DPO) mechanism of this invention addresses the mismatch in supervision granularity between micro-step supervision and macro-event semantics in streaming generation by decoupling the optimization granularity from the inference granularity. While maintaining the streaming inference architecture, a fixed-length macro-time window, such as 30 seconds, is defined. Preference optimization is performed by comparing the macro-window trajectories starting from the same streaming prefix state and calculating the cumulative log-likelihood difference. This strategy enables the model to capture complete event semantics spanning several seconds, effectively eliminating training instability and output gaps caused by overly localized step-level supervision.

[0060] Step S5: Hybrid Streaming Inference Step

[0061] For the streaming narration model obtained through macro-window streaming direct preference optimization, during the deployment phase, a streaming causal mask mechanism and a key-value cache (KV Cache) mechanism are adopted. The streaming narration model receives continuously arriving video frame streams and maintains the generated historical narration text and the arrived video segments as historical context states. At each micro-time step, the model only accesses the video and text information of the current time step and before, does not access future video frames, and generates word-by-word or phrase-by-phrase based on the current and historical information, thereby achieving low-latency real-time narration output.

[0062] This invention also provides a macro-window streaming direct preference optimization system, such as... Figure 2 As shown, it includes:

[0063] Data processing pipeline module: used to perform time-series processing on video stream segments and narration text to obtain time-anchored narration data; this module performs speech recognition, speaker separation and phoneme-level entity calibration through an automated pipeline to build high-precision time-aligned streaming supervision data;

[0064] Preference construction module: used to construct preference triples based on video stream segments, their narration text, and macro window trajectories;

[0065] Macro Window Optimization Training Module: This module performs direct preference optimization training on the streaming interpretation model based on preference triples and macro windows, resulting in an optimized streaming interpretation model. It constructs macro window trajectory pairs containing true facts and counterfactual hard negative samples based on shared streaming prefixes, and calculates the cumulative log-likelihood difference to optimize event-level semantics for DPO training.

[0066] In summary, this invention achieves the automated construction of high-quality temporal anchoring data for streaming commentary. By integrating a fully automated pipeline of ASR transcription, phoneme-level forced alignment, quality gating filtering, and entity normalization, it can extract millisecond-level precisely aligned "video-text" supervision data from the original event broadcast without the need for manual, meticulous timestamp annotation. This effectively solves the pain points of existing streaming data, such as time drift, noise interference, and entity errors, and significantly reduces the cost of acquiring high-quality training data.

[0067] For videos where key events span several seconds or even longer time windows, this invention also solves the problem of "supervision granularity mismatch" in streaming generation. By innovatively introducing a macro-window preference alignment mechanism, the optimization unit is expanded from micro-level single-step generation to macro-level event trajectories, such as a complete 30-second event cycle. This enables the model to capture the full picture of key events spanning several seconds, effectively suppressing temporal drift and the generation of meaningless filler content, and significantly improving the event coverage and temporal consistency of the narration.

[0068] This invention also achieves a deep decoupling between factual accuracy and linguistic fluency. By constructing strong counterfactual negative samples through a "type mismatch mining" strategy, it ensures that the negative samples are linguistically fluent but the event descriptions are completely inconsistent with the video facts. This forces the model to rely on visual evidence rather than linguistic priors for generation, fundamentally suppressing the factual illusions common in sports commentary, such as fabricated goals and misjudging event types.

[0069] This invention also balances low latency and long-term stability in streaming inference. By employing KV cache reuse, sliding window attention, and bounded memory management strategies, it maintains the advantages of macro-window training while ensuring constant memory usage and millisecond-level latency during inference. This satisfies the real-time requirements of live sports broadcasts and supports continuous processing of infinitely long video streams, making it highly valuable for engineering deployment.

Claims

1. A video description generation method based on macro-window streaming direct preference optimization, characterized in that, A macro-window streaming direct preference optimization system is constructed to optimize the streaming narration model and perform streaming video narration, including the following steps: Step S1: Construction of time-series anchored data: Construct time-series anchored narration data and establish the time-series correspondence between video stream segments and video narration text; Step S2: Macro Window Trajectory Definition: Based on the aforementioned time-series anchored interpretation data, define a length of... The macro window; for any macro window, the video stream segments that have arrived before the start time of the macro window, the generated or aligned historical narration text, and the historical state maintained by the key-value cache (KV Cache) are determined as the shared streaming prefix state. ; will be from this shared streaming prefix state Starting from the macro window, the set of narration increments generated at several consecutive micro time steps within the macro window and the corresponding video segments they arrive at are defined as the macro window trajectory. Step S3: Constructing counterfactual strong negative samples: Based on the shared streaming prefix state Constructing preference triples from macro-window trajectories ,in The positive sample trajectory is the trajectory of the real event explanation that is consistent with the video evidence and real events within the current macro window; The counterfactual strong negative sample trajectory is an explanatory trajectory constructed from explanatory texts describing different events that contradict the video evidence within the current macro window. This counterfactual strong negative sample trajectory shares the same streaming prefix state as the positive sample trajectory. ; Step S4: Macro-window streaming direct preference optimization training: based on the preference triples constructed in step S3 In the same shared streaming prefix state And, under the condition that video segments arrive in causal order within the same macro window, calculate the positive sample trajectory respectively. and negative sample trajectory The cumulative log-likelihood difference is calculated; and a macro-window streaming direct preference optimization loss function is constructed based on the cumulative log-likelihood difference. By maximizing the preference probability of positive sample trajectories relative to negative sample trajectories, the streaming interpretation model is guided to align semantics at the macro-event scale, while maintaining streaming inference capability at the micro-step size, thus obtaining the optimized streaming interpretation model.

2. The video description generation method based on macro-window streaming direct preference optimization according to claim 1, characterized in that, For the streaming narration model obtained through macro-window streaming direct preference optimization, a streaming causal masking mechanism and a key-value cache (KV Cache) mechanism are adopted during the deployment phase. The model receives continuously arriving video frame streams and maintains the generated historical narration text and the arrived video segments as historical context states. At each micro-time step, the model only accesses the video and text information of the current time step and before, does not access future video frames, and generates word-by-word or phrase-by-phrase based on the current and historical information, thereby achieving low-latency real-time narration output.

3. The video description generation method based on macro-window streaming direct preference optimization according to claim 1, characterized in that, The video stream clips are from sports events.

4. The video description generation method based on macro-window streaming direct preference optimization according to claim 1, characterized in that, Constructing negative sample trajectories At that time, the commentary trajectory is constructed by selecting commentary texts from the real commentary texts corresponding to different event types in the same game or the same video dataset.

5. The video description generation method based on macro-window streaming direct preference optimization according to any one of claims 1-4, characterized in that, In step S1, the original video and its corresponding audio are processed through an automated data processing pipeline. The automated data processing pipeline includes audio preprocessing, speech noise reduction, automatic speech recognition (ASR), speaker separation, and timestamp calibration. Among them, automatic speech recognition (ASR) is used to transcribe the narration in the audio into text and generate word-level or character-level timestamps. Speaker separation is used to distinguish between narrator speech, interview speech, background noise, and other non-narration speech. Timestamp calibration is used to map words or characters in the transcribed text to the video timeline, thereby establishing a precise temporal correspondence between video stream segments and narration text.

6. The video description generation method based on macro-window streaming direct preference optimization according to claim 5, characterized in that, The automated data processing pipeline also includes quality gating filtering and entity normalization. The quality gating filtering includes filtering the transcribed commentary text fragments based on ASR confidence, fragment duration, word density, silence interval, repetition, speaker label, competition relevance, and entity legitimacy, and deleting advertisements, interviews, audience noise, empty filler words, and text fragments unrelated to the current competition content; wherein, the competition relevance and entity legitimacy are determined through multi-round consistency judgment using a large language model; The entity normalization process includes introducing a phoneme-aware entity calibration module. This module first extracts player, team, coach, referee, and event-related entities from the commentary text, converts the extracted text entities into phoneme sequences or pronunciation codes, and performs similarity matching with the standard names, aliases, and phoneme sequences of candidate entities in the match metadata. Combining edit distance, team roster, jersey numbers, match information, and contextual constraints, it determines the standard entities corresponding to the text entities. Subsequently, misidentified ASR names, misspellings, abbreviations, or aliases are replaced with standard entity names or preset entity placeholders, thus obtaining the entity-calibrated time-anchored commentary text.

7. The video description generation method based on macro-window streaming direct preference optimization according to any one of claims 1-4, characterized in that, The length L of the macro window is 30 seconds.

8. The video description generation method based on macro-window streaming direct preference optimization according to any one of claims 1-4, characterized in that, The streaming narration model is a multimodal large model. The video stream input is processed by a visual encoder, which maps it into high-dimensional visual semantic features. The visual semantic features are then mapped to the embedding space of the language model through a projection layer to achieve modal alignment between visual stream features and language features. In the language processing part, a large language model is used to combine the key-value cache (KV Cache) of historical streaming states with the current visual input to complete the understanding and narration generation of real-time video.

9. A macro-window streaming direct preference optimization system, characterized in that, include: Data processing pipeline module: used to perform time-series processing on video stream segments and narration text to obtain time-series anchored narration data; Preference construction module: used to construct preference triples based on video stream segments, their narration text, and macro window trajectories; Macro Window Optimization Training Module: This module is used to perform direct preference optimization training on the streaming interpretation model based on preference triples and macro windows, resulting in an optimized streaming interpretation model.

Images