A long-time video preference alignment method and system based on two-stage direct preference optimization

Abstract

A long-time video preference alignment method and system based on two-stage direct preference optimization, constructing a long video preference alignment system, outputting question and answer and reasoning chain for input long video, comprising: an anchor segment generation module, constructing a long video simulation sequence through short video segment mixed arrangement, and selecting a target anchor segment to generate a preference triple; a self-reasoning long video alignment module, recursively generating scene-level subtitles for a real long video and constructing a reasoning chain question and answer, realizing cross-scene information integration; a direct preference optimization training module, using the constructed preference triple to directly optimize the strategy model, and a supporting reference model and short context visual language model. The application does not require any artificial long video annotation, improves the information integration and causal reasoning ability in long-time tasks, improves the accuracy and transparency of the answer, significantly reduces the memory and computing resource occupation, while maintaining the short video benchmark performance, and realizes efficient deployment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to long video understanding technology. It is a method and system for long temporal video preference alignment based on two-stage direct preference optimization. Background Technology

[0002] Current long-video understanding technologies primarily rely on decomposing videos into keyframes or short segments, extracting visual feature vectors, and then aligning them with large language models to achieve descriptive, question-answering, reasoning, or interactive functions within long videos. However, since videos often last several minutes or even hours, directly processing the entire video leads to high computational costs, significant temporal redundancy, and difficulties in modeling long dependencies. Furthermore, traditional methods typically rely on manually annotated long-video data, which is not only time-consuming and labor-intensive but also limits the widespread application of these models in real-world ultra-long video scenarios.

[0003] The core technology in this field lies in how to efficiently extract task-relevant key information from long videos and transform it into feature representations suitable for alignment with large language models, thereby supporting various video understanding functions such as content description, event question answering, knowledge reasoning, and interactive dialogue. To address this problem, researchers have proposed several solutions:

[0004] (1) CN119785258A "A Long Video Understanding Method Based on Iterative Hierarchical Keyframe Selection". This method is designed for long video understanding tasks. First, it performs hierarchical frame sampling on the video to obtain multi-level visual features. Then, the keyframe query module combines the video frame features with the input question and contextual semantics to generate a query vector and determine whether it is sufficient to answer the question. If not, it iteratively updates the keyframes in higher frame rate layers until keyframes related to the question are selected. Finally, it combines the keyframes with the question input multimodal large model to generate the understanding result. However, this method mainly relies on iterative updates of features layer by layer, which has a high computational cost. At the same time, the keyframe selection relies too much on the question text, which may lead to insufficient utilization of potential supplementary information in the video. Its efficiency and generalization ability are limited in ultra-long video understanding scenarios.

[0005] (2) CN120409639A "A Cross-Modal Knowledge Reasoning Method Based on a Multimodal Large Model". This method is designed for cross-modal knowledge reasoning, integrating multimodal data such as text, images, videos, and documents; combining supervised fine-tuning, adaptive attention mechanisms, and cross-language processing techniques to achieve feature extraction, interaction, and deep reasoning; and completing complex scene understanding and cross-modal semantic reasoning through complementary analysis of multi-source data and spatiotemporal modeling, thereby improving the model's logical reasoning and semantic understanding capabilities. However, its method lacks sufficient visual and linguistic modal interaction, resulting in limited ability to capture key content in long video question answering or task-oriented understanding.

[0006] (3) CN120388323A "A Sampling Method and System for Long Video Understanding" This method is designed for long video understanding. It inputs video and subtitles and divides them into semantically consistent video segments; it uses a visual encoder to map the segments and generate text summaries; it calculates the inter-frame weights of the segments and assigns different numbers of tokens according to the weights to highlight key frames; it inputs the sampled video frames and text summaries into the model to complete key frame selection and video understanding, while reducing memory overhead and maintaining information integrity. However, its method mainly relies on inter-frame weights and text summary matching to select key frames, lacking a guidance and feedback mechanism for the semantics of the problem, making it difficult to ensure that the key frame selection is highly relevant to the user's problem.

[0007] Direct Preference Optimization (DPO), a large model alignment technique proposed by Stanford University and CZ Bioohub in 2023, is an important alternative to Human Feedback-Based Reinforcement Learning (RLHF). It skips the complex reward model training process and directly optimizes model parameters using human preference data, significantly simplifying the training process while ensuring the model output matches human needs. However, directly applying standard DPO to long-video understanding faces significant technical hurdles: First, there's the "data acquisition bottleneck"—manually annotating long-video preference data, especially preference alignment pairs across scene inference chains, is extremely costly and difficult to scale. Second, there's the technical contradiction of "distorted long-context scoring": the standard DPO mechanism heavily relies on the reference model to accurately score the log-likelihood of the full-length input sequence. However, when inputting extremely long video sequences, due to the long-range decay characteristics of the self-attention mechanism, the model's attention to features in the middle of the sequence significantly decreases, leading to contextual information forgetting. This results in the reference model's score failing to accurately reflect the semantic preferences of the entire long video, instead transmitting incorrect alignment signals to the policy model and easily causing memory overflow. The three existing technologies mentioned above mainly focus on feature sampling and compression in the inference stage. None of them can solve the problem of self-supervised construction of preference data in long video scenarios, nor can they overcome the optimization instability of standard DPO in ultra-long contexts. Summary of the Invention

[0008] The technical problem this invention aims to solve is that existing long video understanding methods have the following shortcomings: First, many methods still rely on encoding the entire video or extracting hierarchical features, such as iterative keyframe selection or token sampling strategies. Although these methods can locate certain key information, they have high computational overhead and low processing efficiency, making it difficult to ensure key information coverage while also considering resource consumption. Second, some methods rely on manual annotation or single-stage processing to obtain preference or keyframe information, lacking an end-to-end self-supervised optimization mechanism, and therefore lack scalability and ease of deployment in ultra-long video scenarios.

[0009] The technical solution of this invention is: a long-time video preference alignment method based on two-stage direct preference optimization, which constructs a long-video preference alignment system for training a model to complete a long-video understanding task, including:

[0010] The first stage involves anchor segment generation. Based on a short-context visual language model, short video segments are mixed to construct a long video simulation sequence. Target anchor segments are then selected, and preference triples are generated [user query Q, correct answer]. Misleading answers ]; Direct preference optimization training is performed on the policy model;

[0011] In the second stage, self-inference long video alignment is performed. The policy model trained in the first stage generates scene-level subtitle sequences from the input long video. A long-context large language model is then invoked to construct a cross-scene user query Q and its corresponding inference chain R based on the scene-level subtitle sequences. This achieves cross-scene information integration, and the model's direct response to the user query Q in the long video is taken as the correct answer. By modifying the reasoning chain R, a flawed reasoning chain is constructed and used as a misleading answer. Then, direct preference optimization training is performed to further optimize the policy model.

[0012] The final policy model obtained through two-stage direct preference optimization is used for long-term video understanding tasks, outputting the answer and the corresponding inference chain for user queries.

[0013] Furthermore, both the strategy model and the short-context visual language model adopt a multimodal large model architecture. In this architecture, visual input is processed by a visual encoder and mapped into high-dimensional visual semantic features. These visual semantic features are then mapped to the embedding space of the large language model through a projection layer MLP to achieve alignment between visual and linguistic features. Through the aligned visual features and text input, the understanding and generation of input information are completed.

[0014] Furthermore, the method of the present invention specifically includes the following steps:

[0015] Phase 1:

[0016] Step S1.1: Generation of combined long sequences

[0017] From a supervised fine-tuned SFT dataset containing multiple short video clips and their corresponding annotations, K visually distinct short video clips are randomly selected. These clips are then interleaved and spliced ​​in a random temporal order to construct a simulated "long video" input sequence X. The total length of sequence X exceeds the short context processing capability of the short context visual language model.

[0018] Step S1.2: Constructing Anchor Questions and Misleading Answers

[0019] From the extracted K segments, one is randomly selected as the target anchor segment X_anchor; using a short-context visual language model, a user query Q is automatically generated based on the content and annotation of the anchor segment X_anchor, where the correct answer to the user query Q is... Obtained from the anchor fragment X_anchor; simultaneously, construct a semantically reasonable but content-incorrect misleading answer for the user query Q. Misleading answers This is generated by extracting information from non-anchor fragments in sequence X and rewriting them to simulate the erroneous response of the model when it locates the wrong anchor in a long sequence.

[0020] Get the preference triple [user query Q, correct answer] Misleading answers The policy model is trained using direct preference optimization by utilizing the constructed preference triples.

[0021] Phase Two:

[0022] Step S2.1: Self-inference fine-tuning

[0023] After completing the first phase of training, the policy model trained in the first phase is applied to real, unedited long videos.

[0024] First, a recursive caption generation strategy is adopted: the long video is divided into multiple scene segments according to time. When the strategy model trained in the first stage generates captions for the current scene segment, it takes the captions of all previous scene segments as context input, thereby generating a scene-level caption sequence with global coherence. Then, using a long-context large language model, based on the scene-level caption sequence, a user query Q that requires reasoning across multiple scene segments to answer and its corresponding complete reasoning chain R are automatically generated. The model's direct answer to Q in the long video is taken as the correct answer. By modifying the reasoning chain R, a flawed reasoning chain is constructed and used as a misleading answer. Finally, the preference triples constructed using long videos are used to further optimize the policy model through direct preference optimization training, enabling the policy model to learn to autonomously integrate information from long videos and generate complete and correct reasoning chains.

[0025] Step S2.2: Reasoning and Output

[0026] During the model deployment phase, for any input long video, the final policy model obtained through two-stage direct preference optimization outputs the final answer to the user query, and also outputs a detailed inference chain that supports the answer and references specific scene identifiers, so as to enhance the interpretability and credibility of the model's decision.

[0027] Furthermore, in the first phase, a dual-filtering process is implemented for data quality:

[0028] Visual similarity filtering: Use a pre-trained visual encoder to extract feature embeddings for each short video segment; calculate the cosine similarity between all non-anchor segments and anchor segments X_anchor; if the similarity of any non-anchor segment exceeds a preset threshold, discard or replace the non-anchor segment to ensure the visual uniqueness and distinguishability of the target anchor segment.

[0029] Question-specific filtering: A large language model is used to perform a specificity test on the generated user query Q. The large language model is prompted to determine whether the user query Q must depend on at least two or three visual elements in the anchor fragment X_anchor to be answered correctly. If the user query Q fails the specificity test, it is determined that the current user query Q may be answered by other fragments, and it is discarded and a new user query is generated.

[0030] Furthermore, in the first stage, during the direct preference optimization training of the policy model, the short-context visual language model is used as the reference model π_ref, in calculating the reference model's response to the correct answer. and misleading answers When calculating the log-likelihood, only the target anchor segment X_anchor is input into the reference model, that is, the log-likelihood is approximately calculated as π_ref( |X_anchor) and π_ref( | X_anchor).

[0031] Furthermore, in the first stage, during the direct preference optimization training of the policy model, a direct preference optimization (DPO) loss function L_stage1 is constructed. Loss function L_stage1 guides the policy model π_θ to maximize its affinity for the correct answer when observing the complete long sequence X. Compared to misleading answers The preferences are determined; at the same time, the supervised fine-tuning SFT loss is jointly optimized with the DPO loss using weight α, forming the total loss L = L_stage1 + α * L_SFT.

[0032] This invention also provides a long-term video preference alignment system based on two-stage direct preference optimization, comprising:

[0033] The anchor segment generation module constructs a long video simulation sequence by mixing short video segments, and selects target anchor segments to generate preference triples for the first stage of direct preference optimization training.

[0034] The self-inference long video alignment module recursively generates scene-level subtitles and constructs inference chain question-and-answer for real long videos, realizing cross-scene information integration and generating preference triples for long videos, which are used for the second stage of direct preference optimization training.

[0035] The direct preference optimization training module optimizes the policy model in stages using the constructed preference ternary components.

[0036] This invention proposes a long-video preference alignment method based on two-stage Direct Preference Optimization (DPO). Building upon the standard DPO framework, it innovatively constructs a two-stage alignment mechanism. Through two-stage DPO training—namely, anchor-driven synthesis preference optimization training in the first stage and cross-scene self-inference fine-tuning optimization training in the second stage—this method addresses the complexity and unstable training issues of traditional reinforcement learning based on human feedback. Furthermore, it overcomes the technical barriers to applying standard DPO to long-video tasks, enabling the training of multimodal large models for long-video understanding tasks and effectively overcoming the technical bottleneck of traditional DPO failure in long sequences. Experimental results show that the performance of this invention is improved from 63.6 to 64.4 on the authoritative long-video understanding benchmark VideoMME, and from 45.3 to 46.8 on LVBench, verifying the effectiveness and superiority of the proposed solution. Its advantages are:

[0037] 1) Zero-label long video training: Preference triples are generated through anchor segment synthesis in the first stage and self-inference in the second stage, respectively, without any manual long video annotation, which significantly reduces the cost of data acquisition and construction.

[0038] 2) Efficient two-stage training mechanism: The first stage generates preference triples by mixing anchor fragments and scores them in an anchor approximation manner, enabling the model to initially learn the ability to understand videos. The second stage recursively generates inference chain questions and answers on real long videos, realizing self-supervised alignment of long videos and improving the model's information integration and causal reasoning ability in long temporal tasks.

[0039] 3) Position bias suppression and accurate answers: Improve the accuracy and transparency of answers by randomizing anchor positions and using an anchor-only approximation scoring strategy.

[0040] 4) Low computational overhead and scalability: In the first stage of DPO training, the reference model only needs to process short anchor segments, which significantly reduces the consumption of GPU memory and computing resources, while maintaining the benchmark performance of short videos and achieving efficient deployment. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the framework for the two-stage direct preference optimization method of the present invention.

[0042] Figure 2 This is a flowchart illustrating a specific embodiment of the present invention.

[0043] Figure 3 This is a flowchart illustrating the first stage of the two-stage optimization process of the method of the present invention.

[0044] Figure 4 This is a schematic diagram of the long temporal video preference alignment system based on two-stage direct preference optimization according to the present invention. Detailed Implementation

[0045] This invention proposes a long video preference alignment method based on two-stage direct preference optimization (DPO), the overall process of which is as follows: Figure 1 As shown, it includes:

[0046] Stage 1, Anchor Segment Generation: Based on a short-context visual language model, short video segments are mixed to construct a long video simulation sequence. Target anchor segments are selected, and preference triples are generated [User query Q, Correct answer]. Misleading answers Direct preference optimization training is performed on the policy model; DPO training is performed on the policy model π_θ using the short context visual language model as the reference model π_ref;

[0047] The second stage, Stage 2, involves self-inference long video alignment. Utilizing the policy model trained in the first stage, it generates scene-level subtitle sequences from the input long video. It then invokes a long-context large language model to construct a cross-scene user query Q and its corresponding inference chain R based on the scene-level subtitle sequences, achieving cross-scene information integration. The model's direct response to the user query Q on the long video is then used as the correct answer. By modifying the reasoning chain R, a flawed reasoning chain is constructed and used as a misleading answer. Then, direct preference optimization training is performed to further optimize the policy model. At this point, the policy model obtained in the first stage of training is used as the reference model, and the optimized model is used as the policy model.

[0048] The final policy model obtained through two-stage direct preference optimization is used for long-term video understanding tasks, outputting the answer and the corresponding inference chain for user queries.

[0049] Both the policy model and the short-context visual language model adopt a multimodal large-scale model architecture. In this multimodal large-scale model, visual input is processed by a visual encoder, mapping it into high-dimensional visual semantic features. Commonly used visual encoders are usually based on variants of the CLIP model, such as EVA-CLIP. Next, the visual features are mapped to the embedding space of the language model through a projection layer (MLP) to achieve alignment between visual and linguistic features. In the language processing part, the model uses a large language model (such as Qwen) to understand and generate input information using aligned visual features and text input. This invention utilizes DPO training to optimize a short-context visual language model with basic video understanding capabilities into a multimodal model for long video understanding tasks through two-stage preference alignment.

[0050] As an example, the specific implementation of the present invention is as follows, such as Figure 2 As shown.

[0051] Step S1: Combinatorial Long Sequence Generation Step

[0052] From a Supervised Fine-Tuning (SFT) dataset containing multiple short video clips and their corresponding annotations (such as subtitles), K visually distinct short video clips are randomly selected. These clips are then interleaved and spliced ​​in a random temporal order to construct a simulated "long video" input sequence X, the total length of which far exceeds the model's original short context processing capability.

[0053] Step S2: Constructing Anchor Questions and Misleading Answers

[0054] From the aforementioned K segments, one is randomly selected as the target anchor segment X_anchor. A short-context visual language model is used. In this embodiment, a short-context visual language model refers to a model with conventional visual context window limitations, primarily used for processing local visual features of single-frame images or very short video segments. For example, its context processing capability is between 2K and 8K tokens, such as InternVL, LLaVA-OneVision, and similar architecture models. Based on the content and annotations of the target anchor segment, the short-context visual language model automatically generates a user query Q. The design of this query must ensure its correct answer. It can only be obtained from the anchor fragment X_anchor; at the same time, for the same query Q, construct a semantically reasonable but content-incorrect misleading answer. This misleading answer By extracting information from non-anchor segments in sequence X and rewriting them, the model's erroneous response when locating incorrect anchors in long sequences is simulated.

[0055] Step S3: Dual Filtering Step for Data Quality

[0056] As an improvement, the present invention performs quality control on the data generated in step S2 to ensure the clarity and effectiveness of the training signal:

[0057] (1) Visual similarity filtering: Use a pre-trained visual encoder (such as DINOv2) to extract the feature embedding of each video segment; calculate the cosine similarity between all non-anchor segments and anchor segments X_anchor; if the similarity of any non-anchor segment exceeds a preset threshold, such as 0.6, discard the sample or replace the non-anchor segment to ensure the visual uniqueness and distinguishability of the anchor segments.

[0058] (2) Question-specific filtering: The generated query Q is reviewed using a large language model (such as Qwen-2.5 32B). The large language model is prompted to determine whether the query Q must depend on at least two or three unique visual elements in the anchor fragment X_anchor, such as a specific object, its attributes, or an event, in order to be answered accurately. If the query Q fails this specificity test, it is determined that it may be answered by other fragments, and it is discarded and the query Q is regenerated.

[0059] Step S4: Approximate score using the reference model

[0060] As a further improvement, in the DPO training framework, to avoid the performance degradation problem caused by the "lost in the middle" effect of the frozen short context reference model π_ref when processing long sequences X, this invention introduces an "anchor-only approximation" strategy; specifically, when calculating the reference model's approximation of the correct answer... and misleading answers When calculating the log-likelihood, instead of inputting the complete long sequence X into the reference model, only the anchor segment X_anchor is input into the reference model, which is approximately calculated as π_ref( | X_anchor) and π_ref( | X_anchor), thereby ensuring the stability and reliability of the scoring benchmark and significantly reducing computational overhead;

[0061] Step S5: First Stage DPO Training

[0062] Use the preference triples filtered in step S3 Based on the approximate score from step S4, construct the first-stage DPO loss function L_stage1. This loss function guides the policy model π_θ to maximize its relevance to the correct answer when observing the complete long sequence X. Compared to misleading answers The preference of the reference model for the correct answer. and misleading answers When calculating the log-likelihood, the approximate score π_ref( from step S4) is used. |X_anchor) and π_ref( | X_anchor), that is:

[0063]

[0064] Indicates the preference intensity coefficient. This represents the Sigmoid function. This represents the expected value; simultaneously, to preserve the model's basic capabilities on short segments, the standard supervised fine-tuning (SFT) loss is jointly optimized with the DPO loss using weight α. α is preferably set to 1 within the range of 0 to 1, and in this embodiment, it is set to 1, forming the total loss:

[0065] L = L_stage1 + α * L_SFT.

[0066] Steps S1-S5 constitute the first stage of the two-stage direct preference optimization of this invention, and the process is as follows: Figure 3 As shown.

[0067] Step S6: Second-stage self-reasoning fine-tuning steps

[0068] After completing the first phase of training, the resulting policy model is applied to real, unedited long videos.

[0069] First, a recursive caption generation strategy is adopted: the long video is divided into multiple scene segments according to time. When generating captions for the current scene segment, the model takes the captions of all previous scene segments as context input, thereby generating a scene-level caption sequence with global coherence. Subsequently, a long-context large-scale language model is used to automatically generate complex queries Q and their corresponding complete inference chains R that require reasoning across multiple scene segments based on these scene captions. In this embodiment, the long-context large-scale language model refers to a model capable of processing extremely long text sequences. Its context window typically supports tens of thousands to hundreds of thousands of tokens, such as 32K to 128K or higher tokens, such as the Qwen-2.5-32B architecture model with long text comprehension capabilities, to ensure that it can completely receive and process the global information of all scene segments in the long video.

[0070] The model's direct response to Q in the complete video, obtained in the first stage, is taken as the correct answer. By modifying the inference chain R, such as deleting key scenarios or introducing irrelevant scenarios, a flawed inference chain can be constructed and used as a misleading answer. Finally, using the standard DPO loss function L_DPO, the policy model π_θ is further optimized, enabling it to learn to autonomously integrate information from long videos and generate complete and correct inference chains.

[0071] Step S7: Reasoning and Output Steps

[0072] During the model deployment phase, for any input long video, the model can not only output the final answer to the user query, but also output a detailed inference chain that supports the answer and references specific scene identifiers (such as "Scene#3"), so as to enhance the interpretability and credibility of the model's decision.

[0073] This invention also provides a long-term video understanding system based on two-stage direct preference optimization, such as... Figure 4 As shown, it includes:

[0074] The anchor segment generation module constructs a long video simulation sequence by mixing short video segments, and selects target anchor segments to generate preference triples for the first stage of direct preference optimization training.

[0075] The self-inference long video alignment module recursively generates scene-level subtitles and constructs inference chain question-and-answer for real long videos, realizing cross-scene information integration and generating preference triples for long videos, which are used for the second stage of direct preference optimization training.

[0076] The direct preference optimization training module optimizes the policy model in stages using the constructed preference ternary components.

[0077] The anchor segment generation module constructs a long video simulation sequence by mixing short video segments and selects target anchor segments to generate preference triples; the self-inference long video alignment module recursively generates scene-level subtitles for real long videos and constructs inference chain question-and-answer to achieve cross-scene information integration; the direct preference optimization training module optimizes and trains the preference triples to policy models constructed by the anchor segment generation module and the self-inference long video alignment module, which together constitute the entire long video preference alignment system.

[0078] This invention designs an anchor-driven synthetic preference triplet construction mechanism. By randomly mixing multiple short segments and designating one as the "target anchor segment," questions and answers that depend solely on the anchor segment are generated around it. Simultaneously, semantically reasonable but misleading answers are constructed based on non-anchor segments, thereby forming high-quality preference triples. This mechanism completely eliminates the reliance on manual annotation of long videos, significantly reducing data acquisition and construction costs while ensuring the accuracy of training signals.

[0079] This invention also designs a "measuring-only anchor point" reference scoring strategy. In DPO training, to avoid performance degradation when the frozen reference model processes long sequences, this invention proposes to input only the target anchor point segment into the reference model for approximate scoring, i.e., π_ref( |X_anchor) and π_ref( | X_anchor). This strategy effectively eliminates the "lost in the middle" bias caused by long input sequences, ensures the stability of the reference score, and significantly reduces computational and memory overhead, thereby improving the scalability of preference optimization.

[0080] Third, this invention also designs a recursive caption-driven cross-scene self-reasoning optimization mechanism. In the second stage of training, this invention recursively generates scene-level captions, gradually accumulating video context, and uses a large language model to generate cross-scene reasoning chains and complex questions. Subsequently, the system uses the reasoning chains to generate preferred and misleading answers, driving the model to learn multi-scene information integration and causal reasoning capabilities. This mechanism not only solves the problem of cross-scene understanding of long videos, but also enables the model to output interpretable reasoning chains, improving the transparency and credibility of the results.

[0081] This invention can be implemented based on a computer program, and the instructions or program code for implementing the methods of this invention can be written in any combination of one or more programming languages. The instructions or program code can be executed entirely on the data processor, partially on the processor, partially on the processor and partially on a remote device as a standalone software package, or entirely on a remote device or server.

Claims

1. A long-temporal video preference alignment method based on two-stage direct preference optimization, characterized by: A long-video preference alignment system is constructed to train a model for long-video understanding tasks, including: The first stage involves anchor segment generation. Based on a short-context visual language model, short video segments are mixed to construct a long video simulation sequence. Target anchor segments are then selected, and preference triples are generated [user query Q, correct answer]. Misleading answers ]; Direct preference optimization training is performed on the policy model; In the second stage, self-inference long video alignment is performed. The policy model trained in the first stage generates scene-level subtitle sequences from the input long video. A long-context large language model is then invoked to construct a cross-scene user query Q and its corresponding inference chain R based on the scene-level subtitle sequences. This achieves cross-scene information integration, and the model's direct response to the user query Q in the long video is taken as the correct answer. By modifying the reasoning chain R, a flawed reasoning chain is constructed and used as a misleading answer. Then, direct preference optimization training is performed to further optimize the policy model. The final policy model obtained through two-stage direct preference optimization is used for long-term video understanding tasks, outputting the answer and the corresponding inference chain for user queries.

2. The long-temporal video preference alignment method based on two-stage direct preference optimization according to claim 1, characterized in that: Strategy Model Both the short-context visual language model and the large-scale multimodal model architecture are adopted. In the large-scale multimodal model architecture, the visual input is processed by the visual encoder and mapped into high-dimensional visual semantic features. The visual semantic features are mapped to the embedding space of the large language model through a projection layer MLP to achieve the alignment of visual and language features. The understanding and generation of input information are completed by the aligned visual features and text input.

3. The long-temporal video preference alignment method based on two-stage direct preference optimization according to claim 1 or 2, characterized in that... Includes the following steps: Phase 1: Step S1.1: Generation of combined long sequences From a supervised fine-tuned SFT dataset containing multiple short video clips and their corresponding annotations, K visually distinct short video clips are randomly selected. These clips are then interleaved and spliced ​​in a random temporal order to construct a simulated "long video" input sequence X. The total length of sequence X exceeds the short context processing capability of the short context visual language model. Step S1.2: Constructing Anchor Questions and Misleading Answers From the extracted K segments, one is randomly selected as the target anchor segment X_anchor; using a short-context visual language model, a user query Q is automatically generated based on the content and annotation of the anchor segment X_anchor, where the correct answer to the user query Q is... Obtained from the anchor fragment X_anchor; simultaneously, construct a semantically reasonable but content-incorrect misleading answer for the user query Q. Misleading answers This is generated by extracting information from non-anchor fragments in sequence X and rewriting them to simulate the erroneous response of the model when it locates the wrong anchor in a long sequence. Get the preference triple [user query Q, correct answer] Misleading answers The policy model is trained using direct preference optimization by utilizing the constructed preference triples. Phase Two: Step S2.1: Self-inference fine-tuning After completing the first phase of training, the policy model trained in the first phase is applied to real, unedited long videos. First, a recursive caption generation strategy is adopted: the long video is divided into multiple scene segments according to time. When the strategy model trained in the first stage generates captions for the current scene segment, it takes the captions of all previous scene segments as context input, thereby generating a scene-level caption sequence with global coherence. Then, using a long-context large language model, based on the scene-level caption sequence, a user query Q that requires reasoning across multiple scene segments to answer and its corresponding complete reasoning chain R are automatically generated. The model's direct answer to Q in the long video is taken as the correct answer. By modifying the reasoning chain R, a flawed reasoning chain is constructed and used as a misleading answer. Finally, the policy model is further optimized by using the preference triples constructed from long videos and training with direct preference optimization, so that the policy model learns to autonomously integrate information from long videos and generate complete and correct reasoning chains. Step S2.2: Reasoning and Output During the model deployment phase, for any input long video, the final policy model obtained through two-stage direct preference optimization outputs the final answer to the user query, and also outputs a detailed inference chain that supports the answer and references specific scene identifiers, so as to enhance the interpretability and credibility of the model's decision.

4. The long-temporal video preference alignment method based on two-stage direct preference optimization according to claim 3, characterized in that: In the first phase, a dual-filtering process for data quality is performed: Visual similarity filtering: Use a pre-trained visual encoder to extract feature embeddings for each short video segment; calculate the cosine similarity between all non-anchor segments and anchor segments X_anchor; if the similarity of any non-anchor segment exceeds a preset threshold, discard or replace the non-anchor segment to ensure the visual uniqueness and distinguishability of the target anchor segment. Question-specific filtering: A large language model is used to perform a specificity test on the generated user query Q. The large language model is prompted to determine whether the user query Q must depend on at least two or three visual elements in the anchor fragment X_anchor to be answered correctly. If user query Q fails the specificity test, it is determined that the current user query Q may be answered by other fragments, and it is discarded, and a new user query is generated.

5. The long-temporal video preference alignment method based on two-stage direct preference optimization according to claim 3, characterized in that: In the first stage, during the direct preference optimization training of the policy model, the short-context visual language model is used as the reference model π_ref. This reference model is then used to calculate the correct answer. and misleading answers When calculating the log-likelihood, only the target anchor segment X_anchor is input into the reference model, that is, the log-likelihood is approximately calculated as π_ref( | X_anchor) and π_ref( | X_anchor).

6. The long-temporal video preference alignment method based on two-stage direct preference optimization according to claim 3, characterized in that: In the first stage, during the direct preference optimization (DPO) training of the policy model, a DPO loss function L_stage1 is constructed. This loss function L_stage1 guides the policy model π_θ to maximize its affinity for the correct answer when observing the complete long sequence X. Compared to misleading answers The preferences are determined; at the same time, the supervised fine-tuning SFT loss is jointly optimized with the DPO loss using weight α, forming the total loss L = L_stage1 + α * L_SFT.

7. The long-temporal video preference alignment method based on two-stage direct preference optimization according to claim 1, characterized in that: The long-context large-scale language model refers to a model capable of processing extremely long text sequences.

8. A long-temporal video preference alignment system based on two-stage direct preference optimization, characterized in that... include: The anchor segment generation module constructs a long video simulation sequence by mixing short video segments, and selects target anchor segments to generate preference triples for the first stage of direct preference optimization training. The self-inference long video alignment module recursively generates scene-level subtitles and constructs inference chain question-and-answer for real long videos, realizing cross-scene information integration and generating preference triples for long videos, which are used for the second stage of direct preference optimization training. The direct preference optimization training module optimizes the policy model in stages using the constructed preference ternary components.

Images