Video reasoning-oriented multi-modal large model enhancement fine tuning method
By constructing a video thought chain dataset and a two-stage training framework, the problems of data scarcity and visual illusion in video understanding tasks for multimodal large models are solved, and video reasoning ability and consistency that conform to human cognitive style are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2025-12-11
- Publication Date
- 2026-04-10
AI Technical Summary
Existing multimodal large models struggle to achieve complex reasoning similar to humans in video understanding tasks, suffering from problems such as scarce video chain reasoning data, reliance on superficial templates leading to a lack of flexibility, and the susceptibility of reasoning results to visual illusions.
We construct a video thinking chain dataset, and through cross-modal thinking chain repair and reinforcement learning training, combined with a reward function, guide a multimodal large model to learn video reasoning ability that conforms to human cognitive style. We adopt a two-stage training framework for supervised fine-tuning and reinforcement learning.
It breaks through the bottleneck of scarce video chain reasoning data, obtains video reasoning ability that conforms to human cognitive style, reduces visual illusions, and ensures the consistency between the reasoning process and video evidence.
Smart Images
Figure CN121835879A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to a method for enhancing and fine-tuning multimodal large models for video reasoning. Background Technology
[0002] Existing multimodal large-scale models have made significant progress in image understanding and text reasoning. However, in video understanding tasks, most models remain at the perception level, such as recognizing objects and scenes in videos, and struggle to achieve complex reasoning similar to that of humans. For example, existing models such as Video-LLaVA and Video-ChatGPT are typically trained only on simple video question-answering datasets with fine-tuning based on instructions, and then perform reasoning based on video input and text questions. The generated answers lack a clear thought process, making the models prone to visual illusions and unable to guarantee the consistency between the reasoning process and video evidence.
[0003] In addition, traditional prompt-based methods use manual template prompts, allowing large models to gradually generate thought chains based on templated prompts. This reliance on templated reasoning clues lacks flexibility and is difficult to adapt to complex and diverse video content.
[0004] In summary, the shortcomings of existing technologies are mainly reflected in: 1. The scarcity of video chain inference data limits the training effectiveness of the model.
[0005] 2. Existing model reasoning often relies on superficial templates and lacks the flexible reasoning ability of human cognitive styles.
[0006] 3. The reasoning results are prone to visual illusions and fail to guarantee consistency with video evidence. Summary of the Invention
[0007] In view of this, the present invention provides a multimodal large model reinforcement and fine-tuning method for video reasoning. The method constructs a video thinking chain dataset containing reasoning thinking chains. The large model trained based on this dataset can obtain video reasoning ability that conforms to human cognitive style.
[0008] Furthermore, by combining cross-modal thinking chain repair and filtering with video thinking chain datasets, and adding a consistency reward mechanism to the reward function, the visual illusion of the reasoning results is reduced, ensuring consistency with video evidence.
[0009] To solve the above-mentioned technical problems, the present invention is implemented as follows.
[0010] A method for enhancing and fine-tuning a multimodal large model for video reasoning includes: Step 1: Obtain the visual question answering dataset Its samples include videos, questions, and answers; Step 2: Perform frame sampling on the video samples, input the data into a multimodal large model that accepts video input, and generate a text-based structured video representation that describes the overall video and frame details. ; Step 3: Structured video representation of the sample videos The video and question are input into a language reasoning model. The model is prompted to imagine and simulate watching the entire video, focusing on understanding the segments relevant to the question. Reasoning is then performed based on the visual description, generating a thought chain including the thought process and its outcome according to a specified thought chain format. The video, question, and thought chain form a video thought chain dataset. ; Step 4: Based on the video mind chain dataset Taking a video and a question as input, the system prompts the multimodal large model to be trained to output inference results according to the described thought chain format; Using the thought chain as a label, supervised fine-tuning training is performed on a multimodal large model to obtain an intermediate video inference model. ; Step 5: Based on the visual question answering dataset For intermediate video inference models Reinforcement learning training was conducted to complete the reinforcement and fine-tuning of the multimodal large model.
[0011] Preferably, in step two, the structured video representation It includes a text description of the entire video and frame-level metadata, where the frame-level metadata includes the timestamp, text description, and key elements for each sampled frame; the key elements include entities, actions, scenes, spatial relationships, and potential interactions.
[0012] Preferably, in step three, the prompts for the language reasoning model are cognitive prompts that conform to the video reasoning thinking paradigm of human cognition. ,in: To simulate observational cues, the indicator model is based on the structured video representation. The text expresses the imagination and simulation of watching the entire video, and gains an overall understanding of what is happening; Provide task comprehension hints, instructing the model to understand the question, analyze the task type required to answer the question, and provide examples of the task type; As a selective focus cue, it instructs the model to think about and focus on segments of the video that are relevant to the input question; Visual reasoning cues instruct the model to analyze and interpret visual content based on natural visual language descriptions of what it sees in the fragments, thereby constructing an answer; For reflective responses, the model provides clear and direct answers that can be selectively incorporated for self-verification or reflection.
[0013] Preferably, the structure of the thought chain is as follows: <think>thought process< / think> <answer> Thinking Results< / answer> ".
[0014] Preferably, in step three, after the language reasoning model generates the thought chain, it further performs cross-modal thought chain repair and filters the thought chain data. The cross-modal thinking chain repair is as follows: input the video, question, and thinking chain into the second multimodal large language model, and through prompting the second multimodal large language model, compare the thinking chain with the video and question, identify inconsistencies, and correct the thinking process while maintaining the logic of the thinking chain; The filtering of the thought chain data involves comparing the thinking results in the thought chain with the sample answers and discarding samples whose thinking results do not match the answers. Visual question answering dataset The videos, questions, and repaired and filtered thought chains in the video thought chain dataset form a video thought chain dataset. .
[0015] Preferably, in the cross-modal thought chain repair, the prompts from the second multimodal large language model include: clarifying one's identity as a multimodal reasoning expert; the task is to compare and correct illusions and errors in the thought chain based on the provided video and thought chain; not significantly modifying the logic of the thought chain, and ensuring that the final conclusion remains unchanged; Specific tasks include: (1) carefully reviewing the video, questions, and thought chains; (2) identifying only reasoning steps that directly conflict with the visual presentation in the video: replacing all text-based references with direct visual observations; using visual expressions; replacing specific timestamps with broader time expressions; not rewriting steps that are already consistent with the visual content; and only replacing or correcting parts that visually contradict the content presented; and (3) ensuring that the rest of the thought chain remains consistent with the original meaning.
[0016] Preferably, the filtered thought chain data specifically includes: For structured tasks with clear labels, samples whose thought chain answers do not match the labels are directly discarded; For open-ended tasks, the CLIP-type model is used to measure the semantic consistency between the reference answer and the thought chain results, and samples whose semantic consistency between the thought results and the reference answer is lower than a set threshold are removed.
[0017] Preferably, in step four, the prompts for the multimodal large model to be trained include: The multimodal large model is clearly defined as an assistant for solving problems based on user-posed questions; the solution method involves first thinking about the reasoning process in one's mind, and then providing the answer to the user; the reasoning process and the answer are generated using a specified thought chain format, and the linguistic symbols that distinguish between the thought process and the thought result are distinguished. Specific tasks include: thinking deeply about the input problem in a way that resembles human thinking; engaging in internal dialogue using expressions used in human thinking or natural language thinking expressions; encouraging self-reflection or verification during the reasoning process; providing detailed reasoning between the linguistic symbols of the thought process; and providing the final answer between the linguistic symbols of the thought outcome. For the answer, specific output requirements will be given according to the question category.
[0018] Preferably, in step five, the reward for reinforcement learning training... Rewards based on format Rewards for accurate answers and semantic consistency reward Comprehensive results; Format rewards : Match the inference results using regular expressions to determine if they follow a predefined structure and assign binary rewards accordingly; Rewards for accurate answers Based on the task type corresponding to the question, the corresponding accuracy index is used to determine the closeness between the reasoning result and the label. The closer the closeness, the higher the accuracy of the answer. Semantic consistency reward : Use regular expressions to locate periods, extract a fixed length M of text from the first period position, and use it as a starting point for... The sentences describing the video extracted from the model's inference trajectory are encoded using a CLIP-type model's text encoder to obtain the text encoding. Sample frames are obtained from the video, and a CLIP-type image encoder is used to calculate the visual representation of each frame and then average them to obtain the video representation. According to text encoding And video indicates Similarity to determine semantic consistency reward The higher the similarity, the greater the semantic consistency reward. The stronger; Format-based rewards Rewards for accurate answers and semantic consistency reward Calculate rewards At that time, for Design a gating function to control the semantic consistency reward. Rewards are given only for accurate answers. It is only activated when the value is non-zero.
[0019] Preferably, the accuracy of the answer is rewarded. The method of obtaining it is: For multiple-choice and numerical questions, an exact matching method is used. When the reasoning result matches the label, the accuracy of the answer is rewarded. It is 1 if it is true, otherwise it is 0; For open-ended questions, the ROUGE metric is used to measure the similarity between the reasoning result and the label; the higher the similarity, the more accurate the answer is rewarded. The larger; For OCR (Optical Character Recognition) problems, the accuracy of the answer is measured by the word error rate of the inference result relative to the label. The lower the word error rate, the higher the accuracy of the answer. The larger; For regression problems, scaled relative accuracy is used as the reward for the accuracy of the answer. .
[0020] Beneficial effects: (1) This invention provides a cognitive-heuristic method for constructing video chain reasoning data, which constructs a video thinking chain dataset containing reasoning thought chains, breaking through the data bottleneck of scarce existing video chain reasoning data. A large model trained on this dataset can obtain video reasoning capabilities that conform to human cognitive styles.
[0021] (2) Since the thought chain generation process is based on text and lacks visual cues, visual illusions may occur. Therefore, this invention adds a cross-modal thought chain repair step when constructing the video thought chain dataset to avoid the occurrence of illusions.
[0022] (3) In the reinforcement learning step, the present invention introduces semantic consistency reward into the reward, which effectively reduces the visual illusion problem and ensures that the reasoning process is consistent with video evidence.
[0023] (4) This invention designs a two-stage (SFT+RL) training framework specifically for video reasoning tasks, and supervises the fine-tuning of SFT based on the video thinking chain dataset. ,by Using thought chains as labels, supervised fine-tuning training is performed on multimodal large-scale models, guiding them to learn and master thought chain reasoning methods that conform to the heuristics of human cognition; the reinforcement learning (RL) stage is based on the original visual question-answering dataset. By combining semantic consistency rewards, the visual illusion of reasoning results is reduced, ensuring consistency with video evidence. Attached Figure Description
[0024] Figure 1This is a flowchart of the video thinking chain data generation process of the present invention.
[0025] Figure 2 This is a schematic diagram of the two-stage reinforcement fine-tuning training method of the present invention. Detailed Implementation
[0026] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0027] To achieve video reasoning capabilities for multimodal large models, this invention proposes a method for enhancing and fine-tuning multimodal large models for video reasoning, specifically including a method for generating thought chain data for video reasoning and a two-stage enhancement and fine-tuning training method.
[0028] The multimodal large model reinforcement and fine-tuning method for video reasoning of the present invention includes the following steps, wherein steps 1 to 5 are video thought chain data generation methods, and steps 6 and 7 are two-stage reinforcement and fine-tuning training methods.
[0029] Step 1: Raw Data Collection: Obtaining the Visual Question Answering Dataset , The samples include videos ,question and answer .
[0030] In this step, video and video question-answering data are collected from publicly available datasets, covering tasks across multiple domains such as general, scientific, mathematical, and spatiotemporal reasoning, to construct a visual question-answering dataset for reinforcement learning. .
[0031] exist Figure 1 In the example, the sample task is to answer multiple-choice questions. What did this person do with their laptop before lying down in bed? A. Turn it on, B. Clean it, C. Organize it, D. Turn it off.
[0032] Step 2: Generate structured video representations The process involves frame sampling of the video samples, inputting them into a multimodal large model that accepts video input, and generating a structured video representation that uses text to describe both the overall video and frame details. .
[0033] In this step, the video samples are uniformly sampled at a set frequency, such as 1 FPS, to obtain sampled frames. These sampled frames are then input into a large multimodal model that can accept video input, such as Qwen3-VL, Seeed1.6-Vision, or GLM-4.5V. With prompts, the large multimodal model is instructed to generate a text-based structured video representation. This structured video representation Text is used to describe the video as a whole and the details of each frame, so that step 3 can use a language reasoning model to understand and reason about the thought process.
[0034] In a preferred embodiment, structured video representation Specifically, this includes the text description of the entire video and frame-level metadata, which can be specified to be recorded in JSON format. The frame-level JSON metadata includes the timestamp, text description, and key elements for each sampled frame; these key elements include information such as entities, actions, scenes, spatial relationships, potential interactions, and salient features.
[0035] exist Figure 1 In the example, through understanding and reasoning about the sampled frames, the video description is obtained as "A young man walks around in a messy room, transitioning from a relaxed state in bed to an active state of using a computer..."; the frame-level metadata includes the timestamp, text description, and key elements of each sampled frame, including objects, actions, scenes, and potential interactions in the sampled frame.
[0036] Step 3: Cognitive-Inspired Mind Chain Data Generation: Structured Video Representation of Sample Videos The system takes a question as input, inputs a language reasoning model, prompts the language reasoning model to imagine and simulate watching the entire video, and focuses on understanding the segments in the video that are relevant to the question. It then reasons based on the visual description and generates a thought chain that includes the thought process and the thought result according to the specified thought chain format.
[0037] In this step, we first design cognitive prompts that mimic human cognitive processes and conform to the paradigm of human cognitive reasoning. ,in: (1) Simulated observation prompts ( The indicator model is based on structured video representation. The text expresses the imagination to simulate watching the entire video, to gain an overall understanding of what is happening, and to form a preliminary high-level understanding.
[0038] (2) Task comprehension prompts ( ): Instructs the model to understand the problem, analyze the type of task required to answer the problem, and provide examples of task types, such as facts, causes, and causal relationships.
[0039] (3) Selective Focus Tips ): Instructs the model to think about and focus on segments of the video relevant to the input question; through Direct the model's attention to specific time segments in the video that are relevant to the input question, and mentally replay those segments to focus on them.
[0040] (4) Visual reasoning cues ( The indicator model constructs an answer by analyzing and interpreting visual content based on natural visual language descriptions of what is seen in a fragment; through... The prompts allow the reasoning process to be built upon visual content, encouraging the analysis of objects, actions, spatiotemporal relationships, and event transitions. Examples of descriptive language can also be provided, such as "This video shows…", "In the latter half of the video…".
[0041] (5) Reflective response prompts ( ): The model indicates that it provides clear and direct answers, and can selectively incorporate self-verification or reflection to ensure the quality of reasoning.
[0042] Then the structured video representation generated in the previous step Video issues and cognitive prompts The input is fed into a language reasoning model (e.g., DeepSeek-R1) to generate initial thought chain data. The generated thought chain data follows a specified thought chain format, including the thought process and the thought result. Figure 1 In the example, the structure of the thought chain is " <think> thought process< / think> <answer> Thinking Results< / answer> ".
[0043] An example of a hint for a language reasoning model is as follows: System prompt: You are an AI assistant responsible for helping users answer questions related to a video. When a user asks a question, you will respond by imagining yourself intently watching the video, just like a human. Your task is to understand the video content through visual and logical analysis to answer the user's questions.
[0044] Please follow the multi-step reasoning method below: 1. Simulate video viewing: Imagine you are watching the entire video from beginning to end to build a general understanding of the content.
[0045] 2. Understand the problem: Reflect on what the user is asking. Carefully consider what kind of answer is needed (e.g., facts, reasons, or comparisons).
[0046] 3. Identify relevant moments: Consider which parts of the video are most relevant to the question. Focus on these segments in your mental replay.
[0047] 4. Visual Reasoning: Describe what you "see" in these clips using natural visual language (e.g., "This video shows..." in the latter half of the video...). Analyze and interpret this visual content to construct your answer. 5. Answer thoughtfully: Provide clear and direct answers. Ensure your reasoning is consistent with the visual events you describe.
[0048] Response Guidelines: 1. Do not reveal in your output that you are answering based on text information. Use phrases that mimic watching a video to answer.
[0049] 2. Do not directly mention any textual metadata, such as “subtitles,” “descriptions,” “frame-level metadata,” “key elements,” etc. If you need to mention these, please use “visual evidence” instead (e.g., “This video shows…”).
[0050] 3. During the thinking process, you can repeatedly examine or question yourself—reflect naturally like a human being.
[0051] 4. Use broad expressions (such as "at the beginning of the video", "near the middle", or "near the end") to refer to specific points in time.
[0052] Step 4: Cross-modal mind chain repair: A key limitation of initial thought chains is that they can be visually illusionful due to the lack of visual cues in the generation of cognitively inspired thought chain data. To address this issue, this invention introduces a cross-modal optimization strategy to refine the thought chains, enabling them to better match actual video input.
[0053] Therefore, this invention inputs the video, question, and thought chain into a second multimodal large language model. By prompting the second multimodal large language model (e.g., Qwen2.5-VL), the thought chain is compared with the video and question to identify inconsistencies. While maintaining the logic of the thought chain, the thought process is corrected as necessary, and the corrected thought chain is output.
[0054] The second multimodal large language model used in this reasoning can be the same model or a different model from the multimodal large language model involved in the context.
[0055] The prompts for the second multimodal large language model may include: in the system prompts, clearly stating that one's identity is a multimodal reasoning expert; the task is to compare and correct illusions and errors in the thought chain based on the provided video and thought chain; not to significantly modify the logic of the thought chain and to ensure that the final conclusion remains unchanged. In the specific task description, clearly state (1) carefully examine the video, question and thought chain; (2) only identify reasoning steps that directly conflict with the visual presentation in the video: replace all text-based references with direct visual observations; use visual expressions; replace specific timestamps with broader time expressions; do not rewrite steps that are already consistent with the visual content; only replace or correct parts that are visually contradictory to the presented content; (3) ensure that the rest of the thought chain remains consistent with the original meaning.
[0056] An example of a suggestion from a second multimodal large language model is as follows: System prompt: You are a multimodal reasoning expert. Your task is to correct illusions and errors in your thought process based on the visual content in the provided video. Do not make significant changes to the original logic or content of your thought process, and ensure that the final conclusion remains unchanged.
[0057] Your task: 1. Carefully review the video, questions, and thought process.
[0058] 2. Only identify reasoning steps that directly conflict with the visual content presented in the video: • Replace all text-based references with direct visual observation; • Use visual vocabulary, such as "shown in the video," "I see…," or "can be obtained from the video," etc. • Replace text-based timestamp representations with broader time representations; • Do not rewrite steps that are already consistent with the visual content; • Only replace or correct parts that contradict the displayed visual content.
[0059] 3. Ensure that the rest of the thought chain remains consistent with the original meaning.
[0060] In this example, the corrected thought process is as follows: Figure 1 As shown in the orange box.
[0061] Step 5: Filter the thought chain data.
[0062] This invention incorporates a filtering stage to ensure the factual accuracy of the generated thought chain annotations. The thought results in the thought chain are compared with sample answers, and samples whose thought results do not match the answers are discarded.
[0063] Specifically: for structured tasks with clear labels, samples with incorrect final answers are directly discarded; for open-ended tasks, the semantic consistency between the reference answer and the thought chain result is measured using the CLIP-type model, and samples with semantic consistency between the thought result and the reference answer below a set threshold are removed.
[0064] Through steps 1 through 5, a large-scale video thought chain dataset can be synthesized. , Including visual question answering datasets The videos and questions included also include repaired and filtered thought chains.
[0065] Step 6: Supervised fine-tuning of training. See also Figure 2 The blue box in the middle.
[0066] This step is based on the video mind chain dataset. Taking a video and a question as input, the system prompts the multimodal large model to be trained to output inference results according to the described thought chain format; Using the thought chain as a label, supervised fine-tuning training is performed on a multimodal large model to obtain an intermediate video inference model. .
[0067] Supervised fine-tuning of large multimodal models can be achieved by calculating the cross-entropy loss function between the output inference results and the labels, and then adjusting the model parameters accordingly.
[0068] The core objectives of this training step are twofold: first, to guide the multimodal large model to learn and master a thought chain reasoning method that conforms to the heuristics of human cognition; and second, to enable it to accurately adapt to… <think> Mind Chain Content< / think> <answer> Answer< / answer> This is the specific output format. Through the above supervised fine-tuning training, an intermediate video inference model with preliminary video inference capabilities can be obtained. .
[0069] The inference cues for the multimodal large model to be trained may include: System prompt: The multimodal large model is an assistant for solving problems based on user-posed questions; the solution method is to first think about the reasoning process in one's mind, and then provide the answer to the user; the reasoning process and the answer are generated using a specified thought chain format, and the linguistic symbols that distinguish between the thought process and the thought result are distinguished. Specific tasks: Think deeply about the input problem in a way that resembles human thinking; engage in internal dialogue using expressions used in human thinking or natural language thinking expressions; encourage self-reflection or verification during the reasoning process; provide detailed reasoning between the linguistic symbols of the thought process and give the final answer between the linguistic symbols of the thought result; For the answer, specific output requirements will be given according to the question category.
[0070] An example of a hint for a large multimodal model to be trained is as follows: System prompt: This is a dialogue between the user and the assistant. The user asks a question, and the assistant answers it. The assistant needs to think through the reasoning process before providing the answer to the user. The reasoning process and the answer are placed separately in... <think>< / think> and <answer>< / answer> In the tag, an example is as follows: <think> The reasoning process is here.< / think> <answer> The answer is here.< / answer> Problem Comprehension: Please think about this problem as deeply as a human would; while thinking, use expressions like "Let me think about it," "Wait a minute," "Hmm," "Oh, I see," and "Let's analyze it step by step" to mentally deduce the problem. You can also use other thought-related expressions that conform to natural language. It is recommended to include a self-reflection or verification process in your reasoning. Please place the detailed reasoning process in...<think>< / think> In the tags, and place the final answer in <answer>< / answer> In the tag.
[0071] Answer Instructions: • Multiple choice questions: Only need to answer in <answer>< / answer> Enter a single option letter (e.g., A, B, C, D, etc.) in the label.
[0072] • Numeric type: Only needs to be in <answer>< / answer> Enter a numerical value in the label (e.g., 42 or 3.14).
[0073] • Optical Character Recognition (OCR): Please clearly transcribe the text in the image / video and fill in the text answer. <answer>< / answer> Within the tag.
[0074] • Free-response type: Please fill in your text answer in [the provided text]. <answer>< / answer> Within the tag.
[0075] • Regression analysis type: only requires... <answer>< / answer> Enter a numerical value in the label (e.g., 42 or 3.14).
[0076] Step 7: Rule-based reinforcement learning. See also Figure 2 The green box in the middle.
[0077] After obtaining the intermediate video inference model Then, in the original visual question-answering dataset The system employs Group Relative Policy Optimization (GRPO) for reinforcement learning training, thereby completing the fine-tuning of the multimodal large model reinforcement.
[0078] Among them, GRPO is a large model generation for each problem. One answer, as a candidate response Then, these responses are evaluated using a defined reward function to derive the reward. These rewards are then normalized to calculate the advantage of each response. The formula is:
[0079] Where mean and std represent the mean and standard deviation of the reward, respectively.
[0080] The model is then optimized by maximizing the following objectives:
[0081] in, This indicates the model parameters to be updated. and These are the current policy model and the old policy model, respectively. Indicates the reference strategy, It is the KL divergence regularization coefficient. It is a regularization coefficient used to prevent the strategy from deviating too much from the reference model. Expressing expectations, This indicates the calculation of the KL divergence.
[0082] This invention designs a reward function for reinforcement learning: the invention mainly includes three types of reward items: (1) Format rewards This reward ensures that the model's output follows a predefined structure. It uses regular expressions to match whether the inference results follow the predefined structure and assigns a binary reward to the inference process accordingly.
[0083] In the above example, the output of the large model must be included <think> ……< / think> Within the mark, and in <answer> ……< / answer> The tag is used to verify compliance via regular expression matching and to allocate binary rewards accordingly. For example... Figure 2 As shown, if the answer conforms to the format requirements, it will be considered correct. It is 1 if it is true, otherwise it is 0.
[0084] (2) Accuracy Reward To provide reliable supervision across different tasks, task-specific accuracy metrics were employed. Based on the task type corresponding to the question, the appropriate accuracy metric was used to determine the closeness between the inference result and the label; the closer the closeness, the higher the accuracy of the answer.
[0085] Specifically: For multiple-choice and numerical questions, an exact matching method is used. When the reasoning result matches the label, the accuracy of the answer is rewarded. It is 1 if it is true, otherwise it is 0; For open-ended questions, the ROUGE metric is used to measure the similarity between the reasoning result and the label; the higher the similarity, the more accurate the answer is rewarded. The larger; For OCR (Optical Character Recognition) problems, the accuracy of the answer is measured by the word error rate of the inference result relative to the label. The lower the word error rate, the higher the accuracy of the answer. The larger; For regression problems, scaled relative accuracy is used as the reward for the accuracy of the answer. .
[0086] (3) Semantic consistency reward :Model The generated reasoning trajectory typically consists of three distinct parts: problem analysis, video description, and abstract reasoning.
[0087] like Figure 2 As shown. To extract the sentence describing the video from the generated response, this invention uses a regular expression to locate the first period, at position [position missing]. The part following this punctuation mark corresponds to the visual description in the model's output. From this position... Begin by extracting a segment of fixed length. The text marked with 't' is denoted as ' That is, from The sentences describing the video are separated from the model's inference trajectory. A text encoder using a CLIP-type model, such as the SigLIP model, is used to process the text. Encode to obtain text encoding .
[0088] In addition, from the video Uniform sampling Frame, denoted as And calculate each frame using the SigLIP image encoder. Visual representation: .
[0089] Then, through visual representation of all frames in the video... Taking the average, we obtain the final video representation: .
[0090] Then semantic consistency reward Defined as:
[0091] in, Represents cosine similarity. It is a scaling constant. Ensure the non-negativity of rewards, and Training is stabilized by limiting rewards. This stage-aware formula is designed to reward only the reasoning parts relevant to visual understanding, without penalizing abstract reasoning that appropriately extends beyond the visual range.
[0092] The semantic consistency reward can be replaced by other visual-text alignment methods, such as CLIP, BLIP-2, etc., to calculate similarity.
[0093] Final comprehensive format reward Rewards for accurate answers and semantic consistency reward Calculate the total reward .
[0094] In an optimal solution, the final reward for a response is... The calculation is as follows:
[0095] in It is an indicator function, when It returns 1 if the condition is met, and 0 otherwise. It acts as a gating mechanism to control the semantic consistency reward. Is it only for accuracy rewards? Activation is only applied when the response is non-zero, which avoids reinforcing semantically plausible but factually incorrect reasoning. Each response in reinforcement learning is used to calculate the final reward.
[0096] This concludes the process.
[0097] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for enhancing and fine-tuning a multimodal large model for video reasoning, characterized in that, include: Step 1: Obtain the visual question answering dataset Its samples include videos, questions, and answers; Step 2: Perform frame sampling on the video samples, input the data into a multimodal large model that accepts video input, and generate a text-based structured video representation that describes the overall video and frame details. ; Step 3: Structured video representation of the sample videos The video and question are input into a language reasoning model. The model is prompted to imagine and simulate watching the entire video, focusing on understanding the segments relevant to the question. Reasoning is then performed based on the visual description, generating a thought chain including the thought process and its outcome according to a specified thought chain format. The video, question, and thought chain form a video thought chain dataset. ; Step 4: Based on the video mind chain dataset Taking a video and a question as input, the multimodal large model to be trained is prompted to output the reasoning result according to the aforementioned thought chain format; by Using the thought chain as a label, supervised fine-tuning training is performed on a multimodal large model to obtain an intermediate video inference model. ; Step 5: Based on the visual question answering dataset For intermediate video inference models Reinforcement learning training was conducted to complete the reinforcement and fine-tuning of the multimodal large model.
2. The method as described in claim 1, characterized in that, In step two, the structured video representation It includes a text description of the entire video and frame-level metadata, where the frame-level metadata includes the timestamp, text description, and key elements for each sampled frame; the key elements include entities, actions, scenes, spatial relationships, and potential interactions.
3. The method as described in claim 1, characterized in that, In step three, the prompts for the language reasoning model are cognitive prompts that conform to the video reasoning thinking paradigm of human cognition. ,in: To simulate observational cues, the indicator model is based on the structured video representation. The text expresses the imagination and simulation of watching the entire video, and gains an overall understanding of what is happening; Provide task comprehension hints, instructing the model to understand the question, analyze the task type required to answer the question, and provide examples of the task type; As a selective focus cue, it instructs the model to think about and focus on segments of the video that are relevant to the input question; Visual reasoning cues instruct the model to analyze and interpret visual content based on natural visual language descriptions of what it sees in the fragments, thereby constructing an answer; For reflective responses, the model provides clear and direct answers that can be selectively incorporated for self-verification or reflection.
4. The method as described in claim 1, characterized in that, The structure of the thought chain is as follows: <think> thought process< / think> <answer> Thinking Results< / answer> ".
5. The method as described in claim 1, characterized in that, In step three, after the language reasoning model generates the thought chain, it further performs cross-modal thought chain repair and filters the thought chain data. The cross-modal thinking chain repair is as follows: input the video, question, and thinking chain into the second multimodal large language model, and through prompting the second multimodal large language model, compare the thinking chain with the video and question, identify inconsistencies, and correct the thinking process while maintaining the logic of the thinking chain; The filtering of the thought chain data involves comparing the thinking results in the thought chain with the sample answers and discarding samples whose thinking results do not match the answers. Visual question answering dataset The videos, questions, and repaired and filtered thought chains in the video thought chain dataset form a video thought chain dataset. .
6. The method as described in claim 5, characterized in that, In the cross-modal thought chain repair process, the prompts from the second multimodal large language model include: clearly identifying oneself as a multimodal reasoning expert; the task is to compare and correct illusions and errors in the thought chain based on the provided video and thought chain; do not significantly modify the logic of the thought chain, and ensure that the final conclusion remains unchanged; Specific tasks include: (1) carefully reviewing the video, questions, and thought chains; (2) identifying only reasoning steps that directly conflict with the visual presentation in the video: replacing all text-based references with direct visual observations; using visual expressions; replacing specific timestamps with broader time expressions; not rewriting steps that are already consistent with the visual content; and only replacing or correcting parts that visually contradict the content presented; and (3) ensuring that the rest of the thought chain remains consistent with the original meaning.
7. The method as described in claim 5, characterized in that, The filtered thought chain data specifically includes: For structured tasks with clear labels, samples whose thought chain answers do not match the labels are directly discarded; For open-ended tasks, the CLIP-type model is used to measure the semantic consistency between the reference answer and the thought chain results, and samples whose semantic consistency between the thought results and the reference answer is lower than a set threshold are removed.
8. The method as described in claim 1, characterized in that, In step four, the prompts for the multimodal large model to be trained include: The multimodal large model is clearly defined as an assistant for solving problems based on user-posed questions; the solution method involves first thinking about the reasoning process in one's mind, and then providing the answer to the user; the reasoning process and the answer are generated using a specified thought chain format, and the linguistic symbols that distinguish between the thought process and the thought result are distinguished. Specific tasks include: thinking deeply about the input problem in a way that resembles human thinking; engaging in internal dialogue using expressions used in human thinking or natural language thinking expressions; encouraging self-reflection or verification during the reasoning process; providing detailed reasoning between the linguistic symbols of the thought process; and providing the final answer between the linguistic symbols of the thought outcome. For the answer, specific output requirements will be given according to the question category.
9. The method as described in claim 1, characterized in that, In step five, the reward for reinforcement learning training is... Rewards based on format Rewards for accurate answers and semantic consistency reward Comprehensive results; Format rewards : Match the inference results using regular expressions to determine if they follow a predefined structure and assign binary rewards accordingly; Rewards for accurate answers Based on the task type corresponding to the question, the corresponding accuracy index is used to determine the closeness between the reasoning result and the label. The closer the closeness, the higher the accuracy of the answer. Semantic consistency reward : Use regular expressions to locate periods, extract a fixed length M of text from the first period position, and use it as a starting point for... The sentences describing the video extracted from the model's inference trajectory are encoded using a CLIP-type model's text encoder to obtain the text encoding. Sample frames are obtained from the video, and a CLIP-type image encoder is used to calculate the visual representation of each frame and then average them to obtain the video representation. According to text encoding And video indicates Similarity to determine semantic consistency reward The higher the similarity, the greater the semantic consistency reward. The stronger; Format-based rewards Rewards for accurate answers and semantic consistency reward Calculate rewards At that time, for Design a gating function to control the semantic consistency reward. Rewards are given only for accurate answers. It is only activated when the value is non-zero.
10. The method as described in claim 9, characterized in that, The accuracy of the answer will be rewarded. The method of obtaining it is: For multiple-choice and numerical questions, an exact matching method is used. When the reasoning result matches the label, the accuracy of the answer is rewarded. It is 1 if it is true, otherwise it is 0; For open-ended questions, the ROUGE metric is used to measure the similarity between the reasoning result and the label; the higher the similarity, the more accurate the answer is rewarded. The larger; For OCR (Optical Character Recognition) problems, the accuracy of the answer is measured by the word error rate of the inference result relative to the label. The lower the word error rate, the higher the accuracy of the answer. The larger; For regression problems, scaled relative accuracy is used as the reward for the accuracy of the answer. .