A multi-modal model fine-tuning and reasoning method and system for pure visual question answering

Abstract

The application discloses a pure visual question-answering-oriented multimodal model fine-tuning and reasoning method and system, relates to the technical field of multimodal model training and reasoning, and comprises the following steps: acquiring original video and supervision data; rendering question text into a question image and embedding the question image into a video frame; sequentially performing preprocessing and frame sampling and inputting the video frame into a multimodal large model to obtain a video-level visual representation; constructing a supervision target sequence; taking the supervision target sequence as an output expectation to trigger structural supervision fine-tuning of the multimodal large model to obtain a fine-tuned multimodal large model; acquiring a video to be reasoned; performing preprocessing, frame sampling, visual feature extraction and time sequence feature aggregation on the video to be reasoned to obtain a video-level visual representation to be reasoned; inputting the video-level visual representation to be reasoned into the fine-tuned multimodal large model to obtain a reasoning result; and stable reading-to-answering reasoning is realized without independent text question input.

Description

Technical Field

[0001] This application belongs to the field of multimodal model training and inference technology, and specifically relates to a method and system for fine-tuning and inference of multimodal models for pure visual question answering. Background Technology

[0002] In recent years, multimodal large-scale models have significantly improved the joint modeling capabilities of vision and language through large-scale image-text alignment pre-training and instruction fine-tuning, gradually developing into general-purpose models. However, the default input paradigm of mainstream visual-language models is an image or video plus an independent text question, which is feasible on standard benchmarks, but there are gaps in real-world scenarios: in applications such as screen screenshots, screen recordings, and inspection videos, key semantics are often directly embedded in the image in the form of visual text, and the model must locate and understand these texts from pixels in order to complete inference.

[0003] Existing technologies for purely visual question answering have significant shortcomings. First, video question answering models are not trained with visually embedded questions. When the question exists only in video frames, it is difficult to recover the question content, leading to guessed answers or irrelevant outputs, resulting in significant performance degradation. Second, the commonly used two-stage process of optical character recognition followed by question answering suffers from error propagation. Recognition errors caused by factors such as video compression, motion blur, small text size, or brief display cannot be corrected by subsequent question answering stages. Third, supervised fine-tuning only optimizes the final answer, lacking explicit constraints on the question recovery process. The model tends to learn shortcuts directly from visual input to the answer, resulting in unstable inference output structures. Typical failures include repeating question text, outputting only scene descriptions, or failing to switch from reading the question to answering. Fourth, existing image-level purely visual question answering methods use triggered structural supervision for question alignment fine-tuning, but they do not address video temporal issues, lacking frame sequence selection, question display time control, multi-frame feature fusion, and temporal inference mechanisms. In summary, there is an urgent need for an end-to-end method for video-level pure visual question answering, which can achieve stable reasoning from reading the question to answering without independent text question input. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method for fine-tuning and inference of a multimodal model for purely visual question answering, comprising the following steps: The process involves acquiring raw video and supervised data, including question text and answer labels. The question text is rendered into question images, which are then embedded into video frames of the original video to form a video-level composite input. This video-level composite input undergoes preprocessing and frame sampling to obtain a sampled frame sequence. This sequence is then input into the visual encoder of a multimodal large-scale model to extract visual features from each frame. These visual features are then mapped to the language model embedding space via a projection layer. Temporal feature aggregation is performed on the mapped visual features to obtain a video-level visual representation. A supervised target sequence is constructed, consisting of question text, pre-defined trigger markers, and answer labels concatenated sequentially. Using the video-level composite input as the input to the multimodal large-scale model and the supervised target sequence as the output expectation, the multimodal large-scale model undergoes trigger-based structural supervised fine-tuning to obtain the fine-tuned multimodal large-scale model. The reasoning stage process is as follows: acquire the video to be reasoned, which contains a visual question text; preprocess the video to be reasoned, perform frame sampling, visual feature extraction and temporal feature aggregation to obtain the video-level visual representation to be reasoned; input the video-level visual representation to be reasoned into the fine-tuned multimodal large model to obtain the output sequence, and extract the content after the trigger mark from the output sequence as the reasoning result.

[0005] Furthermore, the methods for embedding the problematic image into the video frames of the original video include any of the following: overlay rendering, image stitching, frame border annotation, or region replacement.

[0006] Furthermore, the title embedding method used to embed the problem image into the video frame of the original video includes any of the following: Static full-process question image: The question image is continuously embedded and displayed in a fixed position throughout the entire video session; Local time window question: The question image is embedded in the corresponding frame only within a preset time window of the video; Dynamically moving question image: The embedded position of the question image moves over time according to a preset trajectory; Perturbation-robust problem statement: While embedding the problem image, retain the natural text originally present in the video frame, or apply at least one of the perturbations of compression, noise, and blur to the embedded video frame to simulate complex real video scenes.

[0007] Furthermore, frame sampling includes: For the static, full-process question paper, uniform sampling is used; For questions within a local time window, the sampling density is increased to the preset level within the time interval in which the question appears; For dynamically moving question texts, continuous frame sampling is used along the time period where the question text moves. For interference-robust question text, for scenes that are originally present in the video frame, the video frame in which the original text and the question text image are in the same frame is selected as the sampling frame; for scenes that are perturbed after embedding the video frame, the video frames before, during and after the perturbation are retained respectively.

[0008] Furthermore, frame sampling can be performed using any of the following methods: uniform sampling of a fixed number of frames, sliding window sampling, sampling of keyframes plus neighboring frames, temporal pyramid sampling, or short segment coding.

[0009] Furthermore, the temporal feature aggregation method employs splicing, pooling, cross-attention mechanisms, or a pre-defined temporal encoder.

[0010] Furthermore, the preset trigger flags can explicitly distinguish between the question recovery stage and the answer generation stage.

[0011] Furthermore, the multimodal large model includes a visual encoder, a connected projection layer, and a language decoder.

[0012] Furthermore, the trigger-based structure-supervised fine-tuning employs low-rank adaptation fine-tuning, prompt-based optimization, partial parameter unfreezing fine-tuning, full parameter fine-tuning, or hybrid fine-tuning.

[0013] This application also provides a multimodal model fine-tuning and inference system for pure visual question answering, which is used to execute the above-described multimodal model fine-tuning and inference method for pure visual question answering, including: The data reading module is used to acquire the original video, question text, and answer tags; The composite input building module is used to render the question text into a question image and embed the question image into the video frames of the original video to form a video-level composite input; Image processing module: used for preprocessing video-level composite input; The frame sampling module is used to perform frame sampling on the video-level composite input to obtain a sampled frame sequence; The feature encoding module is used to extract visual features from the sampled frame sequence through the visual encoder of the multimodal large model and map them to the language model embedding space; The temporal feature aggregation module is used to aggregate temporal features from the mapped multi-frame visual features to obtain a video-level visual representation. The supervision target construction module is used to construct a sequence of supervision targets formed by sequentially concatenating question text, preset trigger markers, and answer tags; The fine-tuning training module is used to perform triggered structural supervised fine-tuning of a multimodal large model with video-level composite input as input and supervised target sequence as output expectation. The inference decoding module is used to receive the video to be inferred during the inference stage. After being processed by the image processing module, frame sampling module, feature encoding module and temporal feature aggregation module, the obtained video-level visual representation to be inferred is input into the fine-tuned multimodal large model to generate an output sequence, and the content after the trigger mark is extracted as the inference result. Results Management Module: Used to save model parameters, training logs, inference results, and deployment configurations.

[0014] The beneficial effects of the technical solutions provided in this application include at least the following: 1. By constructing a video-level composite input, the question text is embedded visually into the video frame, eliminating the need for the model to rely on a separate text question channel during training and inference. This solution breaks through the conventional "video plus independent text" input paradigm, directly adapting to real-world business scenarios where task instructions are embedded within the frame, such as screenshots, screen recordings, and equipment inspections. It solves the problem that existing multimodal models cannot read and understand question information from pixel space on their own. 2. A trigger-based structure-based supervised fine-tuning strategy is adopted, explicitly introducing a staged structure of "question text - trigger marker - answer text" into the supervised target sequence. This forces the model to learn to fully recover the visually embedded question first, and then switch to answer generation via the trigger marker, establishing a clear "read the question first, then answer" generation path. This effectively overcomes the structural failure problem commonly found in existing question-answering supervision methods that only supervise the answer or lack stage boundaries, thereby significantly improving the completeness of the output structure. 3. An end-to-end fine-tuning and inference method is provided, in which the model directly extracts features from video frames containing questions and outputs the answer, without the need for pre-extraction text extraction using external optical character recognition. This fundamentally avoids recognition errors caused by factors such as video compression distortion, motion blur, small text size, or short display time in existing two-stage pipeline solutions, as well as the risk of irreversible propagation of these errors to downstream question-answering tasks, and maintains stronger inference robustness under adverse visual conditions; 4. The input data is extended to the video domain, and a frame sampling mechanism and a temporal feature aggregation mechanism are proposed. These mechanisms ensure that the model can accurately capture the question information that only appears in a specific time period or dynamic trajectory, and can also integrate multi-frame scene evidence for temporal reasoning. This solves the technical problem that existing image-level methods cannot directly process the question display time period control and multi-frame information fusion in video input. 5. Multiple alternative solutions are provided for basic model types, question embedding methods, image processing techniques, trigger tag forms, and parameter fine-tuning methods. This design enables the technology to adapt to various complex question settings such as static full process, local time window, dynamic trajectory, and interference robustness. It can also be easily implemented on multimodal large models of different scales and architectures, and is easy to form reusable end-to-end training and inference capabilities in artificial intelligence prototype platforms.

[0015] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 The flowchart of a multimodal model fine-tuning and inference method for purely visual question answering proposed in this application is shown; Figure 2 The diagram shows the structure of a multimodal model fine-tuning and inference system for pure visual question answering according to this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Example 1 This embodiment uses video as the input scenario, as detailed below: Please see Figure 1 A method for fine-tuning and inference of a multimodal model for purely visual question answering, comprising the following steps: Step 1: Obtain the raw video and supervision data; the supervision data includes question text and answer labels; It should be noted that the process also includes checking the integrity of samples by obtaining video attributes, which include video duration, frame rate, resolution, and timestamp information. The integrity check includes checking whether the video file is decodeable, whether the question text is empty, and whether the answer tags are missing. If abnormal samples are found, they are removed or re-indexed.

[0020] Step 2: Render the question text into a question image and embed the question image into the video frames of the original video to form a video-level composite input; When the question text is rendered as a question image, a uniform font, font size, color, margin, and line break rules are used to ensure that the question is displayed completely and remains readable; a white background is preferred, and a common font family is preferred; automatic line breaks are used for excessively long questions.

[0021] The method for embedding the problem image into the video frame of the original video includes any of the following: overlay rendering, image stitching, frame border annotation, or region replacement. As long as the text on the problem statement has sufficient readability in the target frame, the same function can be achieved. The transparency of the problem statement display can be fixed or dynamically changed according to time.

[0022] The embedding method used to embed the problem image into the video frame of the original video includes any of the following: (1) Static full-process question image: The question image is continuously embedded and displayed in a fixed position throughout the entire video session; (2) Local time window question: The question image is embedded in the corresponding frame only within the preset time window of the video; (3) Dynamically moving question face: The embedded position of the question image moves according to a preset trajectory over time; (4) Interference robust question: While embedding the question image, retain the natural text that originally existed in the video frame, or apply at least one of the following perturbations to the embedded video frame: compression, noise, or blurring.

[0023] The difficulty of the above four types of question embedding settings generally progresses from static full-process questions to interference-robust questions. However, there may be local overlap between local time window questions and dynamic trajectory questions on different models and different evaluation benchmarks.

[0024] Step 3: Perform preprocessing and frame sampling on the video-level composite input sequentially to obtain a sampled frame sequence; Preprocessing preferably includes input size normalization, contrast enhancement of the question area, text edge sharpening, denoising, deblurring, and local magnification of the question area to improve the recognition stability of visual embedding problems. Preprocessing can also employ color enhancement, edge enhancement, compression artifact removal, motion compensation, local cropping and rearrangement, or text region resampling, etc., as long as it improves the model's recognition stability for visual embedding problems, achieving the same or similar functions. For video-level composite input, preprocessing mainly applies to the frame containing the question and related scene areas.

[0025] The frame sampling includes: (1) For the static full-process question paper, uniform sampling is adopted; (2) For questions within a local time window, increase the sampling density to the preset level within the time interval in which the question appears; (3) For dynamically moving questions, continuous frame sampling is used along the time period where the question moves; (4) For interference-robust question faces, for scenes that are originally present in the video frame, the video frame in which the original text and the question face image are in the same frame is selected as the sampling frame; for scenes that are perturbed after embedding the video frame, the video frames before, during and after the interference are retained respectively.

[0026] It should be noted that frame sampling is not limited to uniform sampling of a fixed number of frames; it can also employ sliding window sampling, keyframe plus neighboring frame sampling, temporal pyramid sampling, or short segment coding.

[0027] Through the above frame sampling method, the model can receive both the question information and subsequent scene evidence related to the answer.

[0028] Step 4: Input the sampled frame sequence into the visual encoder of the multimodal large model, extract the visual features of each frame, and map the visual features of each frame to the language model embedding space through the projection layer. Perform temporal feature aggregation on the mapped visual features of each frame to obtain the video-level visual representation. The multimodal large model includes a visual encoder, a connected projection layer, and a language decoder; it can receive visual input of images or videos and output text sequences, with TinyLLaVA model being preferred.

[0029] The sampled frame sequence is input into the visual encoder of the multimodal large model to extract the visual features of each frame, which are then mapped to the language model embedding space through a projection layer. Since the input is a multi-frame video, the visual features of each sampled frame are then subjected to temporal feature aggregation to obtain a video-level visual representation for subsequent language model decoding. The temporal feature aggregation method is not limited to concatenation and pooling; cross-attention mechanisms or a pre-defined temporal encoder can also be used to complete the aggregation, as long as a unified video-level visual representation can be generated. Through this process, the model obtains a unified visual representation that simultaneously contains both question information and scene evidence information.

[0030] Step 5: Construct the supervision target sequence, which is formed by sequentially concatenating the question text, preset trigger markers, and answer tags; Step 6: Using video-level composite input as input to the multimodal large model and the supervised target sequence as output expectation, perform triggered structural supervision fine-tuning on the multimodal large model to obtain the fine-tuned multimodal large model; To prove that the triggered structure-based supervised fine-tuning is the optimal choice, a sequence of supervised targets is constructed based on the selected supervised fine-tuning strategy. The supervised target sequences corresponding to the three supervised fine-tuning strategies are as follows: (1) When using baseline supervision for fine-tuning, video-level composite input is used as the model input, and the supervision target only includes the answer text. This method is simple to implement, but the model is prone to directly learning the shortcut mapping from composite input to answer, and cannot form a stable question reading behavior.

[0031] (2) When question-and-answer supervision is used for fine-tuning, the supervision target includes the question text and the answer text, in the form of "question text + answer text". By introducing the question reconstruction target, the model learns to recover the visual embedding question during training. However, this method lacks explicit stage boundaries and is prone to structural collapse problems where the model output stops at the question reproduction stage.

[0032] (3) When using a trigger-based structure for supervised fine-tuning, the supervision target includes the question text, the trigger marker and the answer text, in the form of "question text + trigger marker + answer text"; the trigger marker is preferably ASSISTANT:, or other equivalent fixed markers can be used, or other special markers, templated identifiers or implicit state switching symbols can be used, as long as the question recovery stage and the answer generation stage can be explicitly distinguished; this method clearly separates the question parsing stage and the answer generation stage through an explicit sequence structure, so that the model can form a transferable staged generation path during training.

[0033] It is important to emphasize that at the input end of training and inference, the model always only receives video-level composite input. The question text is not provided to the model as an independent input, but only appears in the supervised target sequence at the output end. The model must read and recover the question content from the video footage itself.

[0034] During the training phase, the input to the multimodal large model is always a composite video input, without any independent text questions. If baseline-supervised fine-tuning is used, the model learns to map directly from the composite input to the answer; if question-answering-supervised fine-tuning is used, the model learns to reconstruct the question first and then output the answer; if trigger-based structure-supervised fine-tuning is used, the model learns to output the question text, trigger marker, and answer text sequentially.

[0035] During training, low-rank adaptation fine-tuning, prompting optimization, partial parameter unfreezing fine-tuning, full parameter fine-tuning, or hybrid fine-tuning can be used. As long as the training objective includes staged supervision of question recovery and answer generation, the main functions of this invention can be achieved.

[0036] After training, model parameters, training logs, and validation results are saved. Experimental results show that triggered structure-supervised fine-tuning achieves the most stable performance improvement across various model sizes (including 0.5B, 2B, and 7B parameter levels) and multiple benchmarks. Compared to baseline fine-tuning that only supervises answers, it has a significant advantage in output structure consistency: taking a 0.5B parameter level model as an example, triggered structure-supervised fine-tuning can achieve near-perfect output structure integrity on visually text-dense sub-benchmarks, while question-answering supervised fine-tuning can achieve a structure collapse rate of over 50% on similar sub-benchmarks.

[0037] The reasoning phase process is as follows: acquire the video to be reasoned, which contains a visual question text; preprocess the video to be reasoned, perform frame sampling, visual feature extraction and temporal feature aggregation to obtain the video-level visual representation to be reasoned; input the video-level visual representation to be reasoned into the fine-tuned multimodal large model to obtain the output sequence, and extract the content after the trigger mark from the output sequence as the reasoning result, including the predicted question, role label and answer, for downstream use. The question recovery text before the trigger mark can be retained in the log for result analysis and error attribution.

[0038] like Figure 2 As shown, a multimodal model fine-tuning and inference system for pure visual question answering is used to execute the aforementioned multimodal model fine-tuning and inference method for pure visual question answering, including: The data reading module is used to acquire the original video, question text, and answer tags; The composite input building module is used to render the question text into a question image and embed the question image into the video frames of the original video to form a video-level composite input; Image processing module: used for preprocessing video-level composite input; The frame sampling module is used to perform frame sampling on the video-level composite input to obtain a sampled frame sequence; The feature encoding module is used to extract visual features from the sampled frame sequence through the visual encoder of the multimodal large model and map them to the language model embedding space; The temporal feature aggregation module is used to aggregate temporal features from the mapped multi-frame visual features to obtain a video-level visual representation. The supervision target construction module is used to construct a sequence of supervision targets formed by sequentially concatenating question text, preset trigger markers, and answer tags; The fine-tuning training module is used to perform triggered structural supervised fine-tuning of a multimodal large model with video-level composite input as input and supervised target sequence as output expectation. The inference decoding module is used to receive the video to be inferred during the inference stage. After being processed by the frame sampling module, feature encoding module and temporal feature aggregation module, the obtained video-level visual representation is input into the fine-tuned multimodal large model to generate an output sequence, and the content after the trigger mark is extracted as the inference result. Results Management Module: Used to save model parameters, training logs, inference results, and deployment configurations.

[0039] Example 2 The current scenario is an image-level composite input scenario. The original video is replaced with scene images, and the question image and scene image are combined through stitching or watermark overlay to form an image-level composite input. Image-level composite input can be achieved through top-bottom stitching, left-right stitching, or watermark overlay: stitching arranges the question area and scene area adjacently in space, making the question clear and controllable, suitable for low-interference comparison settings; watermark overlay overlays the question text onto a local area of ​​the scene image, more closely resembling a realistic visual text scene. The remaining preprocessing, supervised target construction, model fine-tuning, and inference processes remain consistent with the video scene.

[0040] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for fine-tuning and inference of a multimodal model for purely visual question answering, characterized in that, Includes the following steps: The process involves acquiring raw video and supervised data, including question text and answer labels. The question text is rendered into question images, which are then embedded into video frames of the original video to form a video-level composite input. This video-level composite input undergoes preprocessing and frame sampling to obtain a sampled frame sequence. This sequence is then input into the visual encoder of a multimodal large-scale model to extract visual features from each frame. These visual features are then mapped to the language model embedding space via a projection layer. Temporal feature aggregation is performed on the mapped visual features to obtain a video-level visual representation. A supervised target sequence is constructed, consisting of question text, pre-defined trigger markers, and answer labels concatenated sequentially. Using the video-level composite input as the input to the multimodal large-scale model and the supervised target sequence as the output expectation, the multimodal large-scale model undergoes trigger-based structural supervised fine-tuning to obtain the fine-tuned multimodal large-scale model. The reasoning stage process is as follows: acquire the video to be reasoned, which contains a visual question text; preprocess the video to be reasoned, perform frame sampling, visual feature extraction and temporal feature aggregation to obtain the video-level visual representation to be reasoned; input the video-level visual representation to be reasoned into the fine-tuned multimodal large model to obtain the output sequence, and extract the content after the trigger mark from the output sequence as the reasoning result.

2. The method for fine-tuning and inference of a multimodal model for pure visual question answering according to claim 1, characterized in that, The method of embedding the problematic image into the video frame of the original video includes any of the following: overlay rendering, image stitching, frame border annotation, or region replacement.

3. The method for fine-tuning and inference of a multimodal model for pure visual question answering according to claim 1, characterized in that, The embedding method used to embed the problem image into the video frame of the original video includes any of the following: Static full-process question image: The question image is continuously embedded and displayed in a fixed position throughout the entire video session; Local time window question: The question image is embedded in the corresponding frame only within a preset time window of the video; Dynamically moving question image: This allows the embedded position of the question image to move over time along a preset trajectory. Perturbation Robust Problem Statement: While embedding the problem image, retain the natural text originally present in the video frame, or apply at least one of the perturbations of compression, noise, and blur to the embedded video frame to simulate complex real video scenes.

4. The multimodal model fine-tuning and inference method for pure visual question answering according to claim 3, characterized in that, The frame sampling includes: For the static, full-process question paper, uniform sampling is used; For questions within a local time window, the sampling density is increased to the preset level within the time interval in which the question appears; For dynamically moving question texts, continuous frame sampling is used along the time period where the question text moves. For interference-robust question text, for scenes that are originally present in the video frame, the video frame in which the original text and the question text image are in the same frame is selected as the sampling frame; for scenes that are perturbed after embedding the video frame, the video frames before, during and after the perturbation are retained respectively.

5. The method for fine-tuning and inference of a multimodal model for pure visual question answering according to claim 1, characterized in that, The frame sampling uses any of the following methods: uniform sampling of a fixed number of frames, sliding window sampling, sampling of keyframes plus neighboring frames, temporal pyramid sampling, or short segment coding.

6. The multimodal model fine-tuning and inference method for pure visual question answering according to claim 5, characterized in that, The temporal feature aggregation method employs splicing, pooling, cross-attention mechanisms, or a preset temporal encoder.

7. The method for fine-tuning and inference of a multimodal model for pure visual question answering according to claim 6, characterized in that, The preset trigger flag can explicitly distinguish between the question recovery stage and the answer generation stage.

8. The multimodal model fine-tuning and inference method for pure visual question answering according to claim 7, characterized in that, The multimodal large model includes a visual encoder, a connected projection layer, and a language decoder.

9. A method for fine-tuning and inference of a multimodal model for pure visual question answering according to claim 8, characterized in that, The trigger-based structure-supervised fine-tuning employs low-rank adaptation fine-tuning, prompt-based optimization, partial parameter unfreezing fine-tuning, full parameter fine-tuning, or hybrid fine-tuning.

10. A multimodal model fine-tuning and inference system for purely visual question answering, characterized in that, It is used to execute the multimodal model fine-tuning and inference method for purely visual question answering as described in any one of claims 1-9, comprising: The data reading module is used to acquire the original video, question text, and answer tags; The composite input building module is used to render the question text into a question image and embed the question image into the video frames of the original video to form a video-level composite input; Image processing module: used for preprocessing video-level composite input; The frame sampling module is used to perform frame sampling on the video-level composite input to obtain a sampled frame sequence; The feature encoding module is used to extract visual features from the sampled frame sequence through the visual encoder of the multimodal large model and map them to the language model embedding space; The temporal feature aggregation module is used to aggregate temporal features from the mapped multi-frame visual features to obtain a video-level visual representation. The supervision target construction module is used to construct a sequence of supervision targets formed by sequentially concatenating question text, preset trigger markers, and answer tags; The fine-tuning training module is used to perform triggered structural supervised fine-tuning of a multimodal large model with video-level composite input as input and supervised target sequence as output expectation. The inference decoding module is used to receive the video to be inferred during the inference stage. After being processed by the image processing module, frame sampling module, feature encoding module and temporal feature aggregation module, the obtained video-level visual representation to be inferred is input into the fine-tuned multimodal large model to generate an output sequence, and the content after the trigger mark is extracted as the inference result. Results Management Module: Used to save model parameters, training logs, inference results, and deployment configurations.

Images