A video inference method, system, device and medium of a multi-modal large model
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG LAB
- Filing Date
- 2026-03-10
- Publication Date
- 2026-05-29
Smart Images
Figure CN121809701B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a video reasoning method, system, device and medium for multimodal large models. Background Technology
[0002] With the rapid development of multimodal large language models, significant breakthroughs have been achieved in understanding and describing static images. However, existing multimodal large models still have obvious shortcomings when dealing with dynamic video data. Video is not merely a stack of images, but also contains complex temporal dependencies, causal logic, and dynamic changes.
[0003] Currently, the main approach to improving the video capabilities of models relies on supervised fine-tuning (SFT) using large-scale Video Question Answering (VQA) datasets. This approach faces two major bottlenecks: First, the cost of annotating high-quality video question answering data is extremely high, making it difficult to improve performance by simply increasing the data size; second, existing SFT data often focus on simple object recognition or action description, lacking deep reasoning training on the inherent logic and temporal causality of videos, leading to "illusions" or logical inconsistencies in models when processing complex, long videos.
[0004] Therefore, how to utilize massive amounts of unlabeled video data to automatically construct high-quality inference tasks and effectively inject these logical capabilities into large models is a pressing technical challenge that needs to be addressed. Summary of the Invention
[0005] Therefore, it is necessary to provide a multimodal large-scale video inference method, system, device, and medium to address the aforementioned technical problems.
[0006] In a first aspect, embodiments of this application provide a video inference method for a multimodal large model, the method comprising:
[0007] Design a course-based task system, which includes at least three video reasoning tasks, with the difficulty of each video reasoning task increasing sequentially.
[0008] Based on the original video files, a self-learning supervision mechanism is used to generate training datasets for each video reasoning task.
[0009] Using reinforcement learning algorithms, the multimodal large model is trained sequentially using the corresponding training datasets according to the order of video reasoning tasks from easy to difficult, to obtain the target multimodal large model;
[0010] The target video file and query request are input into the target multimodal large model, and the video inference result is output.
[0011] In one embodiment, the multimodal large model is fine-tuned and trained sequentially using the corresponding training datasets to obtain the target multimodal large model, which includes:
[0012] Once the multimodal large model reaches a preset convergence metric after training on the current training dataset, iterative training is then performed on the training dataset corresponding to the next difficulty video inference task until all video inference tasks are completed, thus obtaining the target multimodal large model.
[0013] In one embodiment, the course-based task system includes a first video reasoning task, a second video reasoning task, and a third video reasoning task with progressively increasing difficulty.
[0014] The first video inference task includes an abnormal frame localization task based on visual feature perception, used to detect abnormal frames in the video;
[0015] The second video reasoning task includes a temporal logic-aware video segment sorting task, used to reason the correct temporal order of multiple video segments;
[0016] The third video reasoning task includes a causal reasoning task based on reasoning logic perception, used to infer the text description of the middle segment based on the first and last segments of the video.
[0017] In one embodiment, the step of generating the training dataset corresponding to each video inference task based on the original video file using a self-learning supervised mechanism includes:
[0018] At least one video frame is randomly selected from the original video file, and an abnormal operation is performed on it to obtain a first video file;
[0019] The real index of the video frame that performed the abnormal operation is used as the first self-supervised label;
[0020] Based on the first video file and the first self-supervised label, a first training dataset for the first video inference task is constructed.
[0021] In one embodiment, the step of generating the training dataset corresponding to each video inference task based on the original video file using a self-learning supervised mechanism includes:
[0022] The original video file is divided into multiple video segments, and the order of the multiple video segments is used as the second self-supervised label;
[0023] The multiple video segments are randomly shuffled to obtain a second video file;
[0024] Based on the second video file and the second self-supervised label, a second training dataset for the second video inference task is constructed.
[0025] In one embodiment, the step of generating the training dataset corresponding to each video inference task based on the original video file using a self-learning supervised mechanism includes:
[0026] Extract the first segment, middle segment, and last segment from the original video file;
[0027] Extract the correct text description of the intermediate segment and construct multiple interfering text descriptions. Construct an option list based on the correct text description and the multiple interfering text descriptions.
[0028] A third video file is obtained based on the first segment, the last segment, and the option list;
[0029] Based on the option identifiers of the correct text description, a third self-supervised label is obtained;
[0030] Based on the third video file and the third self-supervised label, a third training dataset for the third video inference task is constructed.
[0031] In one embodiment, the reinforcement learning algorithm is used to train the multimodal large model sequentially using the corresponding training datasets according to the order of difficulty of the video reasoning tasks, to obtain the target multimodal large model, which includes:
[0032] The first training dataset is input into the multimodal large model, and the GRPO reinforcement learning algorithm is used to construct a reward function using the first self-supervised label to fine-tune the multimodal large model. When the first preset convergence index is reached, the first multimodal large model is obtained.
[0033] The second training dataset is input into the first multimodal large model, the GRPO algorithm is used to optimize the model parameters, and the reward function is constructed using the second self-supervised label to fine-tune the first multimodal large model. When the second preset convergence index is reached, the second multimodal large model is obtained.
[0034] The third training dataset is input into the second multimodal large model, the GRPO algorithm is used to optimize the model parameters, and the reward function is constructed using the third self-supervised label to fine-tune the training of the second multimodal large model. When the third preset convergence index is reached, the target multimodal large model is obtained.
[0035] Secondly, embodiments of this application also provide a multimodal large-scale video inference system, the system comprising:
[0036] The generation module is used to generate training datasets for each video reasoning task based on the original video files using a self-learning supervised mechanism; wherein the difficulty of each video reasoning task increases sequentially.
[0037] The training module is used to train the multimodal large model sequentially using the corresponding training datasets according to the order of difficulty of the video reasoning task, using reinforcement learning algorithms, to obtain the target multimodal large model.
[0038] The inference module is used to input the target video file and query request into the target multimodal large model and output the video inference result.
[0039] Thirdly, embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method described in the first aspect above.
[0040] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.
[0041] The aforementioned video inference methods, systems, devices, and media for multimodal large models solve the problem of reliance on manual annotation in existing technologies by constructing a self-supervised data generation mechanism that does not require manual annotation, thereby improving the training efficiency of multimodal large models. By designing a course task system with increasing difficulty and using reinforcement learning algorithms to fine-tune the model according to the order of course tasks from easy to difficult, the model's ability to perceive fine-grained video features, understand temporal logic, and make causal inferences across modal events can be effectively enhanced. This avoids the problem of models lacking deep reasoning ability in existing technologies and significantly improves the accuracy and robustness of the model in complex video understanding scenarios.
[0042] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0043] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0044] Figure 1 This is a hardware structure block diagram of a terminal device for a multimodal large-scale video inference method in one embodiment;
[0045] Figure 2This is a flowchart illustrating a video inference method for a multimodal large model in one embodiment;
[0046] Figure 3 This is a flowchart illustrating a video inference method for a multimodal large model in a preferred embodiment;
[0047] Figure 4 This is a structural block diagram of a multimodal large-scale video inference system in one embodiment;
[0048] Figure 5 This is a schematic diagram of the computer device structure in one embodiment. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0050] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the multimodal large-model video inference method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.
[0051] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multimodal large-scale model video inference method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0052] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module for wireless communication with the Internet.
[0053] This application provides a multimodal large-model video inference method, which can be applied to... Figure 1 Taking the terminal in the example of this, for example... Figure 2 As shown, the method includes the following steps:
[0054] Step S201: Design a course-based task system, which includes at least three video reasoning tasks, with the difficulty of each video reasoning task increasing sequentially.
[0055] This step designs a course-style task system that progresses from easy to difficult, which can simulate the human learning process. By training and fine-tuning the multimodal large model through video reasoning tasks of increasing difficulty, the model is guided to optimize training in task iteration, thereby guiding the multimodal large model to gradually master video reasoning capabilities in complex scenarios.
[0056] Step S202: Based on the original video files, a self-learning supervision mechanism is used to generate training datasets corresponding to each video inference task.
[0057] The original video files include a massive amount of unlabeled raw video. A self-learning supervised mechanism is a training strategy that reduces reliance on manual annotation by generating or mining supervisory signals through the model itself, and uses these supervisory signals for learning, continuously improving the learning process itself. In video reasoning tasks, the self-learning supervised mechanism can automatically construct training samples with deterministic answers by combining the inherent structure of multimodal data (such as frame continuity, temporal unidirectionality, event integrity, temporal consistency, cross-modal alignment, causal relationships, etc.). Since the answers come directly from the data itself (such as shuffling, occlusion, and truncation), no manual intervention is required.
[0058] This step utilizes a self-learning supervised mechanism to construct a corresponding training dataset for each video reasoning task, automatically generating training samples containing defined "question-answer" pairs without manual annotation. This solves the problem of existing technologies relying on manual annotation for model training and improves the efficiency of training large multimodal models.
[0059] Step S203: Using a reinforcement learning algorithm, the multimodal large model is trained sequentially using the corresponding training datasets according to the order of difficulty of the video reasoning tasks, in order to obtain the target multimodal large model.
[0060] In this step, reinforcement learning algorithms are used to train the multimodal large model for each video reasoning task. Once the video reasoning task has been trained using the training dataset, the target multimodal large model is obtained.
[0061] Step S204: Input the target video file and query request into the target multimodal large model, and output the video inference result.
[0062] The multimodal large model, trained through a progressively challenging curriculum, possesses complete capabilities ranging from fine-grained perception to high-order causal reasoning. It can be directly applied to complex business scenarios such as video anomaly detection, intelligent monitoring analysis, and movie plot understanding. This solves the problem of existing multimodal large models lacking deep reasoning capabilities.
[0063] In the aforementioned multimodal large-scale video inference method, by constructing a self-supervised data generation mechanism that does not require manual annotation and a course task system with increasing difficulty, and by using reinforcement learning algorithms to fine-tune the model according to the order of course tasks from easy to difficult, the model's ability to perceive fine-grained features of videos, understand temporal logic, and make causal inferences across modal events can be effectively enhanced, significantly improving the model's accuracy and robustness in complex video understanding scenarios.
[0064] In one embodiment, the multimodal large model is fine-tuned and trained sequentially using the corresponding training datasets to obtain the target multimodal large model. This includes: when the multimodal large model reaches a preset convergence index after being trained on the current training dataset, the multimodal large model is iteratively trained on the training dataset corresponding to the next difficulty video reasoning task until all video reasoning tasks are completed, thereby obtaining the target multimodal large model.
[0065] This application trains a multimodal large model sequentially using corresponding training datasets, following an order of increasing difficulty for video reasoning tasks. During training, a reinforcement learning algorithm is employed to fine-tune the model using the training dataset corresponding to the current task, enabling it to complete the current video reasoning task. Once the model reaches a preset convergence criterion, it automatically advances to the next difficulty level. Then, based on the training dataset for the next difficulty level, reinforcement learning is used to iteratively train the already trained model. After training on all task-related datasets is completed, the multimodal large model can complete the highest difficulty video reasoning task, resulting in the target multimodal large model.
[0066] In one embodiment, the course-based task system includes a first video reasoning task, a second video reasoning task, and a third video reasoning task with progressively increasing difficulty. The first video reasoning task includes an anomaly frame localization task based on visual feature perception, used to detect anomaly frames in the video; the second video reasoning task includes a video segment sorting task based on temporal logic perception, used to reason about the correct temporal order of multiple video segments; and the third video reasoning task includes a causal reasoning task based on reasoning logic perception, used to infer the text description of the middle segments based on the first and last segments of the video.
[0067] The training dataset for the first video inference task generated using a self-learning supervision mechanism includes the following: randomly selecting at least one video frame from the original video file and performing an abnormal operation on it to obtain a first video file; using the real index of the video frame on which the abnormal operation was performed as a first self-supervised label; and constructing a first training dataset for the first video inference task based on the first video file and the first self-supervised label.
[0068] Specifically, the first video inference task is used to detect anomalous frames in a video and output the index positions of these anomalous frames. When constructing the training dataset for the first video inference task, a video segment containing N consecutive frames (e.g., 10 frames) can be extracted from each original video. One or more frames are randomly selected, and anomalous operations (including Gaussian blur, pixel occlusion, random noise addition, flipping, or mirroring) are applied to them. The ground truth index of the original anomalously manipulated frame is used as the self-supervised label. The processed N-frame sequence is then input into a multimodal large-scale model for training, requiring the model to output the index positions of the anomalous frames. When the model's accuracy on the validation set of the first video inference task exceeds a preset threshold, it automatically unlocks and enters the training phase of the next difficulty task.
[0069] The abnormal frame localization task in this application aims to train the model's sensitivity to fine-grained visual features. By extracting a small segment (e.g., 10 consecutive frames) from a video, randomly selecting one or more frames to apply anomaly operations, and having the model output the index position of the abnormal frame, the model's ability to identify differences between frames is trained, establishing a basic perception of visual continuity.
[0070] The method of generating the training dataset corresponding to the second video reasoning task using a self-learning supervision mechanism includes the following: dividing the original video file into multiple video segments, and using the order of the multiple video segments as the second self-supervised label; randomly shuffling the multiple video segments to obtain a second video file; and constructing a second training dataset for the second video reasoning task based on the second video file and the second self-supervised label.
[0071] Specifically, the second video inference task is used to infer the correct temporal order of multiple video segments. When constructing the training dataset for the second video inference task, a long video can be divided into M segments (e.g., 3 segments), each segment containing several consecutive frames; the temporal order of these M segments is randomly shuffled to form a set of scrambled segments; the original, undisturbed segment order is used as self-supervised labels. After the training phase of the first video inference task is completed, the scrambled set is input into a multimodal large model, requiring the model to output the correct sequence of segments that restores the original temporal order. When the model's accuracy on the validation set of the second video inference task exceeds a preset threshold, it automatically unlocks and enters the training phase of the next difficulty task.
[0072] The video clip sorting task in this application aims to train the model to understand the passage of time and the development of actions. The video is divided into multiple segments and shuffled, and the model is asked to restore the correct playback order. The training model is used to understand the start, development and end of actions, perceive the dynamic changes in time sequence, and establish medium- and long-term temporal logic.
[0073] The training dataset for the third video reasoning task, generated using a self-learning supervised mechanism, includes the following: extracting the first, middle, and last segments from the original video file; extracting the correct text description from the middle segment and constructing multiple interfering text descriptions; constructing an option list based on the correct text description and the multiple interfering text descriptions; obtaining a third video file based on the first segment, the last segment, and the option list; obtaining a third self-supervised label based on the option identifiers of the correct text description; and constructing a third training dataset for the third video reasoning task based on the third video file and the third self-supervised label.
[0074] In constructing the training dataset for the third video inference task, this application extracts the first and last segments from a complete video event as contextual visual input; it then extracts the middle segment of the video and generates a correct text description for this middle segment using an image / video description algorithm or pre-defined metadata, serving as the positive option; it also constructs several interfering text descriptions that do not match the content of the middle segment as negative options, and uses the correct text description option identifier as a self-supervised label. After the training phase of the second video inference task is completed, the first and last segments, the prompt word, and the option list consisting of the correct and interfering text descriptions are input into the multimodal large model. The model is required to infer the event that occurs in the middle based on the visual context of the first and last segments and select the text description that best fits the logical connection from the option list. When the model's accuracy on the validation set of the third video inference task exceeds a preset threshold, the target multimodal large model is obtained.
[0075] This application's causal reasoning task aims to train the model's cross-modal causal reasoning and logical bridging capabilities. By extracting the "beginning" and "end" segments from a complete video as visual context input, the intermediate process is hidden. Simultaneously, the system automatically extracts text descriptions of the intermediate segments as correct options and automatically generates or retrieves several irrelevant and misleading text descriptions as distractors. The model needs to receive the beginning and end video segments and a list of text options. By analyzing the evolutionary logic of the visual scene (i.e., "how the beginning developed into the ending"), it reasons in the text space and selects the description of the intermediate process that correctly fills the logical gaps. This "video input-text selection" model forces the model to perform deep semantic transformation, rather than remaining at the level of simple visual pixel matching, thereby greatly enhancing the model's logical reasoning and language understanding capabilities.
[0076] In one embodiment, the step of using a reinforcement learning algorithm to train the multimodal large model sequentially using the corresponding training datasets according to the order of difficulty of the video reasoning task, to obtain the target multimodal large model, includes the following steps:
[0077] Step S301: Input the first training dataset into the multimodal large model, use the GRPO reinforcement learning algorithm, construct a reward function using the first self-supervised label to fine-tune the multimodal large model, and obtain the first multimodal large model when the first preset convergence index is reached.
[0078] Step S302: Input the second training dataset into the first multimodal large model, optimize the model parameters using the GRPO algorithm, and fine-tune the first multimodal large model by constructing a reward function using the second self-supervised label. When the second preset convergence index is reached, the second multimodal large model is obtained.
[0079] Step S303: Input the third training dataset into the second multimodal large model, optimize the model parameters using the GRPO algorithm, and fine-tune the second multimodal large model by constructing a reward function using the third self-supervised label. When the third preset convergence index is reached, the target multimodal large model is obtained.
[0080] The first, second, and third training datasets are the training datasets for the first, second, and third video inference tasks, respectively. The difficulty of the first, second, and third video inference tasks increases sequentially.
[0081] In the training process of a multimodal large model, this application employs the GRPO (Group Relative Policy Optimization) reinforcement learning algorithm to fine-tune and optimize the model. Unlike traditional supervised fine-tuning (SFT), GRPO allows the model to generate multiple answers to the same question, optimizing the policy by comparing the rewards of different answers within the same group. In this application, since each self-supervised task has a definite answer, the reward function is clearly designed: a high score for a correct answer and a low score for an incorrect answer. Through the GRPO reinforcement learning algorithm, the model can explore better inference paths, maximize the expected reward, and thus internalize the capabilities learned from the three tasks into the model parameters.
[0082] For each video inference task with self-supervised labels, the multimodal large model acts as a policy network. For the same input (Prompt), the model samples and generates multiple sets of outputs. A reward function is constructed using the self-supervised labels from the training dataset. If the model output matches the self-supervised label, a positive reward is given; otherwise, a negative reward or zero reward is given. The GRPO algorithm is used to calculate the relative advantage within the group and optimize the model parameters so that its output probability distribution shifts towards the direction of higher rewards.
[0083] For each video reasoning task with self-supervised labels, this application can set a dynamic or fixed accuracy threshold, use the GRPO algorithm to fine-tune the training of the multimodal large model, and when the model's accuracy on the validation set of the current difficulty task exceeds the preset threshold (such as 90%), it will automatically unlock and enter the training phase of the next difficulty task until all courses are completed, thereby enhancing the video understanding and reasoning ability of the multimodal large model.
[0084] The present embodiment will now be described and illustrated through preferred embodiments.
[0085] The course-based task system designed in this application includes tasks ranging from easy to difficult: abnormal frame localization, video segment sorting (temporal dynamic task), and video reasoning. Figure 3 This is a preferred flowchart of the video inference method for multimodal large models in this embodiment, such as... Figure 3 As shown, the video inference method for this multimodal large model includes the following steps:
[0086] Step 1: Based on massive amounts of unlabeled raw video data, construct self-supervised training samples for each inference task and automatically generate definite question-answer pairs, including the first training dataset, the second training dataset, and the third training dataset.
[0087] Step 2: Train the multimodal large model using the first training dataset to complete the abnormal frame localization task.
[0088] During training, the GRPO reinforcement learning optimization algorithm is used, employing policy sampling and reward feedback mechanisms to update the parameters of the current multimodal large model until the model reaches the convergence metric of the current task.
[0089] Step 3: Next, use the second training dataset to train the multimodal large model to complete the time-series dynamic task.
[0090] During training, the GRPO reinforcement learning optimization algorithm is used, employing policy sampling and reward feedback mechanisms to update the parameters of the current multimodal large model until the model reaches the convergence metric of the current task.
[0091] Step 4: Train the multimodal large model using the third training dataset to complete the causal reasoning task and obtain the target multimodal large model.
[0092] During training, the GRPO reinforcement learning optimization algorithm is used, employing policy sampling and reward feedback mechanisms to update the parameters of the current multimodal large model until the model reaches the convergence metric of the current task.
[0093] Step 5: Apply the target multimodal large model to actual video understanding services to output enhanced video reasoning results.
[0094] This application also provides a multimodal large-model video inference system, such as... Figure 4 As shown, the system includes a generation module 10, a training module 20, and an inference module 30.
[0095] The generation module 10 is used to generate training datasets for each video reasoning task based on the original video files using a self-learning supervised mechanism; wherein the difficulty of each video reasoning task increases sequentially.
[0096] The training module 20 is used to train the multimodal large model sequentially using the corresponding training datasets according to the order of difficulty of the video reasoning task, using reinforcement learning algorithms, to obtain the target multimodal large model.
[0097] The inference module 30 is used to input the target video file and query request into the target multimodal large model and output the video inference result.
[0098] In one embodiment, the training module 20 is further configured to: after the multimodal large model reaches a preset convergence index by training based on the current training dataset, iteratively train the multimodal large model based on the training dataset corresponding to the next difficulty video reasoning task until all video reasoning tasks are completed to obtain the target multimodal large model.
[0099] In one embodiment, the course-based task system includes a first video reasoning task, a second video reasoning task, and a third video reasoning task with progressively increasing difficulty; the first video reasoning task includes an abnormal frame localization task based on visual feature perception, used to detect abnormal frames in the video; the second video reasoning task includes a video segment sorting task based on temporal logic perception, used to reason about the correct temporal order of multiple video segments; the third video reasoning task includes a causal reasoning task based on reasoning logic perception, used to infer the text description of the middle segment based on the first and last segments of the video.
[0100] In one embodiment, the generation module 10 is further configured to: randomly select at least one video frame from the original video file and perform an abnormal operation on it to obtain a first video file; use the real index of the video frame on which the abnormal operation was performed as a first self-supervised label; and construct a first training dataset for the first video inference task based on the first video file and the first self-supervised label.
[0101] In one embodiment, the generation module 10 is further configured to: divide the original video file into multiple video segments, and use the order of the multiple video segments as a second self-supervised label; randomly shuffle the multiple video segments to obtain a second video file; and construct a second training dataset for the second video inference task based on the second video file and the second self-supervised label.
[0102] In one embodiment, the generation module 10 is further configured to: extract a first segment, a middle segment, and a last segment from the original video file; extract the correct text description of the middle segment and construct multiple interfering text descriptions; construct an option list based on the correct text description and the multiple interfering text descriptions; obtain a third video file based on the first segment, the last segment, and the option list; obtain a third self-supervised label based on the option identifier of the correct text description; and construct a third training dataset for the third video inference task based on the third video file and the third self-supervised label.
[0103] In one embodiment, the training module 20 is further configured to: input the first training dataset into the multimodal large model, use the GRPO reinforcement learning algorithm, construct a reward function using the first self-supervised labels to fine-tune the multimodal large model, and obtain the first multimodal large model when a first preset convergence index is reached; input the second training dataset into the first multimodal large model, use the GRPO algorithm to optimize the model parameters, construct a reward function using the second self-supervised labels to fine-tune the first multimodal large model, and obtain the second multimodal large model when a second preset convergence index is reached; input the third training dataset into the second multimodal large model, use the GRPO algorithm to optimize the model parameters, construct a reward function using the third self-supervised labels to fine-tune the second multimodal large model, and obtain the target multimodal large model when a third preset convergence index is reached.
[0104] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.
[0105] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a communication interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multimodal large-scale video inference method.
[0106] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0107] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above embodiments of the multimodal large-scale video inference method.
[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A video inference method for multimodal large models, characterized in that, The method includes: The course-based task system includes three video reasoning tasks: the first video reasoning task, the second video reasoning task, and the third video reasoning task. The first video reasoning task is an abnormal frame localization task based on visual feature perception, used to detect abnormal frames in a video. The second video reasoning task is a video segment sorting task based on temporal dynamic perception, used to reason about the correct temporal order of multiple video segments. The third video reasoning task is a causal reasoning task based on reasoning logic perception, used to infer the text description of the middle segment based on the first and last segments of the video. Based on the original video files, a self-learning supervision mechanism is used to generate training datasets for each video reasoning task. The multimodal large model is trained sequentially using the corresponding training datasets in the order of the first video reasoning task, the second video reasoning task, and the third video reasoning task, using a reinforcement learning algorithm to obtain the target multimodal large model. The target video file and query request are input into the target multimodal large model, and the video inference result is output.
2. The method according to claim 1, characterized in that, The multimodal large model is trained sequentially using the corresponding training datasets to obtain the target multimodal large model, which includes: Once the multimodal large model reaches a preset convergence metric after training on the current training dataset, iterative training is then performed on the training dataset corresponding to the next difficulty video inference task until all video inference tasks are completed, thus obtaining the target multimodal large model.
3. The method according to claim 1, characterized in that, The training datasets generated for each video inference task based on the original video files and using a self-learning supervised mechanism include: At least one video frame is randomly selected from the original video file, and an abnormal operation is performed on it to obtain a first video file; The real index of the video frame that performed the abnormal operation is used as the first self-supervised label; Based on the first video file and the first self-supervised label, a first training dataset for the first video inference task is constructed.
4. The method according to claim 3, characterized in that, The training datasets generated for each video inference task based on the original video files and using a self-learning supervised mechanism include: The original video file is divided into multiple video segments, and the order of the multiple video segments is used as the second self-supervised label; The multiple video segments are randomly shuffled to obtain a second video file; Based on the second video file and the second self-supervised label, a second training dataset for the second video inference task is constructed.
5. The method according to claim 4, characterized in that, The training datasets generated for each video inference task based on the original video files and using a self-learning supervised mechanism include: Extract the first segment, middle segment, and last segment from the original video file; Extract the correct text description of the intermediate segment and construct multiple interfering text descriptions. Construct an option list based on the correct text description and the multiple interfering text descriptions. A third video file is obtained based on the first segment, the last segment, and the option list; Based on the option identifiers of the correct text description, a third self-supervised label is obtained; Based on the third video file and the third self-supervised label, a third training dataset for the third video inference task is constructed.
6. The method according to claim 5, characterized in that, The reinforcement learning algorithm is used to train the multimodal large model sequentially using the corresponding training datasets according to the order of difficulty of the video reasoning tasks, resulting in the target multimodal large model including: The first training dataset is input into the multimodal large model, and the GRPO reinforcement learning algorithm is used to construct a reward function using the first self-supervised label to fine-tune the multimodal large model. When the first preset convergence index is reached, the first multimodal large model is obtained. The second training dataset is input into the first multimodal large model, the GRPO algorithm is used to optimize the model parameters, and the reward function is constructed using the second self-supervised label to fine-tune the first multimodal large model. When the second preset convergence index is reached, the second multimodal large model is obtained. The third training dataset is input into the second multimodal large model, the GRPO algorithm is used to optimize the model parameters, and the reward function is constructed using the third self-supervised label to fine-tune the training of the second multimodal large model. When the third preset convergence index is reached, the target multimodal large model is obtained.
7. A multimodal large-scale video inference system, characterized in that, The system includes: The generation module is used to generate training datasets for the first video inference task, the second video inference task, and the third video inference task respectively, based on the original video files and using a self-learning supervised mechanism. The first video inference task is an abnormal frame localization task based on visual features, used to detect abnormal frames in the video. The second video inference task is a video segment sorting task based on temporal dynamic perception, used to infer the correct temporal order of multiple video segments. The third video inference task is a causal inference task based on reasoning logic perception, used to infer the text description of the middle segment based on the first and last segments of the video. The training module is used to train the multimodal large model sequentially using the corresponding training datasets in the order of the first video reasoning task, the second video reasoning task, and the third video reasoning task, using reinforcement learning algorithms, to obtain the target multimodal large model. The inference module is used to input the target video file and query request into the target multimodal large model and output the video inference result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.