Video reasoning method and device based on multi-modal large model, storage medium and computer equipment

Abstract

The invention discloses a video reasoning method and device based on a multi-modal large model, a storage medium and computer equipment. The method comprises the steps of obtaining a to-be-reasoned target video and a reasoning problem; constructing cue words based on the target video and the reasoning problem, inputting the cue words into a pre-trained multi-modal large model, performing fragment segmentation on the target video through the multi-modal large model, performing visual semantic information analysis on each video fragment, and obtaining a visual semantic information analysis result; performing reasoning chain-based answer reasoning on the reasoning problem according to the time sequence information of the target video and the corresponding visual semantic information; and outputting the reasoning answer and the reasoning chain of the multi-modal large model. The method can be applied to the field of medical health.

Description

Technical Field

[0001] This application relates to the field of video reasoning technology, which can be applied to the medical and health field, and in particular to a video reasoning method, device, storage medium and computer equipment based on a multimodal large model. Background Technology

[0002] In the healthcare field, video data is playing an increasingly crucial role. For example, in rehabilitation training videos, therapists use information such as the patient's posture, range of motion, and frequency of movement to assess the effectiveness of rehabilitation training and adjust the training plan.

[0003] Currently, mainstream video understanding systems primarily rely on end-to-end classification or description generation methods. These models directly output answers or text descriptions after receiving video features. While they have achieved some success in relatively static and simple tasks, their limitations are quite evident in the complex and ever-changing temporal reasoning scenarios in the healthcare field.

[0004] Traditional video models lack the ability to actively locate key time segments. In medical videos, especially long rehabilitation training videos, not all frames or scenes are closely related to the current diagnostic or assessment problem. Furthermore, the descriptions generated by the models are often holistic, lacking step-by-step explanations with a logical reasoning structure. In medical scenarios, problems such as diagnosing a condition and assessing treatment effectiveness require clear and organized reasoning processes to support conclusions. Taking rehabilitation training video assessment as an example, medical personnel not only want to know the patient's overall recovery status but also want to understand the patient's specific performance at each training stage, which actions played a positive role in rehabilitation, and the reasons why. However, the holistic descriptions output by existing models cannot provide such detailed reasoning paths, making the reasoning results difficult to interpret and hindering medical personnel from making scientifically sound decisions based on the model output. Summary of the Invention

[0005] In view of this, embodiments of this application provide a video reasoning method, apparatus, storage medium, and computer device based on a multimodal large model.

[0006] According to one aspect of this application, a video reasoning method based on a multimodal large model is provided, the method comprising: Obtain the target video to be reasoned and the reasoning question; Based on the target video and the reasoning question, prompt words are constructed and input into a pre-trained multimodal large model. The multimodal large model is used to segment the target video, perform visual semantic information analysis on each video segment, and perform answer reasoning based on the reasoning chain based on the temporal information and corresponding visual semantic information of the target video. Output the reasoning answer and reasoning chain of the multimodal large model.

[0007] Optionally, the training method for the multimodal large model includes: Obtain a video reasoning sample set, wherein the video reasoning sample set contains multiple video reasoning samples, each video reasoning sample includes a video thumbnail frame sequence sample, a temporal information sample of multiple video thumbnail frame sequence segments divided based on the video thumbnail frame sequence sample, a visual semantic information sample of each video thumbnail frame sequence segment, the task question, the task answer, and the task reasoning chain of the video thumbnail frame sequence sample. The multimodal large model is trained stepwise based on the video reasoning sample set to improve its video reasoning capabilities, which include segmentation, visual semantic analysis, and answer reasoning based on reasoning chains. Specifically, the multimodal large model is trained to improve its segmentation capabilities using the video thumbnail sequence samples and temporal information samples of multiple video thumbnail sequence segments; its visual semantic analysis capabilities are trained using multiple video thumbnail sequence segments and corresponding visual semantic information samples; and its answer reasoning capabilities are trained using the temporal information of multiple video thumbnail sequence segments, corresponding visual semantic information samples, task questions, task answers, and task reasoning chains.

[0008] Optionally, after obtaining the video inference sample set, the method further includes: The video inference sample set is divided into a first-stage training sample and a second-stage training sample. The stepwise training of the video reasoning capability of the multimodal large model based on the video reasoning sample set includes: The multimodal large model is trained step-by-step using the first-stage training samples to improve its video reasoning capabilities. After the stepwise training of the video reasoning capability of the multimodal large model based on the video reasoning sample set, the method further includes: The video reasoning capability of the multimodal large model, which has completed step-by-step training, is enhanced using the two-stage training samples; wherein, the enhancement training process includes: Based on the video thumbnail frame sequence samples and the task question, a prompt word sample is constructed. The prompt word sample is input into the multimodal large model that has completed step-by-step training. The multimodal large model that has completed step-by-step training is guided to output task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information according to the process of frame sequence segment segmentation, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information. Rewards are calculated on the output information of the multimodal large model that has completed step-by-step training, and the parameters of the multimodal large model that has completed step-by-step training are optimized based on the calculated reward value.

[0009] Optionally, the reward calculation includes format reward calculation and answer reward calculation. The format reward calculation is used to measure whether the output format of the multimodal large model is correct, and the answer reward calculation is used to measure whether the reasoning answer prediction information of the multimodal large model is correct.

[0010] Optionally, segmenting the target video using the multimodal large model includes: The target video is subjected to keyframe extraction using the multimodal large model to obtain a video thumbnail sequence. The video thumbnail sequence is then segmented, and each segmented video thumbnail sequence segment is taken as a video segment.

[0011] Optionally, obtaining the video inference sample set includes: The original video and the task problem based on the original video are obtained, and the key frames of the original video are extracted to form a video thumbnail frame sequence sample. The first general large model is used to segment the video thumbnail sequence samples based on the task problem to obtain multiple video thumbnail sequence segments and corresponding temporal information samples, as well as generate visual semantic information samples for each video thumbnail sequence segment. The visual semantic information samples of the video thumbnail frame sequence fragments are reviewed and rewritten using the second general large model to correct the visual semantic information samples. The first general large model generates a task reasoning chain and a task answer based on the video thumbnail frame sequence fragments, the corresponding temporal information samples, and the corresponding visual semantic information samples. Based on the video thumbnail frame sequence, the multiple video thumbnail frame sequence segments and their corresponding temporal information samples and visual semantic information samples, the task question, the task reasoning chain and the task answer, a video reasoning sample set is constructed, and a video reasoning sample set is constructed based on the video reasoning sample.

[0012] Optionally, constructing a video inference sample set based on the video inference samples includes: A pre-defined scoring model performs multi-dimensional scoring based on the visual semantic information samples and corresponding temporal information samples, the task reasoning chain, and the task answer. The multi-dimensional scoring includes logical consistency, temporal rationality, and language fluency. The logical consistency score is used to evaluate whether there is logical consistency between the visual semantic information samples and the task reasoning chain, and between the task reasoning chain and the task answer. The temporal rationality score is used to evaluate whether the task reasoning chain is temporally reasonable based on the temporal information samples corresponding to the visual semantic information samples. The language fluency score is used to evaluate whether the task answer is linguistically fluent. The video inference samples are filtered based on multi-dimensional scoring, and the remaining video inference samples are used to construct the video inference sample set.

[0013] According to another aspect of this application, a video inference device based on a multimodal large model is provided, the device comprising: The data acquisition module is used to acquire the target video to be inferred and the inference question; The task reasoning module is used to construct prompt words based on the target video and the reasoning question, input the prompt words into a pre-trained multimodal large model, and use the multimodal large model to segment the target video, perform visual semantic information analysis on each video segment, and perform answer reasoning based on the reasoning chain for the reasoning question according to the temporal information and corresponding visual semantic information of the target video. The answer output module is used to output the reasoning answer and reasoning chain of the multimodal large model.

[0014] Optionally, the apparatus further includes: a model training module, used for: Obtain a video reasoning sample set, wherein the video reasoning sample set contains multiple video reasoning samples, each video reasoning sample includes a video thumbnail frame sequence sample, a temporal information sample of multiple video thumbnail frame sequence segments divided based on the video thumbnail frame sequence sample, a visual semantic information sample of each video thumbnail frame sequence segment, the task question, the task answer, and the task reasoning chain of the video thumbnail frame sequence sample. The multimodal large model is trained stepwise based on the video reasoning sample set to improve its video reasoning capabilities, which include segmentation, visual semantic analysis, and answer reasoning based on reasoning chains. Specifically, the multimodal large model is trained to improve its segmentation capabilities using the video thumbnail sequence samples and temporal information samples of multiple video thumbnail sequence segments; its visual semantic analysis capabilities are trained using multiple video thumbnail sequence segments and corresponding visual semantic information samples; and its answer reasoning capabilities are trained using the temporal information of multiple video thumbnail sequence segments, corresponding visual semantic information samples, task questions, task answers, and task reasoning chains.

[0015] Optionally, the model training module is further configured to: The video inference sample set is divided into a first-stage training sample and a second-stage training sample. The multimodal large model is trained step-by-step using the first-stage training samples to improve its video reasoning capabilities. The video reasoning capability of the multimodal large model, which has completed step-by-step training, is enhanced using the two-stage training samples; wherein, the enhancement training process includes: Based on the video thumbnail frame sequence samples and the task question, a prompt word sample is constructed. The prompt word sample is input into the multimodal large model that has completed step-by-step training. The multimodal large model that has completed step-by-step training is guided to output task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information according to the process of frame sequence segment segmentation, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information. Rewards are calculated on the output information of the multimodal large model that has completed step-by-step training, and the parameters of the multimodal large model that has completed step-by-step training are optimized based on the calculated reward value.

[0016] Optionally, the reward calculation includes format reward calculation and answer reward calculation. The format reward calculation is used to measure whether the output format of the multimodal large model is correct, and the answer reward calculation is used to measure whether the reasoning answer prediction information of the multimodal large model is correct.

[0017] Optionally, the task reasoning module is further configured to: The target video is subjected to keyframe extraction using the multimodal large model to obtain a video thumbnail sequence. The video thumbnail sequence is then segmented, and each segmented video thumbnail sequence segment is taken as a video segment.

[0018] Optionally, the model training module is further configured to: The original video and the task problem based on the original video are obtained, and the key frames of the original video are extracted to form a video thumbnail frame sequence sample. The first general large model is used to segment the video thumbnail sequence samples based on the task problem to obtain multiple video thumbnail sequence segments and corresponding temporal information samples, as well as generate visual semantic information samples for each video thumbnail sequence segment. The visual semantic information samples of the video thumbnail frame sequence fragments are reviewed and rewritten using the second general large model to correct the visual semantic information samples. The first general large model generates a task reasoning chain and a task answer based on the video thumbnail frame sequence fragments, the corresponding temporal information samples, and the corresponding visual semantic information samples. Based on the video thumbnail frame sequence, the multiple video thumbnail frame sequence segments and their corresponding temporal information samples and visual semantic information samples, the task question, the task reasoning chain and the task answer, a video reasoning sample set is constructed, and a video reasoning sample set is constructed based on the video reasoning sample.

[0019] Optionally, the model training module is further configured to: A pre-defined scoring model performs multi-dimensional scoring based on the visual semantic information samples and corresponding temporal information samples, the task reasoning chain, and the task answer. The multi-dimensional scoring includes logical consistency, temporal rationality, and language fluency. The logical consistency score is used to evaluate whether there is logical consistency between the visual semantic information samples and the task reasoning chain, and between the task reasoning chain and the task answer. The temporal rationality score is used to evaluate whether the task reasoning chain is temporally reasonable based on the temporal information samples corresponding to the visual semantic information samples. The language fluency score is used to evaluate whether the task answer is linguistically fluent. The video inference samples are filtered based on multi-dimensional scoring, and the remaining video inference samples are used to construct the video inference sample set.

[0020] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described video inference method based on a multimodal large model.

[0021] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described video inference method based on a multimodal large model.

[0022] By employing the above technical solutions, this application provides a video reasoning method, apparatus, storage medium, and computer device based on a multimodal large model. First, the target video to be reasoned and the reasoning question are obtained. Then, prompt words are constructed based on these two elements and input into a pre-trained multimodal large model. The model segments the target video, analyzes the visual semantic information of each segment, constructs a reasoning chain based on temporal and semantic information to derive the answer, and finally outputs the reasoning answer and the reasoning chain. This application avoids information omissions and misunderstandings by segmenting segments and analyzing information using a multimodal large model. It comprehensively considers temporal and semantic information for answer reasoning, improving the accuracy of video reasoning. Furthermore, the output of the reasoning chain makes the process transparent, facilitating understanding of the decision-making basis by medical personnel and meeting the complex reasoning needs of medical videos.

[0023] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The illustration shows a flowchart of a video inference method based on a multimodal large model provided in an embodiment of this application; Figure 2 This illustration shows a flowchart of another video inference method based on a multimodal large model provided in an embodiment of this application; Figure 3 This illustration shows a structural schematic diagram of a video inference device based on a multimodal large model provided in an embodiment of this application; Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0025] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0026] This embodiment provides a video inference method based on a multimodal large model, such as Figure 1 As shown, the method includes: Step 101: Obtain the target video to be reasoned and the reasoning question.

[0027] In this embodiment, the target video and the reasoning question are first obtained. For example, in a medical scenario, the target video might be a patient's rehabilitation training video, and the reasoning question might be "Which movement in this rehabilitation training is most effective in improving joint mobility?" Secondly, cue words are constructed based on the target video and the reasoning question. The cue words serve to guide the multimodal large model, enabling it to understand the specific requirements of the reasoning task. For example, for the reasoning question of the aforementioned rehabilitation training video, the cue words could include descriptions of rehabilitation training movements, joint mobility, and other related concepts, as well as instructions requiring the model to analyze video segments, extract visual semantic information, and perform reasoning.

[0028] Step 102: Construct prompt words based on the target video and the reasoning question, input the prompt words into a pre-trained multimodal large model, and use the multimodal large model to segment the target video, perform visual semantic information analysis on each video segment, and perform answer reasoning based on the reasoning chain for the reasoning question according to the temporal information of the target video and the corresponding visual semantic information.

[0029] Next, the constructed prompts are input into a pre-trained multimodal large-scale model. This model possesses powerful information processing capabilities, simultaneously handling multiple modalities of information from the video, such as images and audio. In this step, the multimodal large-scale model first segments the target video, dividing the lengthy video into multiple segments with clear semantics. This facilitates more accurate analysis of the video content. Then, visual semantic information analysis is performed on each video segment to extract key information related to the reasoning problem, such as the posture, amplitude, and frequency of the actions. Finally, based on the temporal information and corresponding visual semantic information of the target video, a complete reasoning chain is constructed to progressively deduce the answer to the reasoning problem.

[0030] Step 103: Output the reasoning answer and reasoning chain of the multimodal large model.

[0031] Finally, the inference results of the multimodal large model are presented. The inference answer directly addresses the inference question, providing clear conclusions for medical personnel or other relevant parties. Simultaneously, the output inference chain demonstrates the detailed process by which the model arrives at the answer, making the inference results interpretable. Medical personnel can understand the model's analytical approach based on the inference chain, judge the rationality and reliability of the answer, and thus better apply it to practical medical decision-making.

[0032] In a specific application scenario, suppose a patient is undergoing joint rehabilitation training. The training includes multiple different types of movements, such as flexion-extension, rotation, and weight-bearing, and the training video is 30 minutes long. Medical personnel pose the following reasoning question: "During this rehabilitation training, how do the range of motion of the flexion-extension movements change in the beginning, middle, and end stages, and how does this change affect the improvement of joint mobility?" First, the 30-minute rehabilitation training video of the patient and the aforementioned reasoning question are obtained. Second, prompts are constructed and input into a multimodal large-scale model: the constructed prompts may include, "This is a joint rehabilitation training video, including flexion-extension, rotation, and weight-bearing movements. Please analyze the changes in the range of motion of the flexion-extension movements in the beginning, middle, and end stages of the video, and, combined with relevant indicators of joint mobility, reason about the impact of these changes on the improvement of joint mobility, constructing a complete reasoning chain." After inputting the prompts into the pre-trained multimodal large-scale model, the model first segments the video into multiple segments, then analyzes the visual semantic information of the flexion-extension movements in each stage, such as changes in joint angle and the smoothness of the movements, constructing a reasoning chain based on temporal information. For example, the inference chain might show that the initial flexion and extension movements have a small range of motion, resulting in minimal improvement in joint mobility; the middle stage gradually increases the range of motion, leading to some improvement in joint mobility; and the final stage reaches a large range of motion, resulting in significant improvement in joint mobility. Finally, the model outputs an inference answer, such as, "In this rehabilitation training, the patient's initial flexion and extension movements had a small range of motion, resulting in limited improvement in joint mobility; the middle stage gradually increased the range of motion, leading to some improvement in joint mobility; and the final stage reached a large range of motion, resulting in significant improvement in joint mobility." It also outputs a detailed inference chain, demonstrating how the model infers based on the visual semantic and temporal information of the video clips, providing medical personnel with comprehensive analytical results.

[0033] By applying the technical solution of this embodiment, the target video to be reasoned and the reasoning question are first obtained. Then, prompt words are constructed based on the two and input into a pre-trained multimodal large model. The model segments the target video, analyzes the visual semantic information of each segment, constructs a reasoning chain based on temporal and semantic information to derive the answer, and finally outputs the reasoning answer and the reasoning chain. This embodiment of the application, by segmenting segments and analyzing information through a multimodal large model, avoids information omissions and misunderstandings, comprehensively considers temporal and semantic information for answer reasoning, improves the accuracy of video reasoning, and outputs the reasoning chain to make the process transparent, making it easier for medical personnel to understand the decision-making basis and meeting the complex reasoning needs of medical videos.

[0034] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another video reasoning method based on a multimodal large model is provided, such as... Figure 2 As shown, the method includes: Step 201: Obtain a video reasoning sample set, wherein the video reasoning sample set contains multiple video reasoning samples, and each video reasoning sample includes a video thumbnail frame sequence sample, a temporal information sample of multiple video thumbnail frame sequence segments divided based on the video thumbnail frame sequence sample, a visual semantic information sample of each video thumbnail frame sequence segment, the task question, the task answer, and the task reasoning chain of the video thumbnail frame sequence sample.

[0035] In this embodiment, a video inference sample set is collected and organized, which contains multiple video inference samples. Each sample covers several key elements, including video thumbnail frame sequence samples, which are sequences of keyframes extracted from the complete video and represent the main content of the video; temporal information samples of multiple segments divided based on these thumbnail frame sequence samples, clarifying the temporal position and order of each segment in the video; visual semantic information samples corresponding to each segment, containing a semantic understanding of the image content in the segment; and a task question posed for the video thumbnail frame sequence sample, as well as the corresponding task answer and task inference chain, which demonstrates the detailed logical process from video information to the conclusion.

[0036] Step 202: Divide the video inference sample set into a first-stage training sample and a second-stage training sample.

[0037] In this embodiment, the video inference sample set is divided into a first-stage training sample and a second-stage training sample. The first-stage training sample is used to initially train the basic capabilities of the model, while the second-stage training sample is used to enhance the initially trained model and further improve its performance. This phased training helps the model gradually improve its capabilities and enhances the training effect.

[0038] Step 203: Using the first-stage training samples, perform step-by-step training on the video reasoning ability of the multimodal large model. The video reasoning ability includes segmentation ability, visual semantic analysis ability, and answer reasoning ability based on reasoning chains. Specifically, the multimodal large model is trained on segmentation ability using the video thumbnail sequence samples and temporal information samples of multiple video thumbnail sequence segments; the multimodal large model is trained on visual semantic analysis ability using multiple video thumbnail sequence segments and corresponding visual semantic information samples; and the multimodal large model is trained on answer reasoning ability based on reasoning chains using temporal information of multiple video thumbnail sequence segments, corresponding visual semantic information samples, task questions, task answers, and task reasoning chains.

[0039] In this embodiment, a multimodal large model is trained stepwise using a one-stage training sample to enhance its video reasoning capabilities. This stepwise training includes three aspects: **Segmentation Capability Training:** Using video thumbnail frame sequence samples and temporal information samples of multiple video thumbnail frame sequence segments, the model learns how to accurately segment video thumbnail frame sequences into multiple segments with reasonable temporal relationships. For example, by learning the segmentation rules of different video types, the model masters how to segment segments based on temporal order and content logic in various scenarios. **Visual Semantic Analysis Capability Training:** Using multiple video thumbnail frame sequence segments and corresponding visual semantic information samples, the model is trained to understand and analyze the visual content in each segment at the semantic level. For example, the model can identify semantic information such as action types, object categories, and scene features in the segments. **Answer Reasoning Capability Training Based on Reasoning Chains:** Using the temporal information of multiple video thumbnail frame sequence segments, corresponding visual semantic information samples, task questions, task answers, and task reasoning chains, the model learns how to construct reasonable reasoning chains and derive the correct answer based on the temporal and semantic information of the video. For example, the model learns to infer the cause or result of an event based on the sequence of actions and semantic relationships in a video. Through step-by-step training, the model gradually masters the key capabilities of video reasoning, laying the foundation for subsequent comprehensive reasoning and enabling the model to perform analysis and reasoning in an orderly manner when dealing with complex video reasoning tasks.

[0040] Step 204: Utilize the two-stage training samples to perform enhanced training on the video reasoning capabilities of the multimodal large model that has completed step-by-step training; wherein, the enhanced training process includes: Based on the video thumbnail frame sequence samples and the task question, a prompt word sample is constructed. The prompt word sample is input into the multimodal large model that has completed step-by-step training. The multimodal large model that has completed step-by-step training is guided to output task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information according to the process of frame sequence segment segmentation, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information. Rewards are calculated on the output information of the multimodal large model that has completed step-by-step training, and the parameters of the multimodal large model that has completed step-by-step training are optimized based on the calculated reward value.

[0041] In this embodiment, a two-stage training sample is used to enhance the video reasoning capabilities of a multimodal large-scale model that has completed step-by-step training. Specifically, prompt word samples are constructed based on video thumbnail frame sequence samples and task questions. These prompt word samples are input into the multimodal large-scale model that has completed step-by-step training, guiding the model to follow the process of frame sequence segmentation, visual semantic information analysis, and answer reasoning based on inference chains. The model outputs task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic prediction information of video thumbnail frame sequence segments, and inference chain prediction information. For example, for a new video reasoning task, the model first performs segmentation prediction of the video based on the prompt words, then analyzes the visual semantics of each segment, and finally constructs an inference chain and provides an answer prediction. Rewards are calculated for the various prediction information output by the model, and the model parameters are optimized based on the calculated reward values. The closer the model's prediction information is to the real information (task answer, temporal information, visual semantic information, inference chain, etc.), the higher the reward value. The model will adjust its parameters based on the reward value to improve the accuracy of future predictions. This allows for enhanced training to further improve model performance. Through reward mechanisms, the model can continuously optimize itself, adapt to more complex video reasoning scenarios, and improve the accuracy and reliability of reasoning.

[0042] In this embodiment of the application, optionally, the reward calculation includes format reward calculation and answer reward calculation. The format reward calculation is used to measure whether the output format of the multimodal large model is correct, and the answer reward calculation is used to measure whether the reasoning answer prediction information of the multimodal large model is correct.

[0043] In this embodiment, reward calculation is subdivided into format reward calculation and answer reward calculation, which evaluate the model output from different dimensions. Format reward calculation mainly evaluates whether the format of the multimodal large model's output information is correct. In video inference tasks, the model needs to output task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic prediction information of video thumbnail frame sequence segments, and inference chain prediction information according to specific formats. For example, the inference chain needs to be presented in a specific logical structure or symbolic representation, and temporal information may require output according to a specific time format. Format reward calculation checks whether the model's output strictly follows these preset format requirements. Specifically, a format template can be preset, and the model's output can be compared with the template. If the model's output fully meets the format requirements, a higher format reward value is given; if there are partial format errors, corresponding points are deducted according to the severity and number of errors, and the final format reward score is obtained. For example, if the output format of the inference chain requires each inference step to be separated by a specific symbol, and one step in the model output does not use this symbol, a certain number of points are deducted according to preset rules. This ensures the standardization and consistency of the model's output. Answer reward calculation is used to measure the accuracy of the inference answer predictions of a multimodal large model. In video reasoning tasks, the model needs to construct a reasonable reasoning chain and derive an answer prediction based on the temporal and visual semantic information of video thumbnail frame sequences, combined with the task question. Answer reward calculation evaluates the closeness of this answer prediction to the true answer. Through answer reward calculation, the model can clearly understand its performance in reasoning answers, thereby adjusting parameters in a targeted manner to improve the accuracy of inference.

[0044] In this embodiment of the application, optionally, segmenting the target video using the multimodal large model includes: extracting keyframes from the target video using the multimodal large model to obtain a video thumbnail sequence, segmenting the video thumbnail sequence into segments, and treating each segmented video thumbnail sequence segment as a video segment.

[0045] In this embodiment, when segmenting the target video, firstly, the image processing and analysis capabilities of the multimodal large model are used to extract keyframes from the target video. Keyframes are representative frames that summarize the main content and key information of the video. By extracting keyframes, a large number of redundant similar frames can be removed from the video, transforming the originally continuous and large-scale video into a sequence of video thumbnails composed of keyframes. This step compresses the video data size while retaining the core information of the video. Next, the obtained video thumbnail sequence is segmented. The multimodal large model divides the video thumbnail sequence into multiple parts based on factors such as the temporal relationship, visual content changes, and semantic associations between frames in the video thumbnail sequence. Each part is a video thumbnail sequence segment. Finally, each segmented video thumbnail sequence segment is treated as an independent video segment, thus completing the segmentation process of the target video.

[0046] Optionally, in this embodiment of the application, obtaining the video inference sample set includes: The original video and the task problem based on the original video are obtained, and the key frames of the original video are extracted to form a video thumbnail frame sequence sample. The first general large model is used to segment the video thumbnail sequence samples based on the task problem to obtain multiple video thumbnail sequence segments and corresponding temporal information samples, as well as generate visual semantic information samples for each video thumbnail sequence segment. The visual semantic information samples of the video thumbnail frame sequence fragments are reviewed and rewritten using the second general large model to correct the visual semantic information samples. The first general large model generates a task reasoning chain and a task answer based on the video thumbnail frame sequence fragments, the corresponding temporal information samples, and the corresponding visual semantic information samples. Based on the video thumbnail frame sequence, the multiple video thumbnail frame sequence segments and their corresponding temporal information samples and visual semantic information samples, the task question, the task reasoning chain and the task answer, a video reasoning sample set is constructed, and a video reasoning sample set is constructed based on the video reasoning sample.

[0047] In this embodiment, firstly, the original video and the task problem set for the original video are acquired. Keyframes are extracted from the original video to form a video thumbnail frame sequence sample. Keyframes can summarize the main content and key information of the video. By extracting keyframes, the originally large and continuous video data is compressed into a more concise and representative thumbnail frame sequence, which facilitates subsequent processing. Secondly, a first general large model (e.g., the Gemini-2.5-Pro ​​model) is used to segment the video thumbnail frame sequence sample based on the task problem. This process will obtain multiple video thumbnail frame sequence segments and the corresponding temporal information samples of each segment, while generating visual semantic information samples for each video thumbnail frame sequence segment. The first general large model performs logical analysis and understanding of the video content according to the guidance of the task problem, determines reasonable segment segmentation points, and extracts the temporal and visual semantic information of the segments. Then, a second general large model (e.g., the GPT-4o model) is used to review and rewrite the visual semantic information samples of the video thumbnail frame sequence segments, and corrects the visual semantic information samples. The second general-purpose model can review and optimize the visual semantic information generated by the first general-purpose model, ensuring the accuracy and completeness of the information. Finally, the first general-purpose model is used again to generate a task reasoning chain and task answer based on video thumbnail frame sequence segments, corresponding temporal information samples, and corresponding visual semantic information samples. The first general-purpose model comprehensively utilizes various information from the segments to construct a logical reasoning process and derive the final task answer. Thus, video reasoning samples are constructed based on video thumbnail frame sequences, multiple video thumbnail frame sequence segments and their corresponding temporal and visual semantic information samples, task questions, task reasoning chains, and task answers. Then, multiple such video reasoning samples are combined to construct a video reasoning sample set, providing data support for subsequent model training. Training the model based on such a sample set enables the model to learn more accurate and comprehensive video reasoning capabilities, improving the model's performance and generalization ability on video reasoning tasks.

[0048] Optionally, in this embodiment of the application, constructing a video inference sample set based on the video inference samples includes: A pre-defined scoring model performs multi-dimensional scoring based on the visual semantic information samples and corresponding temporal information samples, the task reasoning chain, and the task answer. The multi-dimensional scoring includes logical consistency, temporal rationality, and language fluency. The logical consistency score is used to evaluate whether there is logical consistency between the visual semantic information samples and the task reasoning chain, and between the task reasoning chain and the task answer. The temporal rationality score is used to evaluate whether the task reasoning chain is temporally reasonable based on the temporal information samples corresponding to the visual semantic information samples. The language fluency score is used to evaluate whether the task answer is linguistically fluent. The video inference samples are filtered based on multi-dimensional scoring, and the remaining video inference samples are used to construct the video inference sample set.

[0049] In this embodiment, the quality of the video reasoning sample set can be optimized. The large scoring model quantifies the scores based on three key dimensions: Logical Consistency: This assesses the logical correlation between the visual semantic information samples and the task reasoning chain, as well as the logical coherence between the reasoning chain and the task answer. For example, if the visual semantic information shows "the patient's range of motion gradually increases," but the reasoning chain concludes "joint mobility has not improved," the logical consistency score will decrease. Temporal Reasonableness: This verifies whether the reasoning chain conforms to the logical order of time by combining temporal information samples. For example, if the sequence "warm-up exercises first, then strength training" in a rehabilitation training video is reversed, the temporal reasonableness score will be affected. Language Fluency: This checks whether the language expression of the task answer is natural and fluent, avoiding grammatical errors or ambiguous expressions. Finally, based on the multi-dimensional scoring results, low-quality samples (such as those with logical contradictions, disordered temporal sequences, or unclear expressions) are removed, and high-scoring samples are retained to construct the final video reasoning sample set. For example, if a sample's logical consistency score is below a threshold, it may be filtered out due to a break in the reasoning chain. By using multi-dimensional scoring to screen samples, we can ensure that each sample in the sample set meets high-quality standards in terms of logic, timing, and language, thus providing a more reliable data foundation for model training.

[0050] Furthermore, as Figure 1 In terms of specific implementation, this application provides a video inference device based on a multimodal large model, such as... Figure 3 As shown, the device includes: The data acquisition module is used to acquire the target video to be inferred and the inference question; The task reasoning module is used to construct prompt words based on the target video and the reasoning question, input the prompt words into a pre-trained multimodal large model, and use the multimodal large model to segment the target video, perform visual semantic information analysis on each video segment, and perform answer reasoning based on the reasoning chain for the reasoning question according to the temporal information and corresponding visual semantic information of the target video. The answer output module is used to output the reasoning answer and reasoning chain of the multimodal large model.

[0051] Optionally, the apparatus further includes: a model training module, used for: Obtain a video reasoning sample set, wherein the video reasoning sample set contains multiple video reasoning samples, each video reasoning sample includes a video thumbnail frame sequence sample, a temporal information sample of multiple video thumbnail frame sequence segments divided based on the video thumbnail frame sequence sample, a visual semantic information sample of each video thumbnail frame sequence segment, the task question, the task answer, and the task reasoning chain of the video thumbnail frame sequence sample. The multimodal large model is trained stepwise based on the video reasoning sample set to improve its video reasoning capabilities, which include segmentation, visual semantic analysis, and answer reasoning based on reasoning chains. Specifically, the multimodal large model is trained to improve its segmentation capabilities using the video thumbnail sequence samples and temporal information samples of multiple video thumbnail sequence segments; its visual semantic analysis capabilities are trained using multiple video thumbnail sequence segments and corresponding visual semantic information samples; and its answer reasoning capabilities are trained using the temporal information of multiple video thumbnail sequence segments, corresponding visual semantic information samples, task questions, task answers, and task reasoning chains.

[0052] Optionally, the model training module is further configured to: The video inference sample set is divided into a first-stage training sample and a second-stage training sample. The multimodal large model is trained step-by-step using the first-stage training samples to improve its video reasoning capabilities. The video reasoning capability of the multimodal large model, which has completed step-by-step training, is enhanced using the two-stage training samples; wherein, the enhancement training process includes: Based on the video thumbnail frame sequence samples and the task question, a prompt word sample is constructed. The prompt word sample is input into the multimodal large model that has completed step-by-step training. The multimodal large model that has completed step-by-step training is guided to output task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information according to the process of frame sequence segment segmentation, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information. Rewards are calculated on the output information of the multimodal large model that has completed step-by-step training, and the parameters of the multimodal large model that has completed step-by-step training are optimized based on the calculated reward value.

[0053] Optionally, the reward calculation includes format reward calculation and answer reward calculation. The format reward calculation is used to measure whether the output format of the multimodal large model is correct, and the answer reward calculation is used to measure whether the reasoning answer prediction information of the multimodal large model is correct.

[0054] Optionally, the task reasoning module is further configured to: The target video is subjected to keyframe extraction using the multimodal large model to obtain a video thumbnail sequence. The video thumbnail sequence is then segmented, and each segmented video thumbnail sequence segment is taken as a video segment.

[0055] Optionally, the model training module is further configured to: The original video and the task problem based on the original video are obtained, and the key frames of the original video are extracted to form a video thumbnail frame sequence sample. The first general large model is used to segment the video thumbnail sequence samples based on the task problem to obtain multiple video thumbnail sequence segments and corresponding temporal information samples, as well as generate visual semantic information samples for each video thumbnail sequence segment. The visual semantic information samples of the video thumbnail frame sequence fragments are reviewed and rewritten using the second general large model to correct the visual semantic information samples. The first general large model generates a task reasoning chain and a task answer based on the video thumbnail frame sequence fragments, the corresponding temporal information samples, and the corresponding visual semantic information samples. Based on the video thumbnail frame sequence, the multiple video thumbnail frame sequence segments and their corresponding temporal information samples and visual semantic information samples, the task question, the task reasoning chain and the task answer, a video reasoning sample set is constructed, and a video reasoning sample set is constructed based on the video reasoning sample.

[0056] Optionally, the model training module is further configured to: A pre-defined scoring model performs multi-dimensional scoring based on the visual semantic information samples and corresponding temporal information samples, the task reasoning chain, and the task answer. The multi-dimensional scoring includes logical consistency, temporal rationality, and language fluency. The logical consistency score is used to evaluate whether there is logical consistency between the visual semantic information samples and the task reasoning chain, and between the task reasoning chain and the task answer. The temporal rationality score is used to evaluate whether the task reasoning chain is temporally reasonable based on the temporal information samples corresponding to the visual semantic information samples. The language fluency score is used to evaluate whether the task answer is linguistically fluent. The video inference samples are filtered based on multi-dimensional scoring, and the remaining video inference samples are used to construct the video inference sample set.

[0057] It should be noted that other corresponding descriptions of the functional units involved in the video inference device based on a multimodal large model provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0058] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 4 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0059] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0060] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0061] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0062] It should be noted that the user personal information involved in the embodiments of this application is all authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals. It should be noted that if any software tools or components other than those of this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.

[0063] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0064] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0065] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video reasoning method based on a multimodal large model, characterized in that, The method includes: Obtain the target video to be reasoned and the reasoning question; Based on the target video and the reasoning question, prompt words are constructed and input into a pre-trained multimodal large model. The multimodal large model is used to segment the target video, perform visual semantic information analysis on each video segment, and perform answer reasoning based on the reasoning chain based on the temporal information and corresponding visual semantic information of the target video. Output the reasoning answer and reasoning chain of the multimodal large model.

2. The method according to claim 1, characterized in that, The training methods for the multimodal large model include: Obtain a video reasoning sample set, wherein the video reasoning sample set contains multiple video reasoning samples, each video reasoning sample includes a video thumbnail frame sequence sample, a temporal information sample of multiple video thumbnail frame sequence segments divided based on the video thumbnail frame sequence sample, a visual semantic information sample of each video thumbnail frame sequence segment, the task question, the task answer, and the task reasoning chain of the video thumbnail frame sequence sample. The multimodal large model is trained stepwise based on the video reasoning sample set to improve its video reasoning capabilities, which include segmentation, visual semantic analysis, and answer reasoning based on reasoning chains. Specifically, the multimodal large model is trained to improve its segmentation capabilities using the video thumbnail sequence samples and temporal information samples of multiple video thumbnail sequence segments; its visual semantic analysis capabilities are trained using multiple video thumbnail sequence segments and corresponding visual semantic information samples; and its answer reasoning capabilities are trained using the temporal information of multiple video thumbnail sequence segments, corresponding visual semantic information samples, task questions, task answers, and task reasoning chains.

3. The method according to claim 2, characterized in that, After obtaining the video inference sample set, the method further includes: The video inference sample set is divided into a first-stage training sample and a second-stage training sample. The stepwise training of the video reasoning capability of the multimodal large model based on the video reasoning sample set includes: The multimodal large model is trained step-by-step using the first-stage training samples to improve its video reasoning capabilities. After the stepwise training of the video reasoning capability of the multimodal large model based on the video reasoning sample set, the method further includes: The video reasoning capability of the multimodal large model, which has completed step-by-step training, is enhanced using the two-stage training samples; wherein, the enhancement training process includes: Based on the video thumbnail frame sequence samples and the task question, a prompt word sample is constructed. The prompt word sample is input into the multimodal large model that has completed step-by-step training. The multimodal large model that has completed step-by-step training is guided to output task answer prediction information, temporal prediction information of video thumbnail frame sequence segments, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information according to the process of frame sequence segment segmentation, visual semantic information prediction information of video thumbnail frame sequence segments, and inference chain prediction information. Rewards are calculated on the output information of the multimodal large model that has completed step-by-step training, and the parameters of the multimodal large model that has completed step-by-step training are optimized based on the calculated reward value.

4. The method according to claim 3, characterized in that, The reward calculation includes format reward calculation and answer reward calculation. The format reward calculation is used to measure whether the output format of the multimodal large model is correct, and the answer reward calculation is used to measure whether the inference answer prediction information of the multimodal large model is correct.

5. The method according to claim 2, characterized in that, The target video is segmented using the multimodal large model, including: The target video is subjected to keyframe extraction using the multimodal large model to obtain a video thumbnail sequence. The video thumbnail sequence is then segmented, and each segmented video thumbnail sequence segment is taken as a video segment.

6. The method according to any one of claims 2 to 5, characterized in that, The acquisition of the video inference sample set includes: The original video and the task problem based on the original video are obtained, and the key frames of the original video are extracted to form a video thumbnail frame sequence sample. The first general large model is used to segment the video thumbnail sequence samples based on the task problem to obtain multiple video thumbnail sequence segments and corresponding temporal information samples, as well as generate visual semantic information samples for each video thumbnail sequence segment. The visual semantic information samples of the video thumbnail sequence fragments are reviewed and rewritten using the second general large model to correct the visual semantic information samples. The first general large model generates a task reasoning chain and a task answer based on the video thumbnail frame sequence fragments, the corresponding temporal information samples, and the corresponding visual semantic information samples. Based on the video thumbnail frame sequence, the multiple video thumbnail frame sequence segments and their corresponding temporal information samples and visual semantic information samples, the task question, the task reasoning chain and the task answer, a video reasoning sample set is constructed, and a video reasoning sample set is constructed based on the video reasoning sample.

7. The method according to claim 6, characterized in that, The construction of the video inference sample set based on the video inference samples includes: A pre-defined scoring model performs multi-dimensional scoring based on the visual semantic information samples and corresponding temporal information samples, the task reasoning chain, and the task answer. The multi-dimensional scoring includes logical consistency, temporal rationality, and language fluency. The logical consistency score is used to evaluate whether there is logical consistency between the visual semantic information samples and the task reasoning chain, and between the task reasoning chain and the task answer. The temporal rationality score is used to evaluate whether the task reasoning chain is temporally reasonable based on the temporal information samples corresponding to the visual semantic information samples. The language fluency score is used to evaluate whether the task answer is linguistically fluent. The video inference samples are filtered based on multi-dimensional scoring, and the remaining video inference samples are used to construct the video inference sample set.

8. A video inference device based on a multimodal large model, characterized in that, The device includes: The data acquisition module is used to acquire the target video to be inferred and the inference question; The task reasoning module is used to construct prompt words based on the target video and the reasoning question, input the prompt words into a pre-trained multimodal large model, and use the multimodal large model to segment the target video, perform visual semantic information analysis on each video segment, and perform answer reasoning based on the reasoning chain for the reasoning question according to the temporal information and corresponding visual semantic information of the target video. The answer output module is used to output the reasoning answer and reasoning chain of the multimodal large model.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images