Video generation evaluation method and system based on MLLM
By using a video generation evaluation method based on MLLM, which converts keyframes into text descriptions and performs chained queries and few-sample comparisons, the problem of unstable evaluation in existing technologies is solved, and highly consistent and interpretable automated evaluation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NINGBO ARTIFICIAL INTELLIGENCE RES INST OF SHANGHAI JIAOTONG UNIV
- Filing Date
- 2025-12-18
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies using MLLM for video generation evaluation suffer from problems such as failing to accurately reflect human preferences, insufficient evaluation granularity, and severe cross-modal illusions, leading to unstable automated evaluation.
By receiving video data and extracting keyframes, MLLM is used to convert it into text descriptions, generating a chained query question set for question-by-question answering, and performing reflective re-description. The scores are calibrated by combining few-sample comparisons, and the scores of each dimension are aggregated to generate a comprehensive evaluation result.
It significantly improves the stability of assessment and consistency with human preferences, reduces cross-modal illusions and information omissions, achieves high consistency between automated assessment and human scoring, and has scalability and practicality.
Smart Images
Figure CN121884218A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal evaluation technology at the intersection of computer vision and natural language processing, and in particular to a video generation evaluation method and system based on MLLM. Background Technology
[0002] In recent years, with the rapid development of deep learning and artificial intelligence technologies, generative AI, centered on large models, has received widespread attention and application. Among these, video generation technology, as an important branch of generative AI, has demonstrated enormous potential in content creation, virtual reality, and advertising production. This technology generates dynamic video content from input data (text, images, or text plus images, etc.) using deep learning models, providing new possibilities for automated creation and personalized content customization. However, how to effectively evaluate the quality, realism, and application value of generated videos remains a highly relevant research topic.
[0003] The development of video generation technology can be simply divided into the following three stages: 1. Initial Exploration Phase: Early research on video model generation using traditional methods and simple models primarily relied on traditional computer vision methods, such as manual modeling and optical flow estimation, to generate low-quality, simple dynamic sequences. These methods required a large amount of prior knowledge and were difficult to extend to complex scenes, resulting in content lacking realism and diversity.
[0004] 2. The introduction of deep learning: a leap forward in the quality of generated videos. Deep learning technology brought about a qualitative change in video generation models, especially the introduction of Generative Adversarial Networks (GANs). Combined with temporal models (such as LSTM and GRU), it significantly improved the generation quality and further improved the temporal consistency of videos.
[0005] 3. Transformers and Large Models: A New Era for Generative Video. The Transformer architecture and large-scale pre-trained models have driven the development of generative video technology. VideoGPT and VideoTransformer utilize global information in both temporal and spatial dimensions to generate high-quality videos. Diffusion models generate high-resolution, realistic videos through denoising processes and support multimodal generation from text to video, further improving the generation effect and the breadth of application scenarios.
[0006] With the continuous development of video generation technology, the quality and complexity of generated videos have significantly improved. From initially only being able to generate low-quality, unresponsive simple dynamic sequences, to now being able to generate high-quality videos with complex content and a high degree of conformity to input requirements, video generation technology has achieved revolutionary progress. Currently, generated videos are no longer simple superpositions of dynamic images, but rather image sequences with content coherence, visual realism, and semantic consistency, which can truly be called "videos."
[0007] With advancements in text-to-video generation technology, evaluating the quality of generated videos and text-to-video consistency has become essential. Existing evaluation methods mainly include automated metrics based on measures / embeddings (such as FVD and CLIP embedding) and subjective evaluations based on human annotation. While metric-based methods are automated, they often fail to align adequately with human preferences; human annotation is time-consuming, labor-intensive, and suffers from poor consistency.
[0008] Recent research using large language models (LLM) or multimodal large models (MLLM) for visual evaluation has shown better potential for aligning with human preferences, but it faces two major challenges: first, direct cross-modal comparisons (text and video) can easily lead to MLLM illusions and missed details; second, converting qualitative textual comments into explicit quantitative scores has fuzzy discrimination boundaries, causing the model to tend to give intermediate / average scores and making it difficult to distinguish subtle quality differences. Existing work has proposed preliminary solutions (such as single-round scoring, description + question answering, etc.), but they are still not systematic and stable enough.
[0009] Therefore, those skilled in the art are dedicated to developing a new video generation and evaluation method and system to address the aforementioned deficiencies in the existing technology. Summary of the Invention
[0010] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to solve the problems that are easy to occur in the visual evaluation of the prior art using MLLM, such as the inability to accurately reflect human preferences, insufficient evaluation granularity, and serious cross-modal illusion, and improve the stability of automated video evaluation.
[0011] To achieve the above objectives, the present invention provides a video generation and evaluation method based on MLLM, characterized in that the method includes the following steps: Step 1: Receive the video data to be evaluated and the corresponding text prompts; Step 2: Preprocess the video data and extract the keyframe sequence; Step 3: Use MLLM to convert the keyframe sequence into a text description, which includes a detailed description and a summary description; Step 4: Based on the text prompts and text descriptions, generate a chained query question set, which contains questions from multiple dimensions; Step 5: Use MLLM to answer each question in the chained query question set, and perform reflective re-description during the answering process to correct or supplement the video content description; Step 6: For multiple candidate videos with the same text prompt, perform a few-sample comparison to calibrate the score, and obtain the score result through relative comparison; Step 7: Aggregate the scores from each dimension, generate a comprehensive evaluation result, and output it.
[0012] Furthermore, the keyframe extraction strategy in step 2 is to extract 1-2 frames per second, and to normalize the resolution and unify the color space of the keyframes, remove duplicate frames, and unify their size.
[0013] Furthermore, step 4 includes the following sub-steps: Step 4.1: Load the preset query dimension template, which includes subject consistency, action consistency, scene consistency, time logic consistency, image quality and visual stability; Step 4.2: Based on the text prompts and text descriptions, automatically generate multiple questions under each dimension to form the chained query question set composed of question chains.
[0014] Furthermore, the reflective re-description in step 5 includes: before each round of question and answer, MLLM re-summarizes the video content, generates supplementary descriptions, and compares them with previous descriptions to correct omissions or errors.
[0015] Furthermore, the few-sample comparison calibration score in step 6 includes: Multiple candidate videos under the same text prompt are constructed into a batch. One candidate video is selected as a reference sample, and the remaining candidate videos are compared with the reference sample. The scores are adjusted according to the relative advantages and disadvantages. If the compared sample is better than the reference sample, a point is added; if it is worse, a point is deducted; if they are equal, the reference score is maintained.
[0016] Furthermore, the size of the extracted keyframes is standardized to 512×512 pixels, the color space is standardized to RGB, and the criterion for judging duplicate frames is adjacent frames with a change of less than 5% of a pre-set threshold.
[0017] Furthermore, the comprehensive evaluation result in step 7 includes: the text prompts, the keyframe sequence, scores for each dimension and the overall score, and the overall comments generated by MLLM.
[0018] Furthermore, it also includes step 8, verifying the stability of the results by performing three independent evaluations on the same video and calculating the average score and standard deviation. If the standard deviation exceeds 0.2, a re-evaluation or manual review will be automatically triggered.
[0019] The present invention also provides an MLLM-based video generation evaluation system for implementing the MLLM-based video generation evaluation method as described in any one of the preceding claims, comprising: The video preprocessing module is used to receive the video data to be evaluated and extract the keyframe sequence; A modality conversion module is used to convert the keyframe sequence into a text description using MLLM, the text description including a detailed description and a summary description; The query generation module is used to generate the chained query question set based on the text prompts and the text description; The Response and Reflection module is used to answer each question of the chained query question set using MLLM and perform reflective re-description. The few-sample comparison calibration module is used to perform relative comparison scoring on multiple candidate videos with the same text prompt; The scoring aggregation and output module is used to aggregate scores from various dimensions and output the comprehensive evaluation result.
[0020] Furthermore, the detailed description generated by the modality conversion module includes subject, action, scene, time change, lighting and color information, and the summary description is a general statement.
[0021] The MLLM-based video generation and evaluation system and method provided by this invention have at least the following technical effects: 1. The technical solution provided by this invention can significantly improve human alignment. Through a joint mechanism of "video description - chained question answering - few-shot scoring" driven by a multimodal large language model, it achieves a high degree of alignment between evaluation criteria and human perception, enabling automated scoring to maintain high consistency with human scoring in the subjective dimension. This solves the problem that traditional automatic evaluation methods have low correlation between evaluation results and human subjective preferences because they are mostly based on low-level visual features or general semantic similarity (such as FVD, CLIPScore, etc.).
[0022] 2. The technical solution provided by this invention can reduce cross-modal illusions and information omissions. By converting video content into text description through the "video modality → text modality" conversion step, the video content is first structured into text description, and then text-to-text comparison is performed, which significantly reduces the cross-modal semantic mismatch rate, making the model judgment more objective and interpretable. This solves the problem that existing MLLM direct comparison of "text-video" often misses details or produces illusory descriptions.
[0023] 3. The technical solution provided by this invention has strong scalability and practicality. Through the modular design of the evaluation process, it can be adapted to any multimodal large language model with visual understanding capabilities, and supports the rapid replacement of different generation models or evaluation dimensions. It is convenient to deploy and use directly in academic research, commercial systems and automatic model tuning scenarios.
[0024] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description
[0025] Figure 1 This is a flowchart of an evaluation method according to a preferred embodiment of the present invention; Figure 2 This is a flowchart of a preferred embodiment of the present invention; Figure 3 This is a flowchart of a preferred embodiment of the small-sample scoring method of the present invention; Figure 4 This is a complete flowchart of video evaluation according to a preferred embodiment of the present invention. Detailed Implementation
[0026] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.
[0027] In existing research on visual evaluation using large language models (LLM) or multimodal large models (MLLM), one challenge is that direct cross-modal comparisons (text and video) can easily lead to MLLM illusions and missed details. Another challenge is that converting qualitative textual comments into explicit quantitative scores has fuzzy discrimination boundaries, causing the model to tend to give intermediate / average scores and making it difficult to distinguish subtle quality differences.
[0028] This invention utilizes a multimodal large language model (MLLM) to automatically evaluate the quality and conditional consistency of text-to-video generation model outputs. It achieves this by (A) structuring video content into text to reduce the illusion problem caused by direct cross-modal comparisons, (B) employing a "chain-of-query" approach for multi-round, dimension-by-dimensional probing, and (C) introducing "multimodal few-shot calibration / comparison scoring" to enhance score perception. This results in a highly consistent, stable, and low-cost automated evaluation of generated videos across two main dimensions: "video-conditional alignment" and "video quality," aligning with human preferences.
[0029] Example 1 like Figure 1 and 2 As shown, this embodiment of the invention provides a video generation evaluation method based on MLLM, which combines chained queries and a few-sample comparison mechanism to achieve automated, interpretable, and highly consistent evaluation of text-to-video generation results. Specifically, the method includes the following steps: Step 1: Receive the video data to be evaluated and the corresponding text prompts; Step 2: Preprocess the video data and extract the keyframe sequence; Step 3: Use MLLM to convert the keyframe sequence into a text description, which includes a detailed description and a summary description; Step 4: Based on the text prompts and text descriptions, generate a chained query question set, which contains questions from multiple dimensions; Step 5: Use MLLM to answer each question in the chained query question set, and reflectively re-describe during the answering process to correct or supplement the video content description; Step 6: For multiple candidate videos with the same text prompt, perform a few-sample comparison to calibrate the score, and obtain the score result through relative comparison; Step 7: Aggregate the scores from each dimension, generate a comprehensive evaluation result, and output it.
[0030] Specifically, in one example of step 1, the text prompt and the generated video are as follows: Prompt: "A dog is running across a snowy field." Video: A video clip output by the generated model, approximately 5 seconds long, with a frame rate of 30 FPS.
[0031] Example 2 Based on Example 1, the keyframe extraction strategy in step 2 is to extract 1-2 frames per second, and to normalize the resolution and unify the color space of the keyframes, remove duplicate frames and unify their size.
[0032] Specifically, the keyframe sampling interval is 1 frame every 0.5 or 1 second; the sampling strategy is to sample uniformly over time, prioritizing the coverage of the start and end times of the action; the output frame sequence consists of approximately 10 keyframes, named as follows: .
[0033] Specifically, the extracted keyframes are standardized to 512×512 pixels in size and RGB in color space. The criteria for identifying duplicate frames are adjacent frames with a change of less than 5% of a pre-set threshold.
[0034] After extracting keyframes and preprocessing them, a keyframe sequence is obtained. .
[0035] Example 3 Based on Example 1 or 2, the conversion process in step 3 achieves structured information extraction from video modality to text modality, reducing the bias of direct cross-modal comparisons. Specifically, it includes the following sub-steps: Step 3.1: Input the keyframe sequence into an MLLM, such as GPT-4o or LLaVA; Step 3.2: Generate a detailed description MLLM outputs a detailed natural language description of the video content, including a complete description of details such as the subject, actions, scene objects, time changes, lighting, and color. For example: "A brown dog is running on white snow in the video, moving from left to right. The background is snow and distant trees, the camera is fixed, and the lighting is bright." Step 3.3: Generate summary description MLLM uses a brief description that summarizes the main content of the video in one sentence, such as: "A dog is running in the snow." in, and Used for subsequent semantic consistency evaluation.
[0036] Example 4 Based on Examples 1-3, step 4 includes the following sub-steps: Step 4.1: Load the preset query dimension templates. The query dimension templates include Subject Consistency, Action Consistency, Scene Consistency, Temporal Consistency, and Quality & Coherence. Step 4.2: Based on text prompts and text descriptions, automatically generate multiple questions under each dimension to form a chained query question set composed of question chains, where each question chain is designed around a dimension.
[0037] For example: a chain of problems under the dimension of action consistency: Q1: Is the main subject in the video a dog? Does it match the description in the prompt? Q2: Is running the primary action of a dog? Q3: Are the direction and timing of the actions consistent with the prompts? Q4: Is the background in the video a snowy scene? Q5: Are there any elements in the video that are irrelevant to the prompts? The generated chain of query questions is denoted as Q = {Q1, Q2, …, Q5}. The generated question chain is used to guide MLLM to perform fine-grained reasoning on each dimension step by step.
[0038] Example 5 Based on Examples 1-4, step 5 includes the following sub-steps: Step 5.1: Use MLLM to answer the chained query question set question by question, allowing the model to answer while maintaining the context. Step 5.2: Reflective re-description during the response process to correct or supplement the video content description. Reflective re-description in Step 5 includes: before each round of question and answer, MLLM summarizes the video content, generates supplementary descriptions, and compares them with previous descriptions to correct omissions or errors.
[0039] An example answer is as follows: A1: The subject is indeed a dog, consistent with the hint.
[0040] A2: The dog's main action is running, moving from left to right, which is consistent with the prompt.
[0041] A3: The direction of the movement is correct, and there is no pause or reverse movement.
[0042] A4: The background is snow, consistent with the hint.
[0043] A5: No additional objects mentioned or indicated in the video.
[0044] The final answer chain A = {A1, A2, A3, A4, A5} is obtained and stored for subsequent scoring.
[0045] Example 6 Based on Examples 1-5, the few-sample comparison calibration score in step 6 includes: For multiple candidate video samples under the same text prompt "Prompt", a few-shot batch B = {V1, V2, V3, …} is constructed. One candidate video (e.g., the first video by default) is selected as the reference sample, and the remaining candidate videos are compared with the reference sample. The score is adjusted according to the relative merits: if the compared sample is better than the reference sample, a score is added; if it is worse, a score is deducted; if they are comparable, the reference score is maintained. This method improves the model's sensitivity to score distribution and its ability to distinguish hierarchical levels.
[0046] Specifically, step 6 includes the following sub-steps: Step 6.1: Perform dimensional scoring. Based on the answer content and scoring criteria, map each dimension to a quantified score. An example scoring table is shown below: Step 6.2: Perform few-shot calibration (if there are multiple samples). If there are multiple candidate videos under the same prompt (e.g., from different generative models), the system will group the three videos {V1, V2, V3} into a comparison batch. Using V1 as the reference video, MLLM will score V2 and V3 according to their relative merits, making the final distribution closer to the differences in human scoring.
[0047] Example 7 Building upon Examples 1-6, step 7 weights and summarizes the scores for each dimension to generate a final comprehensive score. This comprehensive evaluation result includes: the original text prompt, keyframe sequence, scores for each dimension and the comprehensive score, and an overall comment generated by MLLM, such as: "The video content matches the prompt description, the movements are natural, and the picture quality is high." It also outputs a dimension distribution table, score confidence level, automatically generated natural language comments, and a question-and-answer chain summary, forming a complete evaluation report. The report can be exported as a JSON or PDF file for archiving and comparison.
[0048] Specifically, it also includes step 8, verifying the stability of the results, performing three independent evaluations on the same video and calculating the average score and standard deviation. If the standard deviation exceeds 0.2, it will automatically trigger a re-evaluation or manual review.
[0049] Example 8 like Figure 4 As shown, this embodiment of the invention also provides a video generation evaluation system based on MLLM, used to implement the video generation evaluation method based on MLLM provided in any of the preceding embodiments, including: The video preprocessing module receives the video data to be evaluated, extracts key frame sequences, performs resolution normalization, optical flow feature extraction, and frame sampling, and generates a visual frame set for subsequent modality conversion.
[0050] The modality conversion module is used to convert keyframe sequences into text descriptions using MLLM. The text descriptions include detailed descriptions and summary descriptions. In other words, MLLM is used to perform semantic understanding and description of the visual frame set, so that the video content is represented in a structured text form for subsequent text alignment evaluation.
[0051] The query generation module is used to generate a chain of query questions based on text prompts and text descriptions. Each dimension contains several questions to detect the consistency between the video and the text prompts (such as actions, colors, scenes, objects, and temporal relationships).
[0052] The Response and Reflection module is used to answer a chain of query questions using MLLM and perform reflective re-description, which involves summarizing the video content and correcting or supplementing previous descriptions to reduce illusions and omissions.
[0053] The few-shot comparison calibration module is used to perform relative comparison scoring on multiple candidate videos with the same text prompt. It constructs a few-shot batch group for relative comparison and calibrates the scoring range by judging the relative quality of the reference video, thereby enhancing the model's ability to distinguish small quality differences.
[0054] The scoring aggregation and output module is used to aggregate scores from various dimensions and output a comprehensive evaluation result. It converts the answers from each dimension into quantitative scores (such as 1–3 or 1–5 points) and aggregates them according to weights to generate a comprehensive score. It also outputs detailed dimension distribution, confidence level and evaluation report.
[0055] Specifically, the detailed description generated by the modality conversion module includes information on the subject, action, scene, time changes, lighting, and color, while the summary description is a concise statement.
[0056] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A video generation evaluation method based on MLLM, characterized in that, The method includes the following steps: Step 1: Receive the video data to be evaluated and the corresponding text prompts; Step 2: Preprocess the video data and extract the keyframe sequence; Step 3: Use MLLM to convert the keyframe sequence into a text description, which includes a detailed description and a summary description; Step 4: Based on the text prompts and text descriptions, generate a chained query question set, which contains questions from multiple dimensions; Step 5: Use MLLM to answer each question in the chained query question set, and perform reflective re-description during the answering process to correct or supplement the video content description; Step 6: For multiple candidate videos with the same text prompt, perform a few-sample comparison to calibrate the score, and obtain the score result through relative comparison; Step 7: Aggregate the scores from each dimension, generate a comprehensive evaluation result, and output it.
2. The MLLM-based video generation and evaluation method as described in claim 1, characterized in that, The keyframe extraction strategy in step 2 is to extract 1-2 frames per second, and to normalize the resolution and color space of the keyframes, remove duplicate frames, and unify their size.
3. The MLLM-based video generation evaluation method as described in claim 1, characterized in that, Step 4 includes the following sub-steps: Step 4.1: Load the preset query dimension template, which includes subject consistency, action consistency, scene consistency, time logic consistency, image quality and visual stability; Step 4.2: Based on the text prompts and text descriptions, automatically generate multiple questions under each dimension to form the chained query question set composed of question chains.
4. The MLLM-based video generation evaluation method as described in claim 1, characterized in that, The reflective re-description in step 5 includes: before each round of question and answer, MLLM re-summarizes the video content, generates supplementary descriptions, and compares them with previous descriptions to correct omissions or errors.
5. The MLLM-based video generation evaluation method as described in claim 1, characterized in that, The few-sample comparison calibration score in step 6 includes: Multiple candidate videos under the same text prompt are constructed into a batch. One candidate video is selected as a reference sample, and the remaining candidate videos are compared with the reference sample. The scores are adjusted according to the relative advantages and disadvantages. If the compared sample is better than the reference sample, a point is added; if it is worse, a point is deducted; if they are equal, the reference score is maintained.
6. The MLLM-based video generation evaluation method as described in claim 2, characterized in that, The extracted keyframes are all sized to 512×512 pixels and use the same color space as RGB. The criteria for identifying duplicate frames are adjacent frames whose changes are less than 5% of a pre-set threshold.
7. The MLLM-based video generation evaluation method as described in claim 1, characterized in that, The comprehensive evaluation result in step 7 includes: the text prompts, the keyframe sequence, scores for each dimension and the overall score, and the overall comments generated by MLLM.
8. The MLLM-based video generation evaluation method as described in claim 1, characterized in that, It also includes step 8, verifying the stability of the results, performing three independent evaluations on the same video and calculating the average score and standard deviation. If the standard deviation exceeds 0.2, a re-evaluation or manual review will be automatically triggered.
9. A video generation evaluation system based on MLLM, used to implement the video generation evaluation method based on MLLM as described in any one of claims 1 to 8, characterized in that, include: The video preprocessing module is used to receive the video data to be evaluated and extract the keyframe sequence; A modality conversion module is used to convert the keyframe sequence into a text description using MLLM, the text description including a detailed description and a summary description; The query generation module is used to generate the chained query question set based on the text prompts and the text description; The Response and Reflection module is used to answer each question of the chained query question set using MLLM and perform reflective re-description. The few-sample comparison calibration module is used to perform relative comparison scoring on multiple candidate videos with the same text prompt; The scoring aggregation and output module is used to aggregate scores from various dimensions and output the comprehensive evaluation result.
10. The MLLM-based video generation and evaluation system as described in claim 9, characterized in that, The detailed description generated by the modality conversion module includes subject, action, scene, time change, lighting and color information, and the summary description is a general statement.