Multimodal automated evaluation method and system for text-to-video

Abstract

The application provides a multi-modal automatic evaluation method and system for text-to-video, wherein the method comprises the following steps: constructing a Q-Save evaluation dataset, wherein the Q-Save evaluation dataset comprises a Prompt set containing dynamic elements, text-to-video, and labeled data, attribution text and MOS scores for each video; constructing an initial model based on a visual-linguistic large model, training the initial model by using the Q-Save evaluation dataset to obtain an evaluation model; taking a to-be-predicted text-to-video as an input of the evaluation model, and performing weighted average processing on the model output to obtain a final prediction score and attribution. The application can guarantee the video quality scoring capability and also provides reliable fine-grained attribution capability.

Description

Technical Field

[0001] This invention relates to the field of video understanding technology, specifically to a multimodal automated evaluation method and system for text-based videos, and also to a corresponding evaluation device and computer-readable storage medium. Background Technology

[0002] Text-to-Video technology is a generative model based on artificial intelligence that enables models to understand text and automatically convert natural language descriptions into continuous, dynamic video footage.

[0003] With the rapid development of generative AI, text-to-video (T2V) content generation is becoming increasingly popular. However, most T2V content has varying degrees of defects, and the cost of manual evaluation is too high. Therefore, automated evaluation technology for text-to-video has emerged.

[0004] Currently, the commonly used automated evaluation technologies mainly include the following two types:

[0005] T2VQA: Subjective alignment datasets and metrics for text-to-video quality assessment;

[0006] Q-Eval-100K: Evaluates the visual quality and alignment level of text-to-visual content.

[0007] in:

[0008] T2VQA, a method that constructs a T2VQA-DB dataset and fused text, uses video alignment (BLIP) and video fidelity (Swin-T) features, and is based on an LLM regression T2VQA model. It has achieved certain results in evaluating the quality of text-generated videos, but the dataset does not cover high-specification videos, limiting the model's generalization; and it lacks a dedicated temporal module, resulting in insufficient accuracy in evaluating dynamic action-type text.

[0009] Q-Eval-100K is a method that constructs a dataset containing Q-Eval-100K and proposes a Q-Eval-Score model trained using CE+MSE loss. Q-Eval-Score has achieved certain results in visual quality and alignment evaluation and can effectively handle long text alignment. However, the evaluation dimension does not include dynamic quality, and the evaluation effect on videos is generally poor. Furthermore, it can only provide scalar results of MOS scores and cannot provide specific attribution explanations.

[0010] The search revealed:

[0011] The paper "Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content" discloses a technique for evaluating the visual quality and alignment of text-to-visual content. However, this technique still has the following technical problems: the evaluation dimension does not include dynamic quality, the evaluation effect on video is generally poor, and it can only give scalar results of MOS scores, without providing specific attribution explanations.

[0012] In summary, existing technologies still have many shortcomings in terms of datasets, dynamic model evaluation capabilities, and attribution explanation capabilities. Currently, no descriptions or reports of technologies similar to this invention have been found, and no similar materials have been collected domestically or internationally. Summary of the Invention

[0013] To address the aforementioned shortcomings in the prior art, this invention provides a multimodal automated evaluation method and system for text-based videos, along with a corresponding evaluation device and computer-readable storage medium.

[0014] According to a first aspect of the present invention, a multimodal automated evaluation method for text-based videos is provided, comprising:

[0015] Construct a Q-Save evaluation dataset, which includes: a Prompt set containing dynamic elements, textual videos, and labeled data, attribution text, and MOS scores for each video;

[0016] An initial model is constructed based on the vision-language large model, and the initial model is trained using the Q-Save evaluation dataset to obtain the evaluation model;

[0017] The video to be predicted is used as input to the evaluation model, and the model output is weighted and averaged to obtain the final prediction score and attribution.

[0018] Preferably, the text-based video includes:

[0019] We ranked the performance of current mainstream open-source and closed-source text-to-video generation models, and selected the top n models as video generation sources to obtain text-to-videos with different generation characteristics and defect types.

[0020] Preferably, the annotation data, attribution text, and MOS score for each video include:

[0021] The dimensions of the labeled data include: visual quality, used to evaluate the perceptual quality of video frames, including detail fidelity and sharpness; motion quality, used to evaluate temporal dynamic characteristics, including motion smoothness, coherence, and physical plausibility; and text-video alignment, used to evaluate the semantic matching degree between the generated content and the input Prompt.

[0022] The video is manually pre-annotated using predefined scoring criteria. Parts with scoring consistency and detail completeness below the set threshold are then annotated again. Finally, the annotation results are sampled to obtain annotated data that meets the scoring criteria.

[0023] The average opinion score of the labeled data is obtained, which is the MOS score;

[0024] Using a piecewise mapping formula, the MOS score is converted into a discrete adjective score, expressed as:

[0025]

[0026] In the formula, , , representing the lowest score and the highest score, respectively. Indicates MOS score;

[0027] When the Mean Opinion Score (MOS) is less than or equal to a set score threshold, the video is determined to be a low-quality video, and the attribution is labeled to obtain the attribution text.

[0028] Preferably, the step of constructing an initial model based on a large vision-language model, and training the initial model using the Q-Save evaluation dataset to obtain an evaluation model, includes:

[0029] Based on the Q-Save evaluation dataset, an attribution supervision pair is constructed, which includes the following data: textual video, Prompt, and attribution text. Based on the pre-trained visual-language large model, the visual encoder and text encoder are frozen, and the attribution supervision pair is used to fine-tune the attention layer and output layer of the decoder so that the attribution text generated by the model is semantically consistent with the human attribution text, thus constructing an attribution pre-trained model.

[0030] Based on the Q-Save evaluation dataset, a hybrid supervision pair is constructed, which includes the following data: textual videos, prompts, MOS scores, and attribution texts; the hybrid supervision pair is used to train the attribution pre-trained model for multiple tasks.

[0031] The attribution pre-trained model is iteratively fine-tuned using a gradient ascent strategy to minimize the total loss function, thereby adjusting the model's scoring of video quality scores and the accuracy of attribution, resulting in an evaluation model.

[0032] Preferably, in attribution pre-training, CE loss is used as the loss function. , is represented as:

[0033]

[0034] In the formula, One-hot encoding for the real labels, where the real category corresponds to =1, all others are 0. To predict the probability of the k-th class for the model, For the current category, This represents the total number of categories;

[0035] In multi-task training, a total loss function is established. for:

[0036]

[0037] In the formula, Cross-entropy loss is used to optimize attribution text generation. The mean squared error loss is used to optimize MOS score prediction.

[0038] in:

[0039] The cross-entropy loss Represented as:

[0040]

[0041] In the formula, One-hot encoding for the real labels, where the real category corresponds to =1, all others are 0. To predict the probability of the k-th class for the model, For the current category, This represents the total number of categories;

[0042] The mean square error loss Represented as:

[0043]

[0044] In the formula, For the predicted value of the sample, This represents the true value of the sample.

[0045] Preferably, the step of using the text-based video to be predicted as input to the evaluation model and performing a weighted average of the model output to obtain the final prediction score and attribution includes:

[0046] The video to be predicted is used as input to the evaluation model to obtain the prediction results output by the decoder, including attribution and scoring.

[0047] The five words Excellent, Good, Fair, Poor, and Bad are selected from all the data output by the decoder. After applying a softmax operation, the probabilities of each of the five words are obtained. Finally, a weighted average is used to obtain the final predicted score. , represented as:

[0048]

[0049] In the formula, The probability of evaluating five adjectives. The weights for each score.

[0050] By summing the weighted averages as described above, we can ensure that the final predicted score is consistent with human subjective perception.

[0051] According to a second aspect of the present invention, a multimodal automated evaluation system for text-based videos is provided, comprising:

[0052] The dataset construction module is used to construct the Q-Save evaluation dataset, which includes: a Prompt set containing dynamic elements, textual videos, and labeled data, attribution text, and MOS scores for each video;

[0053] The evaluation model training module constructs an initial model based on the large vision-language model, and trains the initial model using the Q-Save evaluation dataset to obtain the evaluation model.

[0054] The model inference output module is used to take the text video to be predicted as input to the evaluation model and perform weighted averaging on the model output to obtain the final prediction score and attribution.

[0055] According to a third aspect of the present invention, an evaluation apparatus is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can perform the method described in any one of the above invention.

[0056] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the preceding inventions.

[0057] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0058] Regarding the dataset coverage and quality, as well as the annotation dimensions and information richness:

[0059] The T2VQA dataset does not cover high-specification videos, limiting the model's generalization ability. In contrast, the dataset of this invention contains a large number of videos (e.g., 10,000), which are generated through diverse cue sources and multiple generative models, providing broader coverage. Furthermore, it undergoes rigorous human quality control to ensure the accuracy and reliability of the data.

[0060] The Q-Eval-100K evaluation dimension does not include dynamic quality and only provides scalar MOS scores, without offering specific attribution explanations. This invention's dataset not only labels each video with a scalar mean opinion score (MOS), but also provides fine-grained attribution labels along three core dimensions: visual quality, motion quality, and text-video alignment, providing richer supervision signals for model training.

[0061] Regarding the model's dynamic quality assessment capabilities, as well as its comprehensive assessment and generalization capabilities:

[0062] T2VQA lacks a dedicated timing module, resulting in insufficient accuracy in evaluating dynamic action-based text; the Q-Eval-100K evaluation dimension does not include dynamic quality. This invention can evaluate the dynamic quality of video and performs exceptionally well in this area, outperforming other baseline models in both instance-level and model-level evaluations.

[0063] The unified evaluation model of this invention can simultaneously evaluate visual quality, motion quality, and text-video alignment, achieving excellent results across multiple dimensions and demonstrating good generalization ability in cross-dataset validation. In contrast, the T2VQA model has limited generalization, and Q-Eval-100K performs only moderately well in evaluating videos.

[0064] Regarding attribution explanation ability:

[0065] This invention employs a multi-stage training strategy. First, supervised fine-tuning is performed on a fully annotated attribution dataset. Then, multi-task learning is conducted using a hybrid dataset containing partial attribution annotations and complete MOS labels. This training method improves the model's interpretability and prediction accuracy. T2VQA and Q-Eval-100K, however, do not mention similar effective training strategies.

[0066] The model of this invention can not only provide accurate quality prediction scores, but also generate interpretable, dimension-specific explanations based on three key dimensions, revealing the underlying principles of the evaluation, while Q-Eval-100K cannot provide specific attribution explanations.

[0067] By integrating the advantages of the above aspects, this invention not only ensures the ability to score video quality, but also provides reliable fine-grained attribution capabilities. Attached Figure Description

[0068] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0069] Figure 1 This is a flowchart illustrating the multimodal automated evaluation method for text-based videos in a preferred embodiment of the present invention.

[0070] Figure 2 This is a schematic diagram of the constituent modules of a multimodal automated evaluation system for text-based videos in a preferred embodiment of the present invention.

[0071] Figure 3 This is a schematic diagram of manual annotation in a specific application example of the present invention.

[0072] Figure 4 This is a schematic diagram of the overall structure of the evaluation model in a specific application example of the present invention. Detailed Implementation

[0073] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0074] In existing technologies, commonly used automated evaluation techniques, such as T2VQA and Q-Eval-100K, still have many shortcomings in terms of datasets, dynamic model evaluation capabilities, and attribution explanation capabilities. To address these shortcomings, one embodiment of this invention provides a multimodal automated evaluation method for text-based videos. This method solves the problems of existing technologies in terms of datasets, dynamic model evaluation capabilities, and attribution explanation capabilities. Specifically, regarding datasets, this method addresses the technical problems of limited dataset coverage and weak model generalization ability in existing T2VQA methods. It aims to construct a dataset with broader coverage and higher quality. Through diversified prompt source design and multiple top-level generative models, a large number of video samples are generated, and rigorous manual quality control ensures that the data covers different scenes, entity types, and generation quality. Regarding the model's dynamic evaluation capability (video dynamic scoring capability), this method addresses the technical issues of insufficient dynamic quality evaluation capability or inadequate accuracy in existing technologies. It explicitly includes "motion quality" as one of the three core evaluation dimensions. By labeling the smoothness, coherence, and physical plausibility of motion in the video, combined with a multi-stage training strategy, the model acquires accurate dynamic quality evaluation capabilities, resolving the problem of video quality evaluation failure in dynamic scenes. Regarding attribution explanation capability (video evaluation interpretability), this method solves the technical problem of existing methods only outputting scalar MOS scores and lacking attribution explanations. By labeling each video with fine-grained attribution tags and training a unified evaluation model, it enables the model to generate natural language explanations based on the three dimensions while outputting quality scores, resolving the technical problem of uninterpretable evaluation results. In terms of comprehensive evaluation methods, this approach overcomes the technical problems of unsystematic evaluation dimensions and difficulty in comprehensively measuring video quality in existing technologies. Through theoretical and empirical analysis, it clearly defines three core dimensions: visual quality, motion quality, and text-video alignment, covering the full-dimensional quality characteristics of video in terms of "space-time-semantics". It constructs a systematic and non-redundant evaluation framework, solving the technical problems of fragmented evaluation dimensions and inability to fully reflect the quality of AI-generated videos.

[0075] Specifically, such as Figure 1 As shown, the multimodal automated evaluation method for text-based videos provided in this embodiment may include:

[0076] S1. Construct the Q-Save evaluation dataset, which includes: a Prompt set containing dynamic elements, textual videos, and labeled data, attribution text, and MOS scores for each video;

[0077] S2, an initial model is built based on the large vision-language model, and the initial model is trained using the Q-Save evaluation dataset to obtain the evaluation model;

[0078] S3 takes the video to be predicted as input to the evaluation model and performs a weighted average of the model output to obtain the final prediction score and attribution.

[0079] In some preferred embodiments, the S1 above, which includes a Prompt set containing dynamic elements, may further include:

[0080] Each Prompt includes a description of the dynamic action or camera movement.

[0081] In some preferred embodiments, the above-mentioned S1, the text-based video, may further include:

[0082] We ranked the performance of current mainstream open-source and closed-source text-to-video (T2V) models, and selected the top n models as video generation sources to obtain text-to-videos with different generation characteristics and defect types.

[0083] In some preferred embodiments, S1 above, for each video's labeled data, attribution text, and MOS score, may further include:

[0084] The dimensions of the labeled data include: visual quality, used to evaluate the perceptual quality of video frames, including detail fidelity and sharpness; motion quality, used to evaluate temporal dynamic characteristics, including motion smoothness, coherence, and physical plausibility; and text-video alignment, used to evaluate the semantic matching degree between the generated content and the input Prompt.

[0085] The video is manually pre-annotated using predefined scoring criteria. Parts with scoring consistency and detail completeness below the set threshold are then annotated again. Finally, the annotation results are manually sampled to obtain annotated data that meets the scoring criteria.

[0086] The average opinion score of the labeled data is obtained, which is the MOS score;

[0087] Using a piecewise mapping formula, the MOS score is converted into a discrete adjective score, expressed as:

[0088]

[0089] In the formula, , , representing the lowest score and the highest score, respectively. Indicates MOS score;

[0090] When the Mean Opinion Score (MOS) is less than or equal to a set score threshold, the video is determined to be a low-quality video, and the attribution is labeled to obtain the attribution text.

[0091] In some preferred embodiments, S2 above, which involves constructing an initial model based on a large vision-language model and training the initial model using the Q-Save evaluation dataset to obtain an evaluation model, may further include:

[0092] S21. Based on the Q-Save evaluation dataset, construct attribution supervision pairs, which include the following data: textual videos, prompts, and attribution texts. Based on the pre-trained visual-language large model, freeze the visual encoder and text encoder, and use the attribution supervision pairs to fine-tune the attention layer and output layer of the decoder so that the attribution text generated by the model is semantically consistent with the human attribution text, thus constructing the attribution pre-trained model.

[0093] S22. Based on the Q-Save evaluation dataset, a hybrid supervision pair is constructed, which includes the following data: textual videos, Prompt, MOS scores, and attribution texts. The hybrid supervision pair is then used to train the attribution pre-trained model for multiple tasks.

[0094] S23 employs a gradient ascent strategy to iteratively fine-tune and optimize the attribution pre-trained model to minimize the total loss function, thereby adjusting the model's scoring of video quality (ranging from 1 to 5 points) and the accuracy of attribution, resulting in an evaluation model.

[0095] In some preferred embodiments, in S21 above, CE loss is used as the loss function during attribution pre-training. , is represented as:

[0096]

[0097] In the formula, One-hot encoding for the real labels, where the real category corresponds to =1, all others are 0. To predict the probability of the k-th class for the model, For the current category, This represents the total number of categories.

[0098] In some preferred embodiments, the above-described S22, in multi-task training, the total loss function for:

[0099]

[0100] In the formula, Cross-entropy loss is used to optimize attribution text generation. The mean squared error loss is used to optimize MOS score prediction.

[0101] in:

[0102] Cross-entropy loss Represented as:

[0103]

[0104] In the formula, One-hot encoding for the real labels, where the real category corresponds to =1, all others are 0. To predict the probability of the k-th class for the model, For the current category, The total number of categories; where the actual category corresponds to =1, and all others are 0;

[0105] Mean square error loss Represented as:

[0106]

[0107] In the formula, For the predicted value of the sample, This represents the true value of the sample.

[0108] In some preferred embodiments, S3 above, which uses the textual video to be predicted as input to the evaluation model and performs a weighted average of the model output to obtain the final prediction score and attribution, may further include:

[0109] The video to be predicted is used as input to the evaluation model to obtain the prediction results output by the decoder, including attribution and scoring.

[0110] The five words Excellent, Good, Fair, Poor, and Bad are selected from all the data output by the decoder. After applying a softmax operation, the probabilities of each of the five words are obtained. Finally, a weighted average is used to obtain the final predicted score. , is represented as:

[0111]

[0112] In the formula, The probability of evaluating five adjectives. The weights for each score.

[0113] By summing the weighted averages as described above, and using PLCC and SRCC indicators as consistency criteria, we can ensure that the final predicted score is consistent with human subjective perception.

[0114] Based on the same inventive concept, one embodiment of the present invention also provides a multimodal automated evaluation system for text-based videos.

[0115] Specifically, such as Figure 2As shown, the multimodal automated evaluation system for text-based videos provided in this embodiment may include:

[0116] The dataset construction module is used to build the Q-Save evaluation dataset, which includes: a Prompt set containing dynamic elements, textual videos, and labeled data, attribution text, and MOS scores for each video;

[0117] The evaluation model training module builds an initial model based on the large vision-language model, trains the initial model using the Q-Save evaluation dataset, and obtains the evaluation model.

[0118] The model inference output module takes the text video to be predicted as input to the evaluation model and performs a weighted average of the model output to obtain the final prediction score and attribution.

[0119] The following describes in detail, with reference to preferred embodiments, the specific implementation methods of each functional module constituting the system provided in the above embodiments of the present invention.

[0120] 1. Dataset building module, used to build Q-Save evaluation dataset.

[0121] This module generates a dual-supervised dataset consisting of scalar scoring and fine-grained attribution, namely the Q-Save evaluation dataset. This provides a foundation for subsequent model training and evaluation, addressing the issues of low quality and lack of systematically defined dimensions in existing datasets. Specifically, the Prompt dataset must contain dynamic elements, video generation must cover mainstream AIGV models, and annotations must undergo quality control to ensure reliable test results. Videos with low scores must be annotated with attributions to provide data for model attribution training. The following is a detailed explanation of the module's implementation:

[0122] 1. In the input source design phase, to ensure data diversity and representativeness, the Prompt construction abandons the traditional "existing data sampling" method and adopts a strategy of manual creation and multiple rounds of optimization to design a sizable high-quality Prompt set. Each Prompt contains a description of dynamic actions or camera movements. After manual review and optimization, it ensures the completeness of category coverage and semantic richness, and avoids interference from low-quality and repetitive Prompts. At the same time, the performance of current mainstream open-source and closed-source text-to-video (T2V) models is ranked, and the top models are selected as video generation sources to cover the generation characteristics and defect types of different models.

[0123] 2. In the manual annotation stage, to ensure the accuracy and reliability of annotation, three core evaluation dimensions were first identified based on theoretical and empirical analysis: visual quality (evaluating the perceptual quality of video frames, including detail fidelity, sharpness, and aesthetic appeal), motion quality (evaluating temporal dynamic characteristics, including motion smoothness, coherence, and physical plausibility), and text-video alignment (evaluating the semantic matching degree between the generated content and the input prompt, i.e., generation accuracy). This avoids dimensional redundancy and covers key AIGV quality indicators. Subsequently, a four-step process of "training-pilot-formal annotation-quality review" was adopted to conduct annotation work. First, annotators received unified training on the text-defined scoring criteria. Then, pilot annotation was conducted, and annotators with low scoring consistency and insufficient detail completeness received secondary training. Finally, the annotation results were manually sampled, and non-compliant annotation data was removed. A schematic diagram of manual annotation is shown in a specific application example. Figure 3 As shown in the figure, the process involves three steps: training annotators and aligning them with annotation standards; collecting human annotation scores and attribution data; and processing the human annotation data.

[0124] 3. In the labeled data processing stage, to adapt to the model training requirements, the average opinion score (MOS, 1-5) of the labeled data is converted into discrete adjective scores that are easy for the model to understand, using a segmented mapping formula:

[0125]

[0126] in, , The minimum and maximum scores are represented to address the issue of LLM's ambiguity in understanding numerical scoring. On the other hand, for low-quality videos with MOS≤3, concise diagnostic statements from annotators are first collected, and then the explanations are refined and expanded using Gemini 2.5 Pro to generate long text explanations containing specific defect details and physical and logical deviations. At the same time, explanations with large deviations from human judgment are manually cleaned to form high-quality supervised fine-tuning (SFT) data.

[0127] II. Evaluation Model Training Module, used to train a unified evaluation model.

[0128] This module is used to train a multimodal model that "simultaneously outputs scores and attributions", solving the problem of "separation of scores and explanations" in existing models. In a specific application example, it can be built based on Qwen2.5-VL-7B-Instruct.

[0129] (a) Attribution Pre-training Unit

[0130] In this training section, attribution pre-training is performed to initially align the model with the AIGV attribution annotation data. Supervision pairs of "video + prompt + attribution text" (ideally 40,000 pairs) are selected from the evaluation dataset and divided into attribution training and validation sets in an 8:2 ratio. The pre-trained Qwen2.5-VL-7B-Instruct model is then loaded, the visual encoder and text encoder are frozen, and only the decoder's attention and output layers are fine-tuned to reduce computation. The loss function is CE loss, and the optimization objective is to ensure that the attribution text generated by the model is semantically consistent with the human-generated attribution text.

[0131] (ii) Multi-task fine-tuning unit

[0132] In this training, multi-task training is performed to initially align the model with the AIGV scoring and annotation data. A hybrid supervision pair of "video + Prompt + MOS score + attribution text" is selected from the evaluation dataset. Then, the attribution pre-trained model output from the attribution pre-training is loaded to prepare for multi-task training. The total loss is:

[0133]

[0134] in Used to optimize attribution text generation This module is used to optimize MOS score prediction. The following is a detailed explanation of its implementation:

[0135] 1. Cross-entropy loss ( )

[0136] To help large language models (LMMs) learn the general rules and necessary knowledge of question-answering formats, and to ensure that the model output conforms to the distribution of rating categories annotated by humans, cross-entropy loss is used. Defined as:

[0137]

[0138] Among them, only the true category corresponds to =1, and the rest are 0.

[0139] 2. Mean square error loss ( )

[0140] To more accurately predict video quality scores, we introduce mean squared error loss. The mean squared error loss function... Defined as:

[0141]

[0142] in, These are the predicted values ​​for the sample. It is the true value of the sample.

[0143] (III) Gradient Ascent Optimization Unit

[0144] To optimize the objective function, we employ a gradient ascent strategy to iteratively fine-tune the optimization model to minimize the total loss function, thereby adjusting the accuracy of model scoring and attribution.

[0145] III. Model Inference Output Module

[0146] In this module, five words—Excellent, Good, Fair, Poor, and Bad—are selected from all the data output by the decoder. After a softmax operation, the probabilities of each of the five words are obtained. Then, a weighted average is used to obtain the final predicted score, as shown in the following formula:

[0147]

[0148] in, The probability of evaluating five adjectives. We assign weights to each rating to ensure that the rating results are consistent with human subjective perception.

[0149] In a specific application example, the overall structure diagram of the evaluation model is as follows: Figure 4 As shown, it includes a visual encoder, a text encoder, and a decoder, and performs softmax and weighted summation operations on the output layer.

[0150] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0151] An embodiment of the present invention also provides an evaluation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform any of the methods described in the above embodiments of the present invention.

[0152] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0153] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0154] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0155] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention.

[0156] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0157] The multimodal automated evaluation method and system for text-based videos provided in the above embodiments of the present invention are as follows:

[0158] Overcoming the shortcomings of traditional datasets, this invention provides highly reliable annotation support. The constructed dataset clearly defines three core dimensions: visual quality, motion quality, and text-video alignment, covering key indicators at both the perceptual and semantic levels, avoiding redundancy and omissions. Simultaneously, it annotates a large number of videos (e.g., 10,000 videos) with Mean Opinion Scores (MOS) and natural language attribution explanations, filling the gap in traditional datasets that "only provide scores without explanations," thus laying a highly reliable foundation for model training.

[0159] This invention achieves integrated scoring and attribution, enhancing assessment interpretability. Within a single architecture, it simultaneously predicts MOS scores and generates dimensional attribution explanations. Furthermore, through multi-stage training, the attribution signals in turn improve scoring accuracy, addressing the pain point of uninterpretable assessment results.

[0160] This invention offers strong generalization and adaptability to multiple scenarios, reducing the cost of cross-scenario applications. It utilizes manually designed prompts covering various categories, such as "humans," "animals," and "virtual subjects," and can also employ multiple mainstream AIGV models to generate videos, resulting in text-based videos with different generation characteristics and defect types. These videos include detailed descriptions such as style and camera parameters, ensuring dataset diversity. This allows for adaptation to new scenarios without retraining during third-party dataset validation, thus reducing adaptation costs.

[0161] Empowering industries to reduce costs and increase efficiency, and promoting the reliable implementation of AIGV. This invention automates the entire evaluation process, significantly improving efficiency compared to manual methods, and its consistency with human judgment far exceeds traditional indicators. It can replace manual methods for large-scale quality screening, greatly reducing industry evaluation costs; attribution explanation helps developers accurately locate defective modules in the generated model, accelerating technology iteration.

[0162] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0163] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A multimodal automated evaluation method for text-based videos, characterized in that, include: A Q-Save evaluation dataset is constructed, comprising: a Prompt set containing dynamic elements, textual videos, and labeled data, attribution text, and MOS scores for each video; wherein the labeled data, attribution text, and MOS scores for each video include: The dimensions of the labeled data include: visual quality, used to evaluate the perceptual quality of video frames, including detail fidelity and sharpness; motion quality, used to evaluate temporal dynamic characteristics, including motion smoothness, coherence, and physical plausibility; and text-video alignment, used to evaluate the semantic matching degree between the generated content and the input Prompt. The video is manually pre-annotated using predefined scoring criteria. Parts with scoring consistency and detail completeness below the set threshold are then annotated again. Finally, the annotation results are manually sampled to obtain annotated data that meets the scoring criteria. The average opinion score of the labeled data is obtained, which is the MOS score; Using a piecewise mapping formula, the MOS score is converted into a discrete adjective score, expressed as: ； In the formula, , , representing the lowest score and the highest score, respectively. Indicates MOS score; When the mean opinion score (MOS) is less than or equal to a set score threshold, the video is judged to be a low-quality video, and attribution is assigned, resulting in attribution text containing human interpretation. An initial model is constructed based on the large vision-language model. The initial model is then trained using the Q-Save evaluation dataset to obtain the evaluation model; this includes: Based on the Q-Save evaluation dataset, an attribution supervision pair is constructed, which includes the following data: textual video, Prompt, and attribution text. Based on the pre-trained visual-language large model, the visual encoder and text encoder are frozen, and the attribution supervision pair is used to fine-tune the attention layer and output layer of the decoder so that the attribution text generated by the model is semantically consistent with the human attribution text, thus constructing an attribution pre-trained model. Based on the Q-Save evaluation dataset, a hybrid supervision pair is constructed, which includes the following data: textual videos, prompts, MOS scores, and attribution texts; the hybrid supervision pair is used to train the attribution pre-trained model for multiple tasks. The gradient ascent strategy is used to iteratively fine-tune and optimize the attribution pre-training model to minimize the total loss function, thereby adjusting the model's scoring of video quality scores and the accuracy of attribution, resulting in an evaluation model. The video to be predicted is used as input to the evaluation model, and the model output is weighted and averaged to obtain the final prediction score and attribution.

2. The multimodal automated evaluation method for text-based videos according to claim 1, characterized in that, The Prompt set containing dynamic elements includes: Each Prompt includes a description of the dynamic action or camera movement.

3. The multimodal automated evaluation method for text-based videos according to claim 1, characterized in that, The video includes: We ranked the performance of current mainstream open-source and closed-source text-to-video generation models, and selected the top n models as video generation sources to obtain text-to-videos with different generation characteristics and defect types.

4. The multimodal automated evaluation method for text-based videos according to claim 1, characterized in that, In attribution pre-training, CE loss is used as the loss function. , represented as: ； In the formula, One-hot encoding for the real labels, where the real category corresponds to =1, all others are 0. To predict the probability of the k-th class for the model, For the current category, This represents the total number of categories; In multi-task training, a total loss function is established. for: ； In the formula, Cross-entropy loss is used to optimize attribution text generation. The mean squared error loss is used to optimize MOS score prediction. in: The cross-entropy loss Represented as: ； In the formula, One-hot encoding for the real labels, where the real category corresponds to =1, all others are 0. To predict the probability of the k-th class for the model, For the current category, This represents the total number of categories; The mean square error loss Represented as: ； In the formula, For the predicted value of the sample, This represents the true value of the sample.

5. The multimodal automated evaluation method for text-based videos according to claim 1, characterized in that, The process of using the text-based video to be predicted as input to the evaluation model and performing a weighted average of the model output to obtain the final prediction score and attribution includes: The video to be predicted is used as input to the evaluation model to obtain the prediction results output by the decoder, including attribution and scoring. The five words Excellent, Good, Fair, Poor, and Bad are selected from all the data output by the decoder. After applying a softmax operation, the probabilities of each of the five words are obtained. Finally, a weighted average is used to obtain the final predicted score. , represented as: ； In the formula, The probability of evaluating five adjectives. The weights for each score.

6. A multimodal automated evaluation system for text-based videos, characterized in that, include: A dataset construction module is used to construct the Q-Save evaluation dataset, which includes: a Prompt set containing dynamic elements, textual videos, and labeled data, attribution text, and MOS scores for each video; wherein, the labeled data, attribution text, and MOS scores for each video include: The dimensions of the labeled data include: visual quality, used to evaluate the perceptual quality of video frames, including detail fidelity and sharpness; motion quality, used to evaluate temporal dynamic characteristics, including motion smoothness, coherence, and physical plausibility; and text-video alignment, used to evaluate the semantic matching degree between the generated content and the input Prompt. The video is manually pre-annotated using predefined scoring criteria. Parts with scoring consistency and detail completeness below the set threshold are then annotated again. Finally, the annotation results are manually sampled to obtain annotated data that meets the scoring criteria. The average opinion score of the labeled data is obtained, which is the MOS score; Using a piecewise mapping formula, the MOS score is converted into a discrete adjective score, expressed as: ； In the formula, , , representing the lowest score and the highest score, respectively. Indicates MOS score; When the mean opinion score (MOS) is less than or equal to a set score threshold, the video is judged to be a low-quality video, and attribution is assigned, resulting in attribution text containing human interpretation. The evaluation model training module constructs an initial model based on the large vision-language model, trains the initial model using the Q-Save evaluation dataset, and obtains the evaluation model; it includes: Based on the Q-Save evaluation dataset, an attribution supervision pair is constructed, which includes the following data: textual video, Prompt, and attribution text. Based on the pre-trained visual-language large model, the visual encoder and text encoder are frozen, and the attribution supervision pair is used to fine-tune the attention layer and output layer of the decoder so that the attribution text generated by the model is semantically consistent with the human attribution text, thus constructing an attribution pre-trained model. Based on the Q-Save evaluation dataset, a hybrid supervision pair is constructed, which includes the following data: textual videos, prompts, MOS scores, and attribution texts; the hybrid supervision pair is used to train the attribution pre-trained model for multiple tasks. The gradient ascent strategy is used to iteratively fine-tune and optimize the attribution pre-training model to minimize the total loss function, thereby adjusting the model's scoring of video quality scores and the accuracy of attribution, resulting in an evaluation model. The model inference output module is used to take the text video to be predicted as input to the evaluation model and perform weighted averaging on the model output to obtain the final prediction score and attribution.

7. An evaluation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-5.

Images