Multi-modal model evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides an evaluation method and device of a multi-modal model, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning and the like, and can be applied to scenes of digital people, e-commerce and the like. According to the specific implementation scheme, test video clips are sampled from an original live broadcast video set according to live broadcast e-commerce service distribution information; based on the test video clip, constructing an exclusive structured benchmark of the live broadcast e-commerce scene; the structured benchmark comprises a structured action flow and a time sequence flow; inputting the test video clip into the to-be-tested multi-modal model to obtain a text description result generated by the to-be-tested multi-modal model; performing structured analysis on the text description result to obtain structured text description; and according to the structured benchmark and the structured text description, performing evaluation on the plurality of capability dimensions to obtain an evaluation result of the to-be-tested multi-modal model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision and deep learning, and especially to evaluation methods, devices, electronic devices and storage media for multimodal models, which can be applied to scenarios such as digital humans and e-commerce. Background Technology

[0002] The video understanding evaluation system of multimodal models in related technologies is usually designed for general scenarios, focusing on overall descriptive ability or simple action recognition. This makes it only a coarse-grained scoring method for general tasks. For finer-grained scoring, it usually relies on manual frame-by-frame annotation and manual comparison, which is extremely costly, inefficient and unreproducible, making it difficult to support the high-frequency regression verification needs of e-commerce businesses. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for evaluating multimodal models, which can solve the problem that traditional manual evaluation methods in related technologies cannot meet the needs of e-commerce businesses for fine-grained, reproducible, and highly efficient verification.

[0004] In a first aspect, embodiments of this disclosure provide a method for evaluating a multimodal model, comprising: Based on the distribution information of live-streaming e-commerce businesses, test video clips were sampled from the original live-streaming videos. Based on the test video clips, a structured benchmark specific to the live e-commerce scenario is constructed; the structured benchmark includes structured action flow and time sequence flow; The test video clip is input into the multimodal model under test, and the text description result generated by the multimodal model under test is obtained. The text description result is the test description of the video understanding task. The text description result is parsed in a structured manner to obtain a structured text description; Based on the structured benchmark and the structured text description, evaluations are performed on multiple capability dimensions to obtain the evaluation results of the multimodal model under test.

[0005] Secondly, embodiments of this disclosure provide a multimodal model evaluation apparatus, comprising: The sampling module is used to sample test video segments from the original live video collection based on the distribution information of live e-commerce business. The benchmark construction module is used to construct a structured benchmark specific to the live e-commerce scenario based on the test video clips; the structured benchmark includes a structured action flow and a time sequence flow; The model inference module is used to input the test video clip into the multimodal model under test and obtain the text description result generated by the multimodal model under test. The text description result is the test description of the video understanding task. The structured parsing module is used to perform structured parsing on the text description results to obtain a structured text description; The evaluation module is used to evaluate the multi-capability model under test on multiple capability dimensions based on the structured benchmark and the structured text description, and obtain the evaluation results of the multi-modal model under test.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0007] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in the first aspect above.

[0008] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] According to the technical solution disclosed herein, a multi-dimensional automated evaluation system for multimodal models of e-commerce live streaming videos is provided. The system performs structured evaluation of the model output of the multimodal model through multiple independent capability dimensions, and provides structured evaluation capabilities for fine-grained action chains and product interactions. This replaces the traditional manual evaluation method and meets the e-commerce business's needs for fine-grained, reproducible, and highly efficient verification.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart of the evaluation method for a multimodal model provided in this embodiment of the disclosure; Figure 2 A flowchart of the evaluation method for a multimodal model provided in this embodiment of the disclosure; Figure 3 A flowchart of the evaluation method for a multimodal model provided in this embodiment of the disclosure; Figure 4A flowchart of the evaluation method for a multimodal model provided in this embodiment of the disclosure; Figure 5 Block diagram of the multimodal model evaluation device provided in the embodiments of this disclosure; Figure 6 This is a block diagram of an electronic device used to implement the evaluation method for multimodal models according to embodiments of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] This disclosure relates to the fields of artificial intelligence technology, such as computer vision and large language models.

[0014] Artificial Intelligence (AI) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0015] Large Language Models (LLMs) are deep learning models trained on large amounts of text data that can generate natural language text or understand the meaning of language text. LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0016] Computer vision uses various imaging systems to replace visual organs as the means of input sensing, allowing computers to process and interpret information instead of the brain. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision, just like humans, and to have the ability to autonomously adapt to their environment.

[0017] The model video understanding task refers to using multimodal models to enable computers to "understand" video content like humans do, and to comprehend the events, objects, actions, scenes, and complex relationships between them.

[0018] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0020] It is worth noting that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, this does not mean that the applicant has used or necessarily used such solutions.

[0021] The evaluation method, apparatus, electronic device, and storage medium for multimodal models according to embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0022] It should be noted that the execution subject of the multimodal model evaluation method in this embodiment can be a multimodal model evaluation device, which can be implemented by software and / or hardware, and can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0023] Figure 1 A flowchart illustrating the evaluation method for a multimodal model provided in this disclosure. Figure 1 As shown, the evaluation method for this multimodal model may include, but is not limited to, the following steps.

[0024] In step 101, test video segments are sampled from the original live video collection based on the distribution information of live e-commerce business.

[0025] In some embodiments, the aforementioned live-streaming e-commerce business distribution information may include, but is not limited to, at least one of the following: the proportion of different live-streaming formats; the distribution of product display methods; action density; scene type; and product type distribution. For example, this live-streaming e-commerce business distribution information may include, but is not limited to: the proportion of different live-streaming formats; the distribution of product display methods; action density; scene type; and product type distribution. The different live-streaming formats may include single-host live-streaming, dual-host live-streaming, and host + assistant live-streaming. The product display methods may include, but are not limited to: holding up, displaying in front, rotating, and demonstrating. The action density may include, but is not limited to: high action frequency and stable explanation. The scene type may include, but is not limited to: desktop display, wearable display, and standing explanation. The product type may include, but is not limited to: cosmetics, food, clothing, and small appliances.

[0026] In the embodiments of this disclosure, sampling is performed from the original live video set based on the distribution information of live e-commerce business. For example, sampling can be performed from the original live video set based on the distribution information of live e-commerce business such as the proportion of different live streaming formats, the distribution of product display methods, action density, scene type, and product type to obtain test video segments. This can improve the coverage of the actual behavioral structure of live e-commerce by the test video segments, rather than arbitrary sampling.

[0027] It should be noted that the aforementioned live-streaming e-commerce business distribution information can be obtained through statistics on real live-streaming data. Optionally, in some embodiments, statistics can be performed on the real live-streaming data of the original live-streaming video set. For example, the proportion of different live-streaming formats, the distribution of product display methods, action density, scene types, and product types can be statistically analyzed to obtain live-streaming e-commerce business distribution information. Using this live-streaming e-commerce business distribution information to sample test video segments can ensure that the test set covers the actual behavioral structure of live-streaming e-commerce.

[0028] In step 102, a structured benchmark specific to the live-streaming e-commerce scenario is constructed based on the test video clips.

[0029] In the embodiments of this disclosure, the structured benchmark specific to the live-streaming e-commerce scenario may include a structured action flow and a time-series flow. The structured action flow describes the static logical structure and dependencies between a series of actions or tasks. It does not emphasize temporal sequence but rather the inclusion, hierarchy, and conditional relationships between actions. The time-series flow, based on the structured action flow, describes the chronological order and dynamic arrangement of a series of actions or events along a timeline. It focuses on the start time, end time, duration of each action, and the temporal relationships between them.

[0030] In the embodiments of this disclosure, standardized annotations can be established for test video clips, including but not limited to characters, products, action categories, and the order of action occurrence, to form a structured benchmark specific to the live-streaming e-commerce scenario. For example, standardized annotations can be established for test video clips through manual annotation to obtain a structured benchmark specific to the live-streaming e-commerce scenario. Alternatively, for example, a text description of the test video clip can be obtained first (e.g., obtained through manual understanding of the test video clip), and an annotation tool (such as an annotation model) can be used to annotate the text description to obtain a structured benchmark specific to the live-streaming e-commerce scenario.

[0031] In some embodiments, the structured benchmark can be represented by a triple of “action word + subject + object”, where the subject can be understood as the agent of the action (such as a broadcaster and / or assistant), and the object can be understood as the product object involved.

[0032] In step 103, the test video clip is input into the multimodal model under test to obtain the text description result generated by the multimodal model under test. The text description result is the test description of the video understanding task.

[0033] In embodiments of this disclosure, the multimodal model under test (MMBT) may have video understanding capabilities and be able to perform video understanding tasks. A test video segment can be input into the MMBT, and the MMBT can perform a video understanding task on the test video segment to obtain a text description result generated by the MMBT. This text description result can be the video understanding result of the MMBT on the test video segment.

[0034] In step 104, the text description results are parsed in a structured manner to obtain a structured text description.

[0035] In the embodiments of this disclosure, the text description result can be subjected to structured parsing, sequentially completing action recognition, subject recognition, and semantic normalization processing to obtain the structured text description corresponding to the text description result. In some embodiments, the structured text description can be represented by a triple of "action word + subject + object", where the subject can be understood as the action performer (such as a broadcaster and / or assistant), and the object can be understood as the product object involved.

[0036] In step 105, the multi-capability model is evaluated on multiple dimensions based on the structured benchmark and structured text description to obtain the evaluation results of the multi-modal model under test.

[0037] In some embodiments, the aforementioned multiple capability dimensions may include, but are not limited to, at least two: static understanding, action behavior, temporal consistency, text quality, and hallucination severity. Static understanding refers to the model's ability to understand the visual content of a single frame or keyframe in a video, ignoring dynamic changes over time. For example, static understanding is responsible for understanding whether there is a person or a table in a single frame. Dynamic behavior refers to recognizing action types; for example, if the action in the video is raising a hand, but the model recognizes a waving hand, then the action recognition is inconsistent. Temporal consistency means that both the action type recognition and the temporal sequence must be consistent. Hallucination severity refers to the degree of deviation between the description, answer, or reasoning generated by the model and the actual content of the video, and the potential harm that such deviation may cause. It should be noted that in some embodiments, terms such as "static understanding" and "static image understanding" can be used interchangeably.

[0038] In the embodiments of this disclosure, scores can be calculated on the above-mentioned multiple capability dimensions based on structured benchmarks and structured text descriptions to obtain a score for each capability dimension. The multiple scores corresponding to the multiple capability dimensions are mapped to a unified dimension, and the multiple scores obtained after mapping are fused to obtain the evaluation result of the multimodal model under test.

[0039] For example, taking the aforementioned multiple capability dimensions, including static understanding, action behavior, temporal consistency, text quality, and hallucination severity, scores can be calculated for each of the five capability dimensions—static understanding, action behavior, temporal consistency, text quality, and hallucination severity—based on a structured benchmark and structured text description. The scores for each capability dimension are then mapped to a unified dimension, and the mapped scores are fused to obtain the evaluation result of the multimodal model under test. Therefore, this disclosure proposes a multi-dimensional automated evaluation system that can achieve structured and independent quantitative scoring of static image understanding, action behavior understanding, temporal consistency, text quality, and hallucination severity in live-streaming e-commerce scenarios. This effectively replaces manual evaluation and meets the needs of fine-grained, reproducible, and highly efficient model verification in live-streaming e-commerce scenarios.

[0040] In the above embodiments, this disclosure provides a multi-dimensional automated evaluation system for multimodal models of e-commerce live streaming videos. It performs structured evaluation of the model output of multimodal models through multiple independent capability dimensions, and provides structured evaluation capabilities for fine-grained action chains and product interactions, thereby replacing traditional manual evaluation methods and meeting the e-commerce business's needs for fine-grained, reproducible, and highly efficient verification.

[0041] Figure 2 A flowchart illustrating the evaluation method for a multimodal model provided in this disclosure. Figure 2 As shown, the evaluation method for this multimodal model may include, but is not limited to, the following steps.

[0042] In step 201, test video segments are sampled from the original live video collection based on the distribution information of live e-commerce business.

[0043] In some embodiments, statistical analysis can be performed on the actual live streaming data of the original live streaming video set to obtain live e-commerce business distribution information. This live e-commerce business distribution information may include, but is not limited to, at least one of the following: the proportion of different live streaming formats; the distribution of product display methods; action density; scene type; and product type distribution.

[0044] For example, the actual live streaming data of the original live streaming video set can be statistically analyzed to determine the proportion of different live streaming formats, the distribution of product display methods, the density of actions, the distribution of scene types, and the distribution of product types. This data can serve as distribution information for live streaming e-commerce businesses, making it easier to extract test video clips from the original live streaming video set based on this distribution information.

[0045] Optionally, step 201 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0046] In step 202, a structured benchmark specific to the live-streaming e-commerce scenario is constructed based on the test video clips.

[0047] Optionally, step 202 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0048] In step 203, the test video clip is input into the multimodal model under test to obtain the text description result generated by the multimodal model under test. The text description result is the test description of the video understanding task.

[0049] Optionally, step 203 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0050] In step 204, the text description results generated by the multimodal model under test are processed through a unified inference interface.

[0051] In the embodiments of this disclosure, the aforementioned unified inference interface can be applied to visual language or video multimodal models of different scales and architectures.

[0052] For example, visual language or video multimodal models of different sizes and architectures can be uniformly loaded and inferred. The text description results generated by the multimodal model under test can be processed through a unified inference interface to output formatted text results, which can solve the problems of inconsistent model evaluation processes and poor scalability in related technologies.

[0053] In step 205, the text description results are parsed in a structured manner to obtain a structured text description.

[0054] Optionally, step 205 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0055] In step 206, the multi-capability model is evaluated on multiple dimensions based on the structured benchmark and structured text description to obtain the evaluation results of the multi-modal model under test.

[0056] Optionally, step 206 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0057] In the above embodiments, visual language or video multimodal models of different scales and architectures can be uniformly loaded and inferred. The text description results generated by the multimodal model under test are processed through a unified inference interface, outputting formatted text results. This solves the problems of inconsistent model evaluation processes and poor scalability in related technologies. The unified inference interface can format the text description results generated by the multimodal model under test, and unified evaluation rules ensure that self-developed models and various open-source models are compared under the same standards, improving the comparability between different models and selecting the most suitable model for data generation in live-streaming e-commerce scenarios.

[0058] Figure 3 A flowchart illustrating the evaluation method for a multimodal model provided in this disclosure. Figure 3 As shown, the evaluation method for this multimodal model may include, but is not limited to, the following steps.

[0059] In step 301, test video segments are sampled from the original live video collection based on the distribution information of live e-commerce business.

[0060] Optionally, step 301 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0061] In step 302, a structured benchmark specific to the live-streaming e-commerce scenario is constructed based on the test video clips.

[0062] Optionally, step 302 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0063] In step 303, the test video clip is input into the multimodal model under test to obtain the text description result generated by the multimodal model under test. The text description result is the test description of the video understanding task.

[0064] Optionally, step 303 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0065] In step 304, verb phrases, action actors, and related product objects appearing in the text description results are identified by combining rule templates with language models.

[0066] In embodiments of this disclosure, the rule templates described above may be pre-configured. For example, the rule templates may be regular expressions or templates based on syntax rules, which define the patterns of verb phrases, the identifiers of agents, and the identifiers of product objects.

[0067] In the embodiments of this disclosure, for text description results, rule templates can be used for quick matching first, and then the model can be used to verify and correct the rule matching results; or, the model can be used directly for annotation, and then rule templates can be used for post-processing.

[0068] For example, rule templates can be used for preliminary matching to identify verb phrases, action agents, and related product objects in the text description results, thus obtaining preliminary annotation results. For instance, verb phrases may consist of verbs and nouns, the agent may be determined by specific vocabulary (such as "anchor" or "assistant") or context, and the product object may be represented by a specific noun or noun phrase. Using a pre-trained small language model (such as a sequence labeling model) to perform sequence labeling on the same text description results, annotating the verb phrases, agents, and product objects, yields the model annotation results. For example, BERT can be used for Named Entity Recognition (NER) tasks, labeling words in the text as B-VP, I-VP, B-AGENT, I-AGENT, B-OBJECT, I-OBJECT, etc. The two results mentioned above (i.e., the preliminary annotation results obtained based on the rule template and the annotation results obtained based on the model recognition) can be fused. For example, for each label, if the rules and the model agree, it can be retained; if they conflict, the model can be prioritized; or a confidence mechanism can be designed to retain the model's annotation results when the confidence of the model's annotation results is greater than or equal to a threshold, and retain the annotation results obtained from the rule template when the confidence of the model's annotation results is less than the threshold.

[0069] In step 305, the verb phrases appearing in the identified text description results are normalized based on the standard action category.

[0070] In the embodiments of this disclosure, after identifying the verb phrases, action actors, and related product objects appearing in the text description results, "action word + subject + object" can be parsed into a triple representation. Based on the triple, the diverse expressions of actions of the multimodal model to be tested can be normalized into standard action categories based on standard action categories, thus obtaining the action-normalized triple.

[0071] In some embodiments, the aforementioned standard action categories can be obtained by: constructing a set of standard actions covering common behaviors in live-streaming e-commerce; and combining the standard action set with semantic clustering and / or a mapping table to normalize the diverse expressions of actions in text descriptions generated by different multimodal models into standard action categories. For example, a dictionary can be pre-constructed to map common expressions to standard action categories, resulting in the mapping table. This mapping table, combined with the set of standard actions covering common behaviors in live-streaming e-commerce, normalizes the diverse expressions of actions in text descriptions generated by different multimodal models into standard action categories. If the mapping table cannot cover all expressions, or if it is necessary to automatically discover new expressions, semantic clustering can be used to map both the standard action categories and the action text descriptions to a semantic space (e.g., using sentence embedding). Then, the semantic similarity between the action text description and each standard action category is calculated, and the most similar standard action category is selected. This normalizes the diverse expressions of actions in text descriptions generated by different multimodal models into standard action categories, facilitating unified evaluation rules and further improving the comparability of evaluation results between different models.

[0072] In step 306, a structured text description is constructed for the verb phrases, the agent of the action, and the product object after the expression normalization process, in the form of triplets of action word, subject, and object.

[0073] In step 307, the multi-modal model under test is evaluated on multiple capability dimensions based on the structured benchmark and structured text description to obtain the evaluation results.

[0074] Optionally, step 307 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0075] In the above embodiments, by standardizing the diverse expressions of actions in the text descriptions generated by different multimodal models into a standard action category, it is easier to unify the evaluation rules and further improve the comparability of evaluation results between different models. That is, after evaluating multiple models through the same process, a clear and comparable ranking of model capabilities can be obtained, which can be used for model selection, updates, or quality assessment before going live.

[0076] Figure 4 A flowchart illustrating the evaluation method for a multimodal model provided in this disclosure. Figure 4 As shown, the above-mentioned evaluation of the multimodal model under test based on structured benchmarks and structured text descriptions, which evaluates the model on multiple capability dimensions respectively, may include, but is not limited to, the following steps.

[0077] In step 401, based on the structured benchmark and structured text description, an evaluation is conducted on the corresponding capability dimension using a referee model and / or preset evaluation rules that match each capability dimension, and a score is obtained for each capability dimension.

[0078] In the embodiments of this disclosure, the aforementioned referee model may include, but is not limited to, a general multimodal large model, and may also include an expert small model. For example, for each capability dimension, the following three types of scoring methods can be used individually or in combination to ensure the system's flexibility and scalability.

[0079] (1) Multimodal large model scoring (e.g., LLM-as-a-Judge) This approach utilizes general-purpose multimodal large models (such as the GPT series and Qwen series) as automatic evaluators. It judges the output of the multimodal model under test based on preset prompts, such as determining whether a non-existent product is mentioned or whether the action description matches a standard sequence. This method is suitable for semantic judgment tasks and has the advantages of strong understanding and high adaptability.

[0080] (2) Expert small model scoring (such as lightweight discriminator) For dimensions such as language expression quality, grammatical structure, and redundancy, small language models or lightweight expert models can be used for scoring, serving as independent quality discriminators. Examples include scoring models for text quality dimensions and micro-classifiers for lightweight verb classification. These models offer stable outputs and low cost, making them suitable as the primary scorer in the system.

[0081] (3) Rule-based scoring system (e.g., rule-based) For tasks that can be compared in a structured way (such as consistency in the number of people, matching of action sets, and differences in temporal sequences), rule-based algorithms can be used to achieve accurate judgments; for example, static noun comparison, normalized action set matching, and editing distance temporal alignment scoring. This type of scoring method is highly reproducible and is the core scoring method for structured dimensions.

[0082] In the embodiments of this disclosure, a referee model and / or preset evaluation rules matching each capability dimension can be adopted based on the structured benchmark and structured text description. For example, for each capability dimension, the three scoring methods mentioned above, namely multimodal large model scoring, expert small model scoring, and rule system scoring, can be used individually or in combination to evaluate the corresponding capability dimension and obtain the score for each capability dimension.

[0083] To further improve the accuracy of the evaluation results, optionally, in some embodiments, temporal extraction can be performed on the structured benchmark and the structured text description to obtain the benchmark action sequence corresponding to the structured benchmark and the test action sequence corresponding to the structured text description; wherein, the benchmark action sequence and the test action sequence can be used to calculate the score on at least one of multiple ability dimensions. For example, the benchmark action sequence and the test action sequence can be used to calculate the score on the action behavior ability dimension, the score on the temporal consistency ability dimension, the score on the hallucination severity ability dimension, the score on the static comprehension ability dimension, and the score on the text quality ability dimension, etc.

[0084] In one possible implementation, explicit temporal indicators, linguistic positional relationships, and syntactic dependencies can be used to extract the sequential and repetitive relationships between actions in a structured benchmark or structured text description, constructing a benchmark action sequence corresponding to the structured benchmark or a test action sequence corresponding to the structured text description. In other words, a structured parsing method can be used to extract the temporal sequence from the structured benchmark or structured text description. This involves using explicit temporal indicators (such as "first," "next," "then," "again," "simultaneously," etc.), linguistic positional relationships, and syntactic dependencies to extract the sequential and repetitive relationships between actions in the structured benchmark or structured text description, constructing a benchmark action sequence corresponding to the structured benchmark or a test action sequence corresponding to the structured text description. For example, sequence alignment algorithms (such as dynamic programming, edit distance, or local sequence matching) can be used to match the benchmark action sequence corresponding to the structured benchmark with the test action sequence corresponding to the structured text description, quantifying issues such as reversed order, missing steps, and repeated actions, and outputting a temporal consistency score. This method has a high degree of structure and strong reproducibility, making it suitable for live-streaming e-commerce scenarios with strict requirements for behavioral processes.

[0085] In one possible implementation, a language model combined with prompt words can be used to extract the temporal sequence of a structured benchmark or structured text description, yielding the benchmark action sequence corresponding to the structured benchmark or the test action sequence corresponding to the structured text description. In other words, a semi-structured approach combining a language model with prompt words (such as a large model + Prompt Engineering) can be used to extract the temporal sequence of a structured benchmark or structured text description. For example, a large language model combined with dedicated prompt words can be used to allow the large language model to directly understand the temporal sequence of the structured benchmark or structured text description, obtaining the benchmark action sequence corresponding to the structured benchmark or the test action sequence corresponding to the structured text description. For instance, a Prompt can be used to request the large language model to output a standard-format action chain, such as "[Action 1 → Action 2 → Action 3]", or to request the large language model to determine whether the action chain corresponding to the structured benchmark and the action chain corresponding to the structured text description are consistent. This approach is simple to implement and has strong generalization ability for complex sentence structures, but its output stability is greatly affected by the Prompt, model version, and random sampling, and it is difficult to guarantee complete reproducibility. Therefore, this approach is suitable as a supplementary method or an alternative implementation in resource-constrained scenarios.

[0086] In step 402, the scores of each of the multiple capability dimensions are mapped to a unified dimension, and the multiple scores obtained after mapping are fused to obtain the evaluation results of the multimodal model under test.

[0087] For example, the scores of multiple capability dimensions are mapped to a unified dimension, and the resulting scores are summed. The final value is used as the evaluation result of the multimodal model under test. This evaluation result can be used for model iteration comparison or as a reward signal during the training phase. For instance, during the training phase of the multimodal model under test, the evaluation result of the e-commerce live streaming video understanding capability of the multimodal model under test can be obtained through the evaluation method provided in this disclosure. This evaluation result can be used for model iteration comparison or as a reward signal during the training phase, thereby guiding the model to learn in the right direction through objective comparison and ensuring target alignment; and accelerating the effective evolution of model capabilities through continuous feedback, thus improving training efficiency.

[0088] In the above embodiments, this disclosure does not rely on a specific model architecture and can be adapted to any multimodal model; the evaluation system is an external evaluation method in the model testing and application stage, which does not change the model training method or affect the internal structure of the model. This disclosure effectively improves the automation and fine-grainedness of the evaluation, reduces the cost of manual annotation, and significantly improves the stability and reproducibility of the evaluation through techniques such as structured action flow extraction, normalization, and temporal alignment.

[0089] Figure 5A block diagram of a multimodal model evaluation device provided in this disclosure. (See also...) Figure 5 As shown, the evaluation device for the multimodal model may include a sampling module 501, a benchmark construction module 502, a model inference module 503, a structured parsing module 504, and an evaluation module 505.

[0090] Among them, the sampling module 501 is used to sample test video segments from the original live video collection based on the distribution information of live e-commerce business.

[0091] The benchmark building module 502 is used to build a structured benchmark specific to the live e-commerce scenario based on test video clips; the structured benchmark includes structured action flow and time sequence flow.

[0092] The model inference module 503 is used to input test video clips into the multimodal model under test and obtain the text description results generated by the multimodal model under test. The text description results are the test descriptions for the video understanding task.

[0093] The structured parsing module 504 is used to perform structured parsing on the text description results to obtain a structured text description.

[0094] Evaluation module 505 is used to evaluate the multi-capability dimensions based on structured benchmarks and structured text descriptions to obtain the evaluation results of the multi-modal model under test.

[0095] In some embodiments, the model inference module 503 is further configured to: process the text description results generated by the multimodal model under test through a unified inference interface; wherein the unified inference interface is applicable to visual language or video multimodal models of different sizes and architectures.

[0096] In some embodiments, the structured parsing module 504 is used to: identify verb phrases, action actors, and related product objects appearing in the text description results by combining rule templates with language models; perform expression normalization processing on the verb phrases appearing in the identified text description results based on standard action categories; and construct a structured text description for the expression-normalized verb phrases, action actors, and product objects in the form of triplets of action words, subjects, and objects.

[0097] In some embodiments, the above-mentioned standard action categories are obtained by: constructing a standard action set covering common behaviors in live e-commerce; and combining the standard action set with semantic clustering and / or mapping tables to normalize the diverse expressions of actions in text descriptions generated by different multimodal models into standard action categories.

[0098] In some embodiments, the aforementioned multiple capability dimensions include at least two of the following: static understanding, action behavior, temporal consistency, text quality, and hallucination severity. In embodiments of this disclosure, the evaluation module 505 is used to: evaluate the corresponding capability dimensions based on a structured benchmark and structured text description, using a referee model matched to each capability dimension and / or preset evaluation rules, to obtain a score for each capability dimension; map the scores of multiple capability dimensions to a unified dimension, and fuse the mapped scores to obtain the evaluation result of the multimodal model under test.

[0099] In some embodiments, the evaluation module 505 is further configured to: perform temporal extraction on the structured benchmark and the structured text description to obtain the benchmark action sequence corresponding to the structured benchmark and the test action sequence corresponding to the structured text description; wherein the benchmark action sequence and the test action sequence are used to calculate the score on at least one of the multiple capability dimensions.

[0100] In some embodiments, the evaluation module is used to: extract the order and repetition relationships between actions in a structured benchmark or structured text description by using explicit temporal indicators, linguistic positional relationships, and syntactic dependencies, and construct a benchmark action sequence corresponding to the structured benchmark or a test action sequence corresponding to the structured text description; or, use a language model combined with prompt words to perform temporal extraction on the structured benchmark or structured text description to obtain a benchmark action sequence corresponding to the structured benchmark or a test action sequence corresponding to the structured text description.

[0101] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0102] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0103] like Figure 6 The diagram shown is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0104] like Figure 6As shown, the electronic device includes one or more processors 601, a memory 602, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take the 601 processor as an example.

[0105] The memory 602 is the non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the multimodal model evaluation method provided in this disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the multimodal model evaluation method provided in this disclosure.

[0106] Memory 602, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal model evaluation method in the embodiments of this disclosure (e.g., appendix). Figure 5 The sampling module 501, benchmark construction module 502, model inference module 503, structured parsing module 504, and evaluation module 505 are shown. The processor 601 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 602, thereby implementing the multimodal model evaluation method in the above method embodiments.

[0107] Memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, memory 602 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 602 may optionally include memory remotely located relative to processor 601, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The electronic device may also include an input device 603 and an output device 604. The processor 601, memory 602, input device 603, and output device 604 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0109] Input device 603 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 604 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0110] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0114] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0115] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for evaluating a multimodal model, comprising: Based on the distribution information of live-streaming e-commerce businesses, test video clips were sampled from the original live-streaming videos. Based on the test video clips, a structured benchmark specific to the live e-commerce scenario is constructed; The structured benchmark includes structured action flow and timing flow; The test video clip is input into the multimodal model under test, and the text description result generated by the multimodal model under test is obtained. The text description result is the test description of the video understanding task. The text description result is parsed in a structured manner to obtain a structured text description; Based on the structured benchmark and the structured text description, evaluations are performed on multiple capability dimensions to obtain the evaluation results of the multimodal model under test.

2. The method according to claim 1, further comprising: The text description results generated by the multimodal model under test are processed through a unified inference interface; wherein, the unified inference interface is applicable to visual language or video multimodal models of different sizes and architectures.

3. The method according to claim 1, wherein, The step of performing structured parsing on the text description result to obtain a structured text description includes: The verb phrases, action actors, and related product objects appearing in the text description results are identified by combining rule templates with language models. The verb phrases appearing in the identified text description results are normalized based on the standard action categories; The structured text description is constructed by using a triplet of action word, subject, and object, which consists of the verb phrase after expression normalization, the agent of the action, and the product object.

4. The method according to claim 3, wherein, The standard action categories are obtained in the following ways: Develop a set of standard actions covering common behaviors in live-streaming e-commerce; By combining semantic clustering and / or mapping tables with the standard action set, the diverse expressions of actions in text descriptions generated by different multimodal models are normalized into the standard action category.

5. The method according to claim 1, wherein, The multiple capability dimensions include at least two of the following: static understanding, action behavior, temporal consistency, text quality, and hallucination severity; The evaluation of the multimodal model under test, based on the structured benchmark and the structured text description, across multiple capability dimensions, to obtain the evaluation results, includes: Based on the structured benchmark and the structured text description, an evaluation is conducted on the corresponding capability dimension using a referee model and / or preset evaluation rules that match each capability dimension, and a score is obtained for each capability dimension. The scores of each of the multiple capability dimensions are mapped to a unified dimension, and the multiple scores obtained after mapping are fused to obtain the evaluation result of the multimodal model under test.

6. The method according to claim 5, further comprising: Temporal extraction is performed on the structured benchmark and the structured text description to obtain the benchmark action sequence corresponding to the structured benchmark and the test action sequence corresponding to the structured text description; wherein, the benchmark action sequence and the test action sequence are used to calculate the score on at least one of the multiple capability dimensions.

7. The method according to claim 6, wherein, Temporal extraction is performed on the structured benchmark or the structured text description to obtain the benchmark action sequence corresponding to the structured benchmark or the test action sequence corresponding to the structured text description, including: By utilizing explicit temporal indicators, linguistic positional relationships, and syntactic dependencies, the sequential and repetitive relationships between actions in the structured benchmark or the structured text description are extracted to construct the benchmark action sequence corresponding to the structured benchmark or the test action sequence corresponding to the structured text description; or, Using a language model combined with prompt words, temporal extraction is performed on the structured benchmark or the structured text description to obtain the benchmark action sequence corresponding to the structured benchmark or the test action sequence corresponding to the structured text description.

8. The method according to any one of claims 1-7, further comprising: The distribution information of the live e-commerce business is obtained by statistically analyzing the real live data of the original live video set; wherein the distribution information of the live e-commerce business includes at least one of the following: the proportion of different live streaming formats, the distribution of product display methods, the density of actions, the distribution of scene types, and the distribution of product types.

9. A multimodal model evaluation device, comprising: The sampling module is used to sample test video segments from the original live video collection based on the distribution information of live e-commerce business. The benchmark construction module is used to construct a structured benchmark specific to the live e-commerce scenario based on the test video clips. The structured benchmark includes structured action flow and timing flow; The model inference module is used to input the test video clip into the multimodal model under test and obtain the text description result generated by the multimodal model under test. The text description result is the test description of the video understanding task. The structured parsing module is used to perform structured parsing on the text description results to obtain a structured text description; The evaluation module is used to evaluate the multi-capability model under test on multiple capability dimensions based on the structured benchmark and the structured text description, and obtain the evaluation results of the multi-modal model under test.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images