Method, device, system and storage medium for multimodal large model evaluation

Abstract

The application relates to the technical field of artificial intelligence, and discloses a method, device and system for multi-modal large model evaluation and a storage medium. The method comprises the following steps: analyzing user input evaluation configuration information to obtain a current structured evaluation scheme, wherein the configuration information comprises one or more of to-be-evaluated multi-modal large model information, an evaluation field, an evaluation dimension, a level parameter, an evaluation index and a reward model PRM enabling strategy; based on a constructed vector database, for each evaluation dimension in the current structured evaluation scheme, performing level evaluation on the to-be-evaluated multi-modal large model from low to high according to a preset difficulty level; and aggregating evaluation results of all evaluation dimensions, performing result analysis and generating a corresponding current evaluation report. In this way, the multi-modal large model evaluation efficiency is improved, and the evaluation accuracy is also improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, such as methods, apparatus, systems, and storage media for evaluating multimodal large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models that can process various modal data such as text, images, and charts have been widely used in scientific research, education evaluation, and industrial applications. Their performance directly determines the application effect. For example, before scientific large models can be applied to scientific research, education, industry, medicine, and engineering, it is necessary to know whether they are "capable, strong, and stable". Therefore, accurate and efficient evaluation of multimodal large models has become an urgent technical problem to be solved.

[0003] Currently, various open-source evaluation datasets and tools are emerging, and the evaluation system is showing a diversified development trend. However, it is still in a rapid evolution stage and has not yet formed a unified, efficient, and flexibly configurable evaluation framework. There are still many problems to be solved in the evaluation technology for scientific multimodal large models, including: inconsistent evaluation data and indicators, low efficiency of fixed-number evaluation, lack of user control over evaluation configuration, and rigid application of process supervision, etc. In summary, none of the existing technologies provide a scientific multimodal large model evaluation scheme that can simultaneously achieve unification, efficiency, and configurability, and specifically address the aforementioned technical pain points. Consequently, they cannot achieve objective evaluation of model capabilities and effective organization of the evaluation process.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method, apparatus, system, and storage medium for evaluating multimodal large models, addressing the technical problem that the adaptability and accuracy of multimodal large model evaluation still need improvement.

[0007] In some embodiments, the method includes: The evaluation configuration information input by the user is parsed to obtain the current structured evaluation scheme. The configuration information includes one or more of the following: the information of the multimodal large model to be tested, the evaluation domain, the evaluation dimension, the level parameter, the evaluation index, and the reward model PRM activation strategy. Based on the constructed vector database, for each evaluation dimension in the current structured evaluation scheme, the execution level is set from low to high according to the preset difficulty level to perform a climbing evaluation of the multimodal large model to be tested. The vector database stores multimodal question semantic vectors, question original information and multi-level indexes. The multi-level indexes include domain index, dimension index, difficulty level index and step annotation availability index. The evaluation results of all evaluation dimensions are summarized, the results are analyzed, and the corresponding current evaluation report is generated. The evaluation results include: the process score obtained by the PRM model when the corresponding PRM model is determined to be enabled based on the PRM activation strategy and the availability of question step annotation.

[0008] In some embodiments, it also includes: A vector database is constructed based on multimodal evaluation data samples; Deploy a reward model PRM library and an evaluation metric library, wherein the evaluation metric library includes exact matching rules, semantic similarity matching rules, and one or more scoring rules from some credit mechanisms; After linking and indexing the vector database, PRM database, and evaluation index database to obtain a multidimensional knowledge base, the multidimensional knowledge base is loaded into a hybrid storage architecture of vector database and relational database.

[0009] In some embodiments, the construction of the vector database includes: Obtain multimodal evaluation data samples and label each data sample. The labeling includes one or more of the following: step labeling, domain labeling, test dimension labeling, and difficulty level labeling. The multimodal evaluation data samples with completed labeling are semantically vectorized to obtain multimodal question semantic vectors; A multi-level index is constructed based on the annotation results, and the evaluation data, annotation results, source metadata, and semantic vectors are loaded into the vector database.

[0010] In some embodiments, the hill-climbing evaluation of the large multimodal model under test includes: Based on the current evaluation domain, dimension, difficulty level, and availability of step annotations, a preset number of multimodal evaluation questions are retrieved from the vector database, and the large multimodal model to be tested is called to obtain the corresponding current model output results information; Based on the conditionally enabled PRM model, the current comprehensive result score of the current model output information is obtained, and the corresponding evaluation decision is executed according to the current comprehensive result score. The evaluation decision includes: advancing to the next difficulty level, continuing the current level test, or terminating the current dimension evaluation.

[0011] In some embodiments, the current comprehensive result score for obtaining the current model output result information includes: If the corresponding PRM model is activated based on the PRM activation strategy and the availability of question step annotations, the process score obtained by the PRM model is evaluated on the output result information of the current model. Based on the scoring rules in the evaluation index library, the output result information of the current model is evaluated to obtain the result score. In addition, the process score and the result score are integrated and calculated according to the preset weight allocation ratio to obtain the current comprehensive result score. If the corresponding PRM model is not enabled based on the PRM activation strategy and the availability of question step annotations, the output result information of the current model is evaluated based on the scoring rules in the evaluation index library to obtain the result score, and the result score is determined as the current comprehensive result score.

[0012] In some embodiments, the execution of the corresponding evaluation decision includes: If the number of test questions at the current level reaches the minimum number of test questions, and the number of consecutive correct answers corresponding to the current comprehensive result score reaches the advancement threshold, the decision is made to advance to the next difficulty level. If the number of consecutive incorrect answers corresponding to the current comprehensive result score reaches the termination threshold, the decision is made to terminate the level climbing assessment for the current dimension. If the maximum number of test questions for the current level has not been reached, and if the number of consecutive correct answers corresponding to the current overall score has not reached the advancement threshold, and if the number of consecutive incorrect answers corresponding to the current overall score has not reached the termination threshold, then the decision is made to continue the current level test. If the maximum number of test questions for the current level has been reached, and the number of consecutive errors corresponding to the current comprehensive score has not reached the termination threshold, the comprehensive score for the current level is recorded; and if the current level is not the highest difficulty level, a decision is made to advance to the next difficulty level, and if the current level is the highest difficulty level, a decision is made to terminate the level climbing assessment for the current dimension.

[0013] In some embodiments, generating the corresponding current evaluation report includes: Based on the evaluation results of all evaluation dimensions, the results are analyzed to obtain the corresponding current capability analysis results. The evaluation results include: the comprehensive score of each level, PRM activation records and evaluation decisions. The capability analysis results include: the capability strengths, weaknesses and improvement suggestions of the multimodal large model under test in each evaluation domain, dimension and difficulty level. Based on the current structured evaluation scheme, evaluation results, and current energy analysis results, a corresponding current evaluation report is generated.

[0014] In some embodiments, the apparatus for evaluating a multimodal large model includes a processor and a memory storing program instructions, the processor being configured to execute the above-described method for evaluating a multimodal large model when the program instructions are executed.

[0015] In some embodiments, the system for evaluating multimodal large models includes: a device body; the aforementioned apparatus for evaluating multimodal large models is installed on the device body.

[0016] In some embodiments, the storage medium stores program instructions that, when executed, perform the method described above for multimodal large model evaluation.

[0017] The method, apparatus, system, and storage medium for multimodal large model evaluation provided in this disclosure can achieve the following technical effects: The system parses the user-inputted evaluation configuration information to obtain the current structured evaluation scheme. This configuration information includes one or more aspects such as the multimodal large-scale model to be tested and the evaluation domain. Based on a vector database storing multimodal question semantic vectors, original question information, and multi-level indexes, it performs progressive evaluation for each evaluation dimension, from low to high difficulty level. The evaluation results for all dimensions are then aggregated, analyzed, and a current evaluation report is generated. This achieves automated startup and full-process control of scientific multimodal large-scale model evaluation. The structured evaluation scheme supports flexible configuration of evaluation parameters, addressing the lack of user control over existing evaluation configurations. Furthermore, it leverages... The multi-level indexing of the quantitative database enables rapid retrieval of questions adapted to current scientific fields, assessment dimensions, and question characteristics, providing support for unified assessment data and indicators and improving the current fragmented assessment data situation. Through graded climbing assessment, the assessment difficulty can be gradually increased according to the actual capabilities of the scientific multimodal large model, avoiding the waste of resources caused by fixed-number assessment and solving the problem of low efficiency in fixed-number assessment. By summarizing assessment results including process scores and generating reports, the assessment results are comprehensively presented, while providing data support for the reasonable application of PRM, improving the problems of rigid PRM application and low credibility of assessment results, and providing accurate data support for the capability analysis of scientific multimodal large models.

[0018] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of the structure of an intelligent agent for evaluating multimodal large models provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating a method for evaluating multimodal large models provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of a process for constructing a knowledge base provided in an embodiment of this disclosure; Figure 4 This is a flowchart illustrating a method for evaluating multimodal large models provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of a multimodal large model evaluation device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a multimodal large model evaluation device provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of a device body provided in an embodiment of this disclosure. Detailed Implementation

[0020] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0021] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0022] Unless otherwise stated, the term "multiple" means two or more.

[0023] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0024] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0025] In this embodiment, during the initial deployment phase, a vector database, a Process Reward Model (PRM) library, and an evaluation index library are constructed and linked to form a multi-dimensional knowledge base, which is then loaded into a hybrid storage architecture. During the evaluation execution phase, questions are retrieved based on the multi-level index of the vector database, the output results are obtained by calling the large multi-modal model to be tested, and the PRM model is conditionally activated according to the PRM activation strategy and the availability of question step annotations. A comprehensive score is calculated, and evaluation decisions are made to advance, continue, or terminate the evaluation. In the evaluation closing phase, evaluation data is summarized, model capabilities are analyzed, and a complete evaluation report is generated. Simultaneously, it supports adapting evaluations to different scientific fields by replacing the core resources of the multi-dimensional knowledge base. This improves evaluation efficiency. Through automated initial deployment, question retrieval, scoring, and decision-making, no manual intervention is required throughout the process, significantly reducing labor costs. Furthermore, the grade-based evaluation can dynamically terminate or advance based on model capabilities, reducing invalid testing. The multi-level index of the vector database enables rapid question retrieval, further improving evaluation response speed and greatly enhancing evaluation efficiency. Furthermore, by conditionally activating the PRM model, a comprehensive two-dimensional scoring system of "process score + result score" is achieved. This system takes into account the integrity and logical consistency of the reasoning process in large-scale scientific multimodal models. Compared to existing single-result scoring or mandatory PRM scoring, the scoring is more comprehensive and accurate, while also standardizing the application of evaluation indicators and improving the current situation of inconsistent evaluation indicators. At the same time, by linking the vector database, PRM library, and evaluation indicator library to form a multidimensional knowledge base, only the scientific multimodal question samples, PRM models, and scoring rules in the knowledge base need to be replaced to adapt to the evaluation needs of different scientific fields, improving configurability and adaptability.

[0026] In this embodiment of the disclosure, such as Figure 1As shown, the agent used for multimodal large model evaluation comprises a four-layer architecture: a data layer, a preprocessing layer, a knowledge base layer, and an agent execution layer. The data layer is responsible for collecting open-source datasets and labeling them with domain classifications, evaluation dimensions, and difficulty levels. The preprocessing layer standardizes the data format and performs stratified sampling to construct a balanced test set. The knowledge base layer stores a structured question library, an evaluation metric library, and an open-source PRM model library for process reasoning scoring using RAG (Retrieval-Augmented Generation). The agent execution layer consists of three core modules orchestrated by LLM (Large Language Model): an intent understanding module parses user configurations to generate evaluation schemes; an evaluation module dynamically retrieves test subsets based on domain, evaluation dimensions, and levels, and achieves intelligent evaluation by combining process scores and result scores to advance through N consecutive questions or terminate through M consecutive questions; and a report generation module outputs a complete evaluation report including a capability radar chart and error analysis.

[0027] The data layer and preprocessing layer aim to transform scattered open-source datasets into structured, searchable, and unified evaluation resources. The knowledge base construction process is the core link connecting data preparation and agent execution, aiming to integrate open-source process reward models, evaluation metric rules, and structured metadata to form a knowledge hub supporting conditional process supervision and dynamic retrieval. Specifically, in some embodiments, the process for evaluating multimodal large models includes: constructing a vector database based on multimodal evaluation data samples; deploying a reward model (PRM) library and an evaluation metric library, wherein the evaluation metric library includes exact matching rules, semantic similarity matching rules, and one or more scoring rules from some credit mechanisms; associating and indexing the vector database, PRM library, and evaluation metric library to obtain a multidimensional knowledge base, and then loading the multidimensional knowledge base into a hybrid storage architecture of vector database and relational database.

[0028] Specifically, a vector database needs to be constructed to store multimodal question semantic vectors, original question information, and multi-level indexes. The multi-level indexes include domain indexes, dimension indexes, difficulty level indexes, and step annotation availability indexes. In some embodiments, constructing the vector database includes: acquiring multimodal evaluation data samples; labeling each data sample, wherein the labeling includes one or more of step annotations, domain label annotations, test dimension annotations, and difficulty level annotations; performing semantic vectorization processing on the labeled multimodal evaluation data samples to obtain multimodal question semantic vectors; constructing multi-level indexes based on the annotation results; and loading the evaluation data, annotation results, source metadata, and semantic vectors into the vector database.

[0029] Taking the scientific multimodal large model as an example, the process of constructing a vector database includes: First, collecting mainstream evaluation datasets from public channels, i.e., obtaining multimodal evaluation data samples, with corresponding fields including at least basic disciplines such as physics, chemistry, biology, mathematics, materials science, earth sciences, and astronomy. Open-source datasets cover various types such as multidisciplinary knowledge evaluation (e.g., MMLU, C-Eval), mathematical reasoning (GSM8K, MATH), science question answering (ScienceQA), tool usage (TOOLCOMP), and expert-level adversarial questions (GPQA), ensuring broad coverage and representativeness of the fields.

[0030] Subsequently, each data sample is labeled, including step annotation, domain label annotation, test dimension annotation, and difficulty level annotation. Specifically, firstly, step annotation detection is performed on each evaluation data sample to analyze the sample structure and identify whether it contains explicit reasoning step sequences (such as the step-by-step solution of GSM8K or the derivation process of MATH), and the boolean attribute `has_step_annotation` is marked. This mark directly determines whether the process reward model is used in subsequent evaluations, thus completing step annotation. Next, domain labels and difficulty levels are annotated. Domain labels are generated using a combination of manual and automatic methods, based on the original classification of the dataset or semantic matching of the question content to a predefined domain system (physics, chemistry, biology, mathematics, materials, earth science, astronomy, etc.), thus completing domain label annotation. At the same time, the test dimensions are divided into four aspects: concept recognition, factual question answering, scientific reasoning, and scientific multimodal, thus completing test dimension annotation. The difficulty levels are divided into five levels, L1 to L5, based on the inherent attributes of the questions (such as the number of steps in solving GSM8K or the competition level of MATH) or expert experience, forming a fine-grained difficulty hierarchy, thus completing difficulty level annotation.

[0031] Finally, the multimodal evaluation data samples with completed labeling are semantically vectorized to obtain multimodal question semantic vectors. Subsequently, a multi-level index is constructed based on the labeling results, and the evaluation data, labeling results, source metadata, and semantic vectors are loaded into the vector database. The multi-level index structure is built based on the labeling results, specifically using domain as the first-level key, dimension as the second-level key, difficulty level as the third-level key, and step label availability as the fourth-level key. Question content, standard answers, step labels, and source metadata are uniformly encoded and loaded into the vector database along with the multimodal question semantic vectors obtained from semantic vectorization, supporting efficient and accurate retrieval by the subsequent RAG module. This standardization process ensures consistent representation of heterogeneous data sources, providing a reliable data foundation for the dynamic question bank assembly of adaptive evaluation.

[0032] In some embodiments, the difficulty level classification is based on: the original difficulty label of the dataset from which the questions are sourced (such as the AIME / AMC levels of MATH), the number of solution steps / length of the reasoning chain, expert experience assessment, or pass rate statistics based on historical test data. Levels L1-L5 correspond to: Basic Concepts, Simple Applications, Comprehensive Reasoning, Complex Problems, and Expert Challenges, respectively.

[0033] After constructing a vector database based on multimodal evaluation data samples, a corresponding knowledge base can be built, namely, a reward model (PRM) library and an evaluation index library. The evaluation index library includes exact matching rules, semantic similarity matching rules, and one or more scoring rules from some credit mechanisms. After associating and indexing the vector database, PRM library, and evaluation index library to obtain a multidimensional knowledge base, the multidimensional knowledge base is loaded into a hybrid storage architecture of vector database and relational database.

[0034] Similarly, taking the scientific multimodal large model as an example, the specific construction process of the knowledge base includes: First, deploying the reward model PRM library: Deploying open-source process reward models (PRMs) to the inference service, including general mathematical inference models (such as Math-Shepherd), code inference models (such as CodePRM), and other domain-specific models, using only existing open-source weights without additional fine-tuning to reduce system complexity. For each PRM, compiling detailed metadata files: clarifying its applicable domain (mathematics, code, scientific inference, etc.), input format requirements (structured forms such as step pairs, code blocks, inference chains, etc.), output score definition (single-step score or aggregation strategy), and calling interface specifications, establishing the matching mapping relationship between PRMs and datasets, and completing the deployment of the PRM library.

[0035] Secondly, an evaluation metric library is deployed, which includes one or more scoring rules from exact matching rules, semantic similarity matching rules, and partial credit mechanisms. Specifically, the evaluation metric library is constructed, and result scoring rules are defined. For single-choice questions, exact matching rules are used; for multiple-choice questions, a partial credit mechanism (the proportion of correct options minus the penalty for incorrect options) is introduced; for fill-in-the-blank questions, semantic similarity rules (such as BERTScore) are used; and for open-ended questions, an LLM-as-Judge relative scoring strategy is configured, fully covering the three core scoring rules and realizing the deployment of the evaluation metric library.

[0036] Subsequently, the vector database, PRM database, and evaluation indicator database are linked and indexed to obtain a multidimensional knowledge base: the structured question database built in the preprocessing stage (i.e., the evaluation data and annotation information stored in the vector database) is linked and indexed with the PRM model database and evaluation indicator database. With the domain label as the top-level key, the dimension level, difficulty level, step annotation availability, applicable PRM identifier and corresponding scoring rules are linked downwards to form a multidimensional knowledge graph of "domain-dimension-level-question-PRM-indicator", thus completing the construction of the multidimensional knowledge base.

[0037] Finally, the multidimensional knowledge base is loaded into a hybrid storage architecture combining vector and relational databases. The complete knowledge base is stored in this architecture, where question content and its semantic features are vectorized to support similarity retrieval. Metadata relationships are linked through structured tables, ensuring that RAG retrieval can quickly retrieve suitable test subsets and corresponding evaluation strategy configurations based on user-specified domains and levels. The modular design of this knowledge base supports continuous expansion; new open-source datasets or PRM models can be integrated through standardized interfaces, adapting to evolving evaluation needs without modifying the core code.

[0038] The PRM applicability judgment rules include: when `topic.has_step_annotation` is true, `topic.domain` matches `PRM.applicable_domain`, and the model output format conforms to `PRM.input_format`, the PRM is considered usable. When multiple PRMs are applicable, priority is given to domain-specific PRMs (dedicated PRMs > general PRMs) or a user-specified strategy. This rule supports the reasonable use of PRM libraries and ensures the accuracy of evaluation and scoring.

[0039] Once the knowledge base is built, the agent can start the evaluation process of the multimodal large model, that is, the agent's execution layer begins to run. Figure 2 This is a flowchart illustrating a method for evaluating multimodal large models provided in an embodiment of this disclosure. Figure 2 As shown, the evaluation process for multimodal large models includes: Step 201: Parse the evaluation configuration information input by the user to obtain the current structured evaluation scheme. The configuration information includes one or more of the following: information of the multimodal large model to be tested, evaluation domain, evaluation dimension, level parameters, evaluation indicators, and reward model PRM activation strategy.

[0040] The above-mentioned agent execution layer includes three core modules of LLM orchestration. Among them, the intent understanding module can parse the evaluation configuration information input by the user to obtain the current structured evaluation scheme. Specifically, the intent understanding module parses multimodal large model information (name, API endpoint or local path), evaluation domain selection (multi-domain parallelism is possible), evaluation dimension selection, level parameter configuration (minimum / maximum number of test questions per level, consecutive correct advancement threshold, consecutive error termination threshold), and evaluation index settings (result scoring type, PRM activation strategy, process and result score weights), and generates the corresponding structured evaluation scheme. In some embodiments, a summary corresponding to the structured evaluation scheme can also be provided to the user for confirmation. After receiving the user's confirmation instruction, the structured evaluation scheme can be determined as the current structured evaluation scheme.

[0041] Step 202: Based on the constructed vector database, for each evaluation dimension in the current structured evaluation scheme, the execution level is set from low to high according to the preset difficulty level, and the multimodal large model to be tested is evaluated. The vector database stores multimodal question semantic vectors, question original information and multi-level indexes. The multi-level indexes include domain index, dimension index, difficulty level index and step annotation availability index.

[0042] Specifically, the evaluation module can traverse each evaluation dimension under each specified domain, initialize the evaluation state starting from the L1 basic concept level, and dynamically obtain the test subset corresponding to the current domain, current dimension and current difficulty level from the constructed multidimensional knowledge base through the RAG retrieval module (the selection of the test subset is based on the multi-level index of the vector database to ensure that the questions and evaluation requirements are accurately matched). Then, it executes the question testing process in a loop: calling the large multimodal model to be tested to obtain the model output results, calculating the result score based on the scoring rules in the evaluation index library, and conditionally determining whether to enable the PRM model according to the PRM applicability judgment rules - only calling it when the question contains step annotation (has_step_annotation is true) and there is a PRM model that is suitable for the current domain and current question; otherwise, skipping the PRM call and only calculating the result score; after merging the process score (if any) and the result score, updating the level evaluation statistics data, and deciding to advance to the next difficulty level, terminate the current dimension test in the domain or continue the current level test according to the threshold strategy configured by the user (promotion threshold, termination threshold, etc.), until all difficulty level tests under the dimension are completed or the termination condition is triggered.

[0043] In some embodiments, the climbing evaluation of the multimodal large model to be tested includes: retrieving a preset number of multimodal evaluation questions from a vector database based on the current evaluation domain, dimension, difficulty level, and step annotation availability, and calling the multimodal large model to be tested to obtain the corresponding current model output result information; obtaining the current comprehensive result score of the current model output result information based on the conditionally enabled PRM model, and executing the corresponding evaluation decision based on the current comprehensive result score, including: advancing to the next difficulty level, continuing the current level test, or terminating the current dimension evaluation.

[0044] The current comprehensive result score for obtaining the current model output information includes: when the corresponding PRM model is enabled based on the PRM activation strategy and the availability of question step annotations, obtaining the process score obtained by the PRM model evaluating the current model output information, and obtaining the result score by evaluating the current model output information based on the scoring rules in the evaluation index library; and, according to the preset weight allocation ratio, merging the process score and the result score to obtain the current comprehensive result score; when the corresponding PRM model is not enabled based on the PRM activation strategy and the availability of question step annotations, evaluating the current model output information based on the scoring rules in the evaluation index library to obtain the result score, and determining the result score as the current comprehensive result score.

[0045] In some embodiments, the corresponding evaluation decision includes: if the number of test questions for the current level reaches the minimum number of test questions, and the consecutive correct answers corresponding to the current comprehensive result score reach the advancement threshold, decide to advance to the next difficulty level; if the consecutive incorrect answers corresponding to the current comprehensive result score reach the termination threshold, decide to terminate the level climbing evaluation for the current dimension; if the number of test questions for the current level has not reached the maximum number of test questions, and the consecutive correct answers corresponding to the current comprehensive result score have not reached the advancement threshold, and the consecutive incorrect answers corresponding to the current comprehensive result score have not reached the termination threshold, decide to continue testing at the current level; if the number of test questions for the current level has reached the maximum number of test questions, and the consecutive incorrect answers corresponding to the current comprehensive result score have not reached the termination threshold, record the comprehensive score for the current level; and if the current level is not the highest difficulty level, decide to advance to the next difficulty level; if the current level is the highest difficulty level, decide to terminate the level climbing evaluation for the current dimension.

[0046] Similarly, taking the scientific multimodal large model as an example, the evaluation process includes: based on the current evaluation domain, current dimension, current difficulty level, and step annotation availability, a preset number of multimodal evaluation questions are retrieved from the knowledge base via RAG retrieval. For each current level test question, the multimodal large model to be tested is invoked, and the corresponding result score = score_outcome(model_output, ground_truth), where score_outcome is the scoring function based on the corresponding rules in the evaluation indicator library, model_output is the output result of the model to be tested, and ground_truth is the standard answer to the question. If the PRM model is enabled based on the conditions, the corresponding process score = PRM(ground_truth_steps, model_output), and the current comprehensive result score = α × process score + (1-α) × result score. If the PRM model is not enabled based on the conditions, the current comprehensive result score = result score.

[0047] During the evaluation process, the number of test questions for the current level is incremented by 1 in real time. Based on the current overall score, the corresponding consecutive correct or consecutive incorrect counts are updated. Specifically, if a correct answer is determined based on the current overall score, the consecutive correct count is incremented by 1, and the consecutive incorrect count is reset to zero; conversely, if an incorrect answer is determined, the consecutive incorrect count is incremented by 1, and the consecutive correct count is reset to zero. The specific decision-making logic is as follows: If the number of test questions for the current level reaches the minimum number of test questions, and the consecutive correct count reaches the advancement threshold, the level is advanced to the next difficulty level. If the consecutive incorrect count reaches the termination threshold, the current level progression evaluation is terminated. If the number of test questions for the current level has not reached the maximum number of test questions, and the consecutive correct count has not reached the advancement threshold or the consecutive incorrect count has not reached the termination threshold, the current level test continues. If the number of test questions for the current level reaches the maximum number of test questions, and the consecutive incorrect count has not reached the termination threshold, the overall score for the current level is recorded. Furthermore, if the current level is not the highest difficulty level, the level is advanced to the next difficulty level. If the current level is the highest difficulty level, the current level progression evaluation is terminated, until all areas and all dimensions are evaluated.

[0048] Step 203: Summarize the evaluation results of all evaluation dimensions, analyze the results, and generate the corresponding current evaluation report. The evaluation results include: the process score obtained by the PRM model when the corresponding PRM model is activated based on the PRM activation strategy and the availability of question step annotations.

[0049] The report generation module in the intelligent agent can output a complete evaluation report including a capability radar chart and error analysis. In some embodiments, generating the corresponding current evaluation report includes: analyzing the evaluation results based on all evaluation dimensions to obtain the corresponding current capability analysis results, wherein the evaluation results include: the comprehensive score for each level, PRM activation records, and evaluation decisions; and the capability analysis results include: the capability advantages, shortcomings, and improvement suggestions of the multimodal large model under test in each evaluation domain, dimension, and difficulty level; and generating the corresponding current evaluation report based on the current structured evaluation scheme, evaluation results, and current energy analysis results.

[0050] The evaluation results specifically include: the overall score for each difficulty level, PRM activation records (including the activated PRM model identifier, applicable scenarios, number of calls, and process score details), and evaluation decisions (including specific nodes and judgment criteria for advancement, continuation, and termination). In addition, multi-dimensional in-depth analysis is conducted by combining the results scores of each dimension, continuous correct / incorrect statistics, level advancement records, and other data. The final capability analysis results include: the capabilities of the multimodal large model under test in each evaluation domain, each evaluation dimension, and each difficulty level, as well as the capability weaknesses and targeted improvement suggestions. Capability strengths highlight the domains, dimensions, and difficulty levels where the model performs exceptionally well (e.g., excellent scores in the concept recognition dimension at the L1-L2 basic level), clearly identifying the task scenarios the model is suited for. Capability weaknesses precisely pinpoint the weak areas of the model's performance (e.g., low scores in the scientific reasoning dimension at the L4-L5 high difficulty level, inaccurate multimodal information fusion, etc.). Combined with PRM activation records and process scores, the core reasons for these weaknesses are further analyzed (e.g., missing reasoning steps, logical loopholes, insufficient PRM adaptability, etc.). Improvement suggestions, based on the causes of weaknesses and evaluation data, propose feasible optimization directions (e.g., supplementing with high-difficulty level scientific reasoning samples, optimizing PRM model adaptation strategies, strengthening the model's multimodal information fusion capabilities, etc.).

[0051] The report generation module will integrate the current structured evaluation scheme, evaluation results, and current energy analysis results according to the preset standardized template to form a complete and standardized evaluation report, ensuring that the report content is comprehensive, logically clear, and highlights the key points. In some embodiments, the evaluation report includes: First, basic evaluation information, clarifying the core configuration of the current structured evaluation scheme (information on the multimodal large model to be tested, evaluation domain, evaluation dimensions, level parameters, PRM activation strategy, etc.), allowing readers to quickly grasp the evaluation background and execution standards; Second, a summary of evaluation results, clearly presenting the comprehensive scores of each domain, dimension, and difficulty level, specific evaluation data for each level, PRM activation records, and evaluation decision details in the form of tables or lists, ensuring that the evaluation data is verifiable and traceable; Third, a display of capability analysis results, combined with a capability radar chart (visually presenting the capability performance of each domain and dimension), detailing the model's capability advantages, shortcomings, and improvement suggestions, and intuitively presenting the overall capability level of the model; Fourth, supplementary error analysis, classifying and statistically analyzing the questions that the model outputs incorrectly during the evaluation process, linking them to the PRM process scores (if any) and evaluation decision records, deeply analyzing the error types and causes, and providing data support for improvement suggestions; Fifth, a comprehensive evaluation conclusion, based on the evaluation results and capability analysis, giving an overall capability level assessment of the model, clarifying the applicable scope of the model and the focus of subsequent optimization.

[0052] Furthermore, the report generation module supports flexible adjustment of the report's presentation format and content detail based on user-configured evaluation needs. It can selectively output a simplified report (containing only core scores and capability radar charts) or a full report (including all evaluation details, error analysis, and optimization suggestions). Simultaneously, by integrating a multi-dimensional knowledge base, the report generation module can access evaluation indicator rules, PRM application records, and other information from the knowledge base to ensure the professionalism and accuracy of the report analysis. This further enhances the reference value of the evaluation report and provides comprehensive and reliable data support for the iterative optimization of the large multimodal model under test.

[0053] As can be seen, this embodiment of the present disclosure achieves accurate, efficient, and configurable evaluation of large-scale scientific multimodal models by constructing a fully automated evaluation system encompassing "preliminary deployment - evaluation execution - evaluation completion" and relying on the collaborative support of a four-layer intelligent agent architecture, a vector database, and a multi-dimensional knowledge base. Through a conditional activation of the PRM model's dual-dimensional scoring mechanism and a grade-climbing evaluation strategy based on multi-level indexes, it not only solves the technical pain points of low evaluation efficiency, inaccurate scoring, and poor adaptability in existing evaluations, but also achieves rapid adaptation to the evaluation needs of different scientific fields through standardized process design and a modular knowledge base architecture, enabling the updating and expansion of evaluation resources without modifying the core code. The evaluation scheme provided in this embodiment not only provides a standardized and traceable technical path for the capability assessment of large-scale multimodal models, but also provides data-driven support for iterative optimization of models, possessing broad application prospects and practical value.

[0054] The following describes the operational process in a specific embodiment, illustrating the multimodal large model evaluation process provided by the embodiments of the present invention.

[0055] In one embodiment of this disclosure, the multimodal large model is a scientific multimodal large model. To evaluate this multimodal large model, a corresponding knowledge base needs to be constructed. Combined with... Figure 1 , Figure 3 The process of constructing a knowledge graph includes: Step 301: Obtain multimodal scientific evaluation data samples and label each data sample. The labeling includes one or more of the following: step labeling, domain labeling, test dimension labeling, and difficulty level labeling.

[0056] Multimodal evaluation samples from basic disciplines such as physics and chemistry can be collected from public channels, and their reasoning steps, domains, test dimensions, and L1-L5 difficulty levels can be labeled, laying the foundation for the subsequent construction of a vector database.

[0057] Step 302: Perform semantic vectorization on the multimodal evaluation data samples with completed labeling to obtain multimodal question semantic vectors, construct a multi-level index based on the labeling results, and load the evaluation data, labeling results, source metadata, and semantic vectors into the vector database.

[0058] By semantic vectorization, questions are transformed into searchable vectors, and a multi-level index of "domain-dimension-difficulty-step availability" is constructed to ensure the efficiency of subsequent question retrieval.

[0059] Step 303: Deploy the reward model PRM library and the evaluation index library, wherein the evaluation index library includes exact matching rules, semantic similarity matching rules, and one or more scoring rules from some credit mechanisms.

[0060] Deploy open-source PRM models such as Math-Shepherd and compile metadata archives, while defining scoring rules such as exact matching for single-choice questions and partial credit mechanisms for multiple-choice questions.

[0061] Step 304: After linking and indexing the vector database, PRM database, and evaluation index database to obtain a multidimensional knowledge base, load the multidimensional knowledge base into a hybrid storage architecture of vector database and relational database.

[0062] In this way, a multi-dimensional knowledge graph of "domain-dimension-level-question-PRM-indicator" can be constructed, and the hybrid storage architecture can take into account both vector retrieval and structured query requirements to complete the construction of the knowledge base.

[0063] Once the scientific knowledge base is obtained, a large-scale scientific multimodal model can be evaluated. The minimum number of test questions is 3, the maximum number is 10, the advancement threshold is 4, and the termination threshold is 3. Figure 3 , Figure 4 The process of evaluating scientific multimodal large models includes: Step 401: Obtain and parse the evaluation configuration information input by the user to obtain the current structured evaluation scheme. The configuration information includes one or more of the following: the information of the multimodal large model to be tested, the evaluation domain, the evaluation dimension, the level parameter, the evaluation index, and the reward model PRM activation strategy.

[0064] Step 402: Based on the current evaluation domain, dimension, difficulty level, and step annotation availability, retrieve a preset number of multimodal evaluation questions from the vector database.

[0065] Based on the multi-level index of "domain-dimension-difficulty-step availability" of the vector database, questions can be retrieved starting from the L1 (basic concept) level to ensure that the retrieved questions are accurately matched with the current assessment requirements.

[0066] Step 403: Take the retrieved multimodal assessment question of the current dimension and difficulty level as the current multimodal assessment question, call the large multimodal model to be tested, and obtain the corresponding current model output result information.

[0067] Based on a multi-level index of "domain-dimension-difficulty-step availability" of a vector database, questions are retrieved starting from the L1 (basic concept) level to ensure that the retrieved questions are accurately matched with the current assessment requirements.

[0068] Step 404: Based on the PRM activation strategy and the availability of the question steps, determine whether to activate the corresponding PRM model. If yes, proceed to step 405; otherwise, proceed to step 406.

[0069] Step 405: Obtain the process score obtained by the PRM model in evaluating the current model's output information, and obtain the result score by evaluating the current model's output information based on the scoring rules in the evaluation index library. Then, according to the preset weight allocation ratio, merge the process score and the result score to obtain the current comprehensive result score. Proceed to step 407.

[0070] Step 406: Based on the scoring rules in the evaluation index library, evaluate the current model output information to obtain a result score, and determine the result score as the current comprehensive result score. Proceed to step 407.

[0071] Step 407: Increment the number of test questions at the current level by 1, and update the corresponding consecutive correct or consecutive incorrect counts based on the current overall score. If the current overall score determines that the test is correct, increment the consecutive correct count by 1 and reset the consecutive incorrect count to zero; otherwise, if the test is determined to be incorrect, increment the consecutive incorrect count by 1 and reset the consecutive correct count to zero.

[0072] Step 408: Execute the corresponding assessment decision based on the current level test number, consecutive correct answers, and consecutive incorrect answers.

[0073] If the number of questions in the current level test reaches ≥3, and the number of consecutive correct answers is ≥4, and the current level is not the highest difficulty level, then the decision is made to advance to the next difficulty level, that is, the next difficulty level is taken as the current difficulty level, and the process returns to step 403; otherwise, if the number of consecutive incorrect answers is ≥3, then the decision is made to terminate the current dimension's level climbing assessment.

[0074] If there are 3 or fewer questions than the current level test number and less than 10 questions, and the consecutive correct answers are less than 4 and the consecutive incorrect answers are less than 3, then the decision is made to proceed with the current level test, i.e., return to step 403. If there are 10 or more questions than the current level test number and the consecutive incorrect answers are less than 3, then the decision is made to record the overall score for the current level; and if the current level is not the highest difficulty level, then the decision is made to advance to the next difficulty level, i.e., the next difficulty level is used as the current difficulty level, and return to step 403.

[0075] Step 409: Have all dimensions been evaluated? If yes, proceed to step 411; otherwise, proceed to step 410.

[0076] Step 410: Identify an incomplete evaluation dimension as the current dimension, and initialize the evaluation level of that dimension to L1 level. Return to step 403 and start the level climbing evaluation of that dimension to ensure that all preset dimensions can complete a comprehensive evaluation.

[0077] Step 411: Based on the evaluation results of all evaluation dimensions, conduct result analysis to obtain the corresponding current capability analysis results. The evaluation results include: the comprehensive score of each level, PRM activation record and evaluation decision. The capability analysis results include: the capability advantages, shortcomings and improvement suggestions of the multimodal large model under test in each evaluation domain, dimension and difficulty level.

[0078] Step 412: Generate the corresponding current evaluation report based on the current structured evaluation scheme, evaluation results, and current energy analysis results.

[0079] The report includes basic evaluation information, results summary, capability radar chart, error analysis, and comprehensive evaluation conclusions. It can output a simplified or full version according to user needs, providing a standardized and traceable basis for model capability evaluation.

[0080] As can be seen, in this embodiment, after completing multimodal data annotation, vector database construction, deployment of the PRM library and evaluation index library, and loading of the multidimensional knowledge base, a standardized evaluation knowledge base adapted to the scientific field is constructed. Based on this standardized evaluation knowledge base, and following preset minimum / maximum test question counts, advancement and termination thresholds, user configuration parsing, question retrieval, model invocation, dual-dimensional scoring, level progression decision-making, and report generation are completed, achieving automated and accurate evaluation of large-scale scientific multimodal models. This clearly demonstrates the practical application of core technologies such as the four-layer intelligent agent architecture, conditional PRM activation, and multi-level index retrieval, ensuring both the standardization and traceability of the evaluation process, and accurately identifying model capability shortcomings, providing practical and feasible technical support for the iterative optimization of large-scale scientific multimodal models.

[0081] Based on the above process for evaluating multimodal large models, a device for evaluating multimodal large models can be constructed, the function of which can correspond to the intelligent agent used for evaluating multimodal large models, such as... Figure 5 As shown, it may include: an intent understanding module 510, an evaluation module 520, and a report generation module 530.

[0082] The intent understanding module 510 is configured to parse the evaluation configuration information input by the user to obtain the current structured evaluation scheme. The configuration information includes one or more of the following: the information of the multimodal large model to be tested, the evaluation domain, the evaluation dimension, the level parameter, the evaluation index, and the reward model PRM activation strategy.

[0083] The evaluation module 520 is configured to perform a climbing evaluation of the multimodal large model under test based on the constructed vector database for each evaluation dimension in the current structured evaluation scheme, according to the preset difficulty level from low to high. The vector database stores multimodal question semantic vectors, question original information and multi-level indexes, including domain index, dimension index, difficulty level index and step annotation availability index.

[0084] The report generation module 530 is configured to summarize the evaluation results of all evaluation dimensions, perform result analysis, and generate the corresponding current evaluation report. The evaluation results include the process score obtained through the PRM model when the corresponding PRM model is activated based on the PRM activation strategy and the availability of question step annotations.

[0085] It is evident that the functions of the intent understanding module 510, the evaluation module 520, and the report generation module 530 can be precisely matched with the functions of the agent execution layer in the agent used for multimodal large model evaluation. The three work together to fully realize the core execution process of multimodal large model evaluation, ensuring the automation and standardization of the evaluation process.

[0086] In some embodiments, it also includes: The building module is configured to construct a vector database based on multimodal evaluation data samples.

[0087] The deployment module is configured to deploy the reward model PRM library and the evaluation metric library. The evaluation metric library includes exact matching rules, semantic similarity matching rules, and one or more scoring rules from some credit mechanisms.

[0088] The associated loading module is configured to associate and index the vector database, PRM library, and evaluation index library to obtain a multidimensional knowledge base, and then load the multidimensional knowledge base into a hybrid storage architecture of vector database and relational database.

[0089] In some embodiments, the construction module is specifically configured to acquire multimodal evaluation data samples, label each data sample, wherein the labeling includes one or more of the following: step labeling, domain labeling, test dimension labeling, and difficulty level labeling; perform semantic vectorization processing on the multimodal evaluation data samples with completed labeling to obtain multimodal question semantic vectors; construct a multi-level index based on the labeling results, and load the evaluation data, labeling results, source metadata, and semantic vectors into a vector database.

[0090] It is evident that the functions of the construction module, deployment module, and associated loading module can be matched one by one with the functions of the data layer, preprocessing layer, and knowledge base layer in the intelligent agent used for multimodal large model evaluation. The three work together to complete the resource preparation and knowledge base construction work in the early stage of evaluation, forming a complete technical closed loop with the three modules of the execution layer, ensuring the functional integrity and process continuity of the entire evaluation device.

[0091] In some embodiments, the evaluation module 520 includes: The model output unit is configured to retrieve a preset number of multimodal assessment items from the vector database based on the current assessment domain, dimension, difficulty level, and step annotation availability, and then call the large multimodal model to be tested to obtain the corresponding current model output results. The scoring decision unit is configured to use a conditionally enabled PRM model to obtain the current comprehensive result score of the current model output information, and execute the corresponding evaluation decision based on the current comprehensive result score. The evaluation decision includes: advancing to the next difficulty level, continuing the current level test, or terminating the current dimension evaluation.

[0092] In some embodiments, the scoring decision unit is specifically configured to, when determining to enable the corresponding PRM model based on the PRM activation strategy and the availability of question step annotations, obtain a process score obtained by evaluating the current model's output information using the PRM model, and evaluate the current model's output information based on the scoring rules in the evaluation index library to obtain a result score, and then, according to a preset weight allocation ratio, merge the process score and the result score to obtain the current comprehensive result score; when determining not to enable the corresponding PRM model based on the PRM activation strategy and the availability of question step annotations, evaluate the current model's output information based on the scoring rules in the evaluation index library to obtain a result score, and determine the result score as the current comprehensive result score.

[0093] In some embodiments, the scoring decision unit is further configured to: when the number of test questions for the current level reaches the minimum number of test questions, decide to advance to the next difficulty level if the consecutive correct answers corresponding to the current comprehensive result score reach the advancement threshold; and decide to terminate the level climbing assessment for the current dimension if the consecutive incorrect answers corresponding to the current comprehensive result score reach the termination threshold; when the number of test questions for the current level has not reached the maximum number of test questions, decide to continue testing at the current level if the consecutive correct answers corresponding to the current comprehensive result score have not reached the advancement threshold, and if the consecutive incorrect answers corresponding to the current comprehensive result score have not reached the termination threshold; when the number of test questions for the current level has reached the maximum number of test questions, decide to record the comprehensive score for the current level if the consecutive incorrect answers corresponding to the current comprehensive result score have not reached the termination threshold; and if the current level is not the highest difficulty level, decide to advance to the next difficulty level; and if the current level is the highest difficulty level, decide to terminate the level climbing assessment for the current dimension.

[0094] In some embodiments, the report generation module 530 is configured to perform result analysis based on the evaluation results of all evaluation dimensions to obtain the corresponding current capability analysis results. The evaluation results include: the comprehensive score for each level, PRM activation records and evaluation decisions. The capability analysis results include: the capability advantages, shortcomings and improvement suggestions of the multimodal large model under test in each evaluation domain, dimension and difficulty level. Based on the current structured evaluation scheme, evaluation results and current energy analysis results, the corresponding current evaluation report is generated.

[0095] As can be seen, in this embodiment, the device for evaluating multimodal large models can be modularly designed, breaking down the various functions of pre-evaluation preparation, mid-evaluation execution, and post-evaluation cleanup into corresponding modules. Each module has a clear responsibility and works together efficiently, achieving precise functional matching with each level and module of the multimodal large model evaluation agent, while also possessing strong scalability and maintainability. Through the refined design of the evaluation modules, the automation and precision of the graded climbing evaluation are achieved. Combined with the two-dimensional scoring mechanism of conditionally activating the PRM model, the comprehensiveness and accuracy of the scoring are improved. Through the in-depth analysis function of the report generation module, not only can standardized evaluation reports be output, but also the model's capability shortcomings can be accurately located and targeted improvement suggestions can be provided, providing strong support for the capability assessment and iterative optimization of multimodal large models.

[0096] Furthermore, the modular architecture of this device allows each module to be upgraded independently and flexibly adapted to different scenario requirements. When adapting to new scientific evaluation fields, it is only necessary to update the multimodal evaluation data through the construction module and update the PRM library and evaluation index library through the deployment module, without modifying the core code of the device, thus quickly completing the adaptation and significantly reducing the maintenance cost and expansion difficulty of the evaluation device. At the same time, the functional design of each module of the device strictly follows the corresponding evaluation process specifications mentioned above, ensuring the standardization and traceability of the evaluation process, making the evaluation results comparable for different users and in different scenarios, further enhancing the practicality and promotion value of the evaluation device. It can be widely used in various scientific multimodal large model capability evaluation scenarios, providing reliable evaluation support for the technological iteration and industrial application of multimodal large models.

[0097] Combination Figure 6 This disclosure provides an apparatus 600 for multimodal large model evaluation, comprising: The processor 1000 and memory 1001 may further include a communication interface 1002 and a bus 1003. The processor 1000, communication interface 1002, and memory 1001 can communicate with each other via the bus 1003. The communication interface 1002 can be used for information transmission. The processor 1000 can call logical instructions in the memory 1001 to execute the method for multimodal large model evaluation described in the above embodiment.

[0098] Furthermore, the logic instructions in the aforementioned memory 1001 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0099] The memory 1001, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 1000 executes functional applications and data processing by running the program instructions / modules stored in the memory 1001, that is, it implements the method for multimodal large model evaluation in the above method embodiments.

[0100] The memory 1001 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 1001 may include high-speed random access memory and may also include non-volatile memory.

[0101] This disclosure provides an apparatus for evaluating a multimodal large model, including: a processor and a memory storing program instructions, wherein the processor is configured to execute a method for evaluating a multimodal large model when executing the program instructions.

[0102] The multimodal large model evaluation system includes: the device itself, combined with... Figure 7 This disclosure provides a device body 700, on which the aforementioned multimodal large model evaluation device 500 (600) is mounted. The mounting relationship described herein is not limited to placement within the product, but also includes mounting connections with other components of the product, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the multimodal large model evaluation device 500 (600) can be adapted to feasible device bodies to achieve other feasible embodiments.

[0103] This disclosure provides a storage medium storing program instructions that, when executed, perform the method described above for multimodal large model evaluation.

[0104] This disclosure provides a computer program product, which includes a computer program stored on a storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the above-described method for evaluating multimodal large models.

[0105] The aforementioned storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0106] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0107] The foregoing description and accompanying drawings fully illustrate embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included or replace parts and features of other embodiments. The scope of the embodiments of this disclosure includes the entire scope of the claims and all available equivalents of the claims. While the terms “first,” “second,” etc., may be used in this application to describe elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be called a second element without changing the meaning of the description, and similarly, a second element may be called a first element, provided that all occurrences of “first element” are consistently renamed and all occurrences of “second element” are consistently renamed. First and second elements are both elements, but may not be the same element. Moreover, the terminology used in this application is only for describing embodiments and is not intended to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. Similarly, the term “and / or” as used herein means including one or more of the associated listed any and all possible combinations. Additionally, when used herein, the terms “comprise” and its variations “comprises” and / or “comprising” refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase “comprising an…” does not exclude the presence of additional identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0109] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for evaluating large multimodal models, characterized in that, include: The evaluation configuration information input by the user is parsed to obtain the current structured evaluation scheme. The configuration information includes one or more of the following: the information of the multimodal large model to be tested, the evaluation domain, the evaluation dimension, the level parameter, the evaluation index, and the reward model PRM activation strategy. Based on the constructed vector database, for each evaluation dimension in the current structured evaluation scheme, the execution level is set from low to high according to the preset difficulty level to perform a climbing evaluation of the multimodal large model to be tested. The vector database stores multimodal question semantic vectors, question original information and multi-level indexes. The multi-level indexes include domain index, dimension index, difficulty level index and step annotation availability index. The evaluation results of all evaluation dimensions are summarized, the results are analyzed, and the corresponding current evaluation report is generated. The evaluation results include: the process score obtained by the PRM model when the corresponding PRM model is determined to be enabled based on the PRM activation strategy and the availability of question step annotation.

2. The method according to claim 1, characterized in that, Also includes: A vector database is constructed based on multimodal evaluation data samples; Deploy a reward model PRM library and an evaluation metric library, wherein the evaluation metric library includes exact matching rules, semantic similarity matching rules, and one or more scoring rules from some credit mechanisms; After linking and indexing the vector database, PRM database, and evaluation index database to obtain a multidimensional knowledge base, the multidimensional knowledge base is loaded into a hybrid storage architecture of vector database and relational database.

3. The method according to claim 2, characterized in that, The constructed vector database includes: Obtain multimodal evaluation data samples and label each data sample. The labeling includes one or more of the following: step labeling, domain labeling, test dimension labeling, and difficulty level labeling. The multimodal evaluation data samples with completed labeling are semantically vectorized to obtain multimodal question semantic vectors; A multi-level index is constructed based on the annotation results, and the evaluation data, annotation results, source metadata, and semantic vectors are loaded into the vector database.

4. The method according to claim 2, characterized in that, The hill-climbing evaluation of the large multimodal model under test includes: Based on the current evaluation domain, dimension, difficulty level, and availability of step annotations, a preset number of multimodal evaluation questions are retrieved from the vector database, and the large multimodal model to be tested is called to obtain the corresponding current model output results information; Based on the conditionally enabled PRM model, the current comprehensive result score of the current model output information is obtained, and the corresponding evaluation decision is executed according to the current comprehensive result score. The evaluation decision includes: advancing to the next difficulty level, continuing the current level test, or terminating the current dimension evaluation.

5. The method according to claim 4, characterized in that, The current comprehensive result score obtained from the current model output information includes: If the corresponding PRM model is activated based on the PRM activation strategy and the availability of question step annotations, the process score obtained by the PRM model is evaluated on the output result information of the current model. Based on the scoring rules in the evaluation index library, the output result information of the current model is evaluated to obtain the result score. In addition, the process score and the result score are integrated and calculated according to the preset weight allocation ratio to obtain the current comprehensive result score. If the corresponding PRM model is not enabled based on the PRM activation strategy and the availability of question step annotations, the output result information of the current model is evaluated based on the scoring rules in the evaluation index library to obtain the result score, and the result score is determined as the current comprehensive result score.

6. The method according to claim 4, characterized in that, The evaluation and decision-making process includes: If the number of test questions at the current level reaches the minimum number of test questions, and the number of consecutive correct answers corresponding to the current comprehensive result score reaches the advancement threshold, the decision is made to advance to the next difficulty level. If the number of consecutive incorrect answers corresponding to the current comprehensive result score reaches the termination threshold, the decision is made to terminate the level climbing assessment for the current dimension. If the maximum number of test questions for the current level has not been reached, and if the number of consecutive correct answers corresponding to the current overall score has not reached the advancement threshold, and if the number of consecutive incorrect answers corresponding to the current overall score has not reached the termination threshold, then the decision is made to continue the current level test. If the maximum number of test questions for the current level has been reached, and the number of consecutive errors corresponding to the current comprehensive score has not reached the termination threshold, the comprehensive score for the current level is recorded; and if the current level is not the highest difficulty level, a decision is made to advance to the next difficulty level, and if the current level is the highest difficulty level, a decision is made to terminate the level climbing assessment for the current dimension.

7. The method according to any one of claims 1-6, characterized in that, The generation of the corresponding current evaluation report includes: Based on the evaluation results of all evaluation dimensions, the results are analyzed to obtain the corresponding current capability analysis results. The evaluation results include: the comprehensive score of each level, PRM activation records and evaluation decisions. The capability analysis results include: the capability strengths, weaknesses and improvement suggestions of the multimodal large model under test in each evaluation domain, dimension and difficulty level. Based on the current structured evaluation scheme, evaluation results, and current energy analysis results, a corresponding current evaluation report is generated.

8. An apparatus for evaluating multimodal large models, the apparatus comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to perform the method for multimodal large model evaluation as described in any one of claims 1 to 7 when executing the program instructions.

9. A system for evaluating large multimodal models, comprising: Equipment body; The apparatus for multimodal large model evaluation as described in claim 8 is installed on the device body.

10. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the method for evaluating multimodal large models as described in any one of claims 1 to 7.

Images