A multi-modal large model evaluation system, method and computer readable storage medium

Abstract

The application relates to the technical field of computers, in particular to a multi-modal large model evaluation system and method and a computer readable storage medium; the system comprises a data storage and global knowledge labels; a first evaluation device, a second evaluation device; a comprehensive score device; the application provides a system and method capable of distinguishing the visual understanding ability and the knowledge expression ability of a model and objectively and quantitatively evaluating the two; the problems that the existing technology has strong subjectivity, lacks diagnosis and is difficult to objectively and quantitatively evaluate unstructured output when evaluating a multi-modal large model are solved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to testing and evaluation technology for artificial intelligence models, and particularly to an evaluation system and method for multimodal large models in a specific knowledge domain (such as the appreciation of traditional Chinese painting). Background Technology

[0002] In recent years, generative artificial intelligence (AI) based on large-scale neural networks and self-attention mechanisms has made groundbreaking progress. In particular, the Multimodal Large Language Model (MLLM) combines powerful visual encoders (such as Vision Transformers, ViTs) with large language models (LLMs) to achieve a unified understanding and processing capability for multiple information modalities, including images and text. In the highly specialized vertical field of traditional Chinese painting appreciation, MLLMs have demonstrated enormous application potential. For example, a user can input an image of an ancient painting, and the model can generate an analytical text that describes the painting's theme, composition, brushwork techniques, historical background, and artistic conception.

[0003] However, how to scientifically, objectively, and comprehensively evaluate the performance of such models remains a serious challenge in the current technological field. Existing evaluation methods mainly suffer from the following insurmountable technical problems: Subjectivity and Instability of Evaluation Results: Appreciation of traditional Chinese painting is a complex aesthetic activity, heavily influenced by personal subjectivity. If manual evaluation is used, different experts may give drastically different assessments of the same model-generated text, leading to a lack of consistency and reproducibility. Even with the recently popular "LLM-as-a-Judge" approach (using a more powerful language model as the judge), where the model assigns a macro-level quality score (e.g., 1-10), the judgment criteria remain vague and uninterpretable, and the model itself may be biased, resulting in equally unstable evaluation results.

[0004] Confusion in evaluation dimensions and lack of diagnostic value: The reasons behind a model generating an incorrect analytical text can be multifaceted. One possibility is insufficient underlying visual understanding, failing to accurately identify key elements in the image—in other words, "not understanding it." For example, mistaking a crane in a painting for an egret. Another possibility is that while the model understands the image, its internal knowledge base is flawed, or there is "hallucination" in the language generation process—in other words, "incorrect interpretation." For example, while correctly identifying the crane, it incorrectly associates it with a typical subject of an incorrect era and an incorrect painter. Existing end-to-end evaluation methods often only yield a general "good" or "bad" conclusion, failing to differentiate between these two levels of capability deficiencies. Therefore, the evaluation results lack diagnostic value and cannot provide model developers with clear and actionable optimization directions.

[0005] The challenge of objectively quantifying the evaluation of unstructured output: Analysis texts generated by multimodal large models are typical examples of unstructured natural language, with highly varied content organization, sentence structure, and word choice. This makes traditional automated evaluation methods based on exact matching or simple rules virtually impossible. How to objectively and quantitatively assess the accuracy of the knowledge points contained in a free-flowing descriptive text is a problem that current technologies have not effectively solved. Summary of the Invention

[0006] This invention aims to address the problems of strong subjectivity, lack of diagnostic capabilities, and difficulty in objectively quantifying unstructured outputs in existing technologies for evaluating large multimodal models. Specifically, this invention strives to provide a system and method capable of distinguishing between a model's visual understanding ability and its knowledge representation ability, and capable of objectively and quantitatively evaluating both separately.

[0007] To achieve the above objectives, the core technical solution adopted by this invention is as follows: A multimodal large model evaluation system, comprising: A data store containing baseline truth data associated with at least one image, the baseline truth data including: Visual information about at least one local object in the image; and at least one global knowledge label about the image; The first evaluation device is configured to: initiate a first query to a large multimodal model under test based on visual information, obtain its first response to the visual information, and compare the first response with the visual information to generate a first evaluation result; The second evaluation device is configured to: acquire unstructured text generated by the multimodal large model under test as an image, call a general large language model independent of the multimodal large model under test, instruct it to extract structured information corresponding to global knowledge tags from the unstructured text, and compare the structured information with the global knowledge tags to generate a second evaluation result. The comprehensive scoring device is configured to calculate a comprehensive evaluation score based on the first evaluation result and the second evaluation result.

[0008] Further, preferably, the visual information includes the location information of local objects; and, when comparing the first response with the visual information, the first evaluation device is configured to: calculate the intersection-union ratio (IUGR) between the location indicated in the first response and the location information, and determine the correctness of the first response based on whether the IUGR reaches a preset threshold. Further, preferably, the general large language model is a pre-trained large language model with general instruction following capabilities, and its parameter count and model structure are designed to be independent of the multimodal large model under test. Further, preferably, the global knowledge tag is selected from at least one of the following: the subject matter depicted in the image, the techniques used in the image creation process, the historical period of the image's creation, the image's author, or the art movement to which the image belongs. Further, preferably, it also includes a question generation device, configured to: automatically generate a query statement for initiating the first query and prompt words for generating unstructured text based on the visual information and the global knowledge tag. Furthermore, preferably, a multimodal large model evaluation method is also provided for performing the multimodal large model evaluation system described above, comprising the following steps: providing benchmark ground truth data associated with at least one image by a data storage, the benchmark ground truth data including visual information about at least one local object in the image, and at least one global knowledge label about the image; The first evaluation process is executed by the first evaluation device. The process includes: initiating a first query to a large multimodal model under test based on visual information to obtain its first response to the visual information, and comparing the first response with the visual information to generate a first evaluation result. The second evaluation process is executed by the second evaluation device. The process includes: acquiring unstructured text generated by the multimodal large model under test as an image, calling a general large language model independent of the multimodal large model under test, instructing it to extract structured information corresponding to global knowledge tags from the unstructured text, and comparing the structured information with the global knowledge tags to generate a second evaluation result. A comprehensive evaluation score is calculated by a comprehensive scoring device, based on the first evaluation result and the second evaluation result. Further, preferably, the visual information includes the location information of local objects; and, in the first evaluation process, the step of comparing the first response with the visual information specifically includes: calculating the intersection-union ratio (IUGR) between the location indicated in the first response and the location information, and determining the correctness of the first response based on whether the IUGR reaches a preset threshold. Further, preferably, the global knowledge tag is selected from at least one of the following: the subject matter depicted in the image, the techniques used in the image creation process, the historical period of the image's creation, the image's author, or the art movement to which the image belongs. Further, preferably, before executing the first evaluation process, a question generation step is included, performed by a question generation device. This step automatically generates a query statement for initiating a first query and prompts for generating unstructured text based on the visual information and the global knowledge tag, wherein the query statement can be constructed as a multiple-choice question containing distracting options. Further, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the above method. This invention provides a multimodal large model evaluation system. The core technical concept of this system lies in establishing a dual-track parallel evaluation process. The system includes: a data storage device, a first evaluation device, a second evaluation device, and a comprehensive scoring device.

[0009] Data storage is used to provide the "gold standard" or "benchmark truth" for evaluation. Its unique feature is that it stores two different levels of information for each image: the first level is specific visual information about local objects in the image, such as the precise location and attributes of a particular object; the second level is abstract, global knowledge tags about the image as a whole, such as the subject matter or technique of the painting.

[0010] The first evaluation device constitutes the first track of the evaluation, specifically designed to deeply explore the underlying image understanding capabilities of the model under test. Utilizing the aforementioned local visual information, it directly initiates targeted, structured queries to the model under test (e.g., "What is the object located at coordinates [x,y,w,h] in the image?"), and precisely compares its response with the baseline truth value, thereby deriving an objective "first evaluation result," such as image understanding accuracy.

[0011] The second evaluation device, constituting the second evaluation track, is the key innovation of this invention, specifically designed to objectively evaluate the quality of unstructured text generated by the model under test. It first allows the model to generate a free-form analytical text. Then, instead of directly evaluating this text, it invokes a separate, typically more powerful, general-purpose language model, assigning it a new role—an "objective information extractor." This general-purpose language model is instructed to extract structured information corresponding to predefined global knowledge tags from the unstructured text generated by the model under test. Finally, by comparing the extracted structured information with the baseline truth value, an objective "second evaluation result," such as the accuracy of the analytical text, is obtained.

[0012] The comprehensive scoring device, as the final convergence point, calculates a final, comprehensive evaluation score by weighted fusion and other methods, taking the evaluation results from the two tracks mentioned above. Accordingly, this invention also provides a multimodal large model evaluation method executed by the aforementioned system, which realizes the functions of each device in the system through corresponding steps.

[0013] Compared with the prior art, the technical solution provided by the present invention brings the following significant beneficial effects: Depth and Diagnostic Aspects of Evaluation: By establishing two parallel evaluation tracks—a "first evaluation device" and a "second evaluation device"—this invention successfully decouples the model's overall capability into two independent dimensions: "accuracy of perception" (reflected by the first evaluation result) and "correctness of expression" (reflected by the second evaluation result). If a model scores high in the first evaluation but low in the second evaluation, it can be clearly diagnosed that the problem lies primarily in the knowledge association or language generation stages, rather than the visual understanding stage. This diagnostic evaluation result greatly improves the efficiency and relevance of model optimization.

[0014] Objectivity and Reproducibility of Evaluation: This invention solves the challenge of objectively evaluating unstructured text through an innovative paradigm shift. It transforms the evaluation task from "having one model subjectively evaluate another model's text" to "having one model objectively extract information from another model's text." Since information extraction is a more convergent and objective task than subjective evaluation, and the extracted structured information can be precisely matched with the baseline truth, it greatly eliminates subjectivity and instability in the evaluation process, resulting in highly objective and reproducible evaluation results.

[0015] Comprehensiveness and Flexibility of Evaluation: This invention simultaneously examines the model's ability to recognize microscopic details in images and its ability to grasp and express macroscopic knowledge, resulting in a more comprehensive evaluation system. Furthermore, by weighted fusion of the first and second evaluation results, users can flexibly adjust the weights according to different evaluation needs (e.g., whether to prioritize the model's faithful descriptive ability or its knowledge richness), achieving customized evaluation.

[0016] Automation and high efficiency: The entire evaluation process, from initiating queries, obtaining responses, calling models to extract information, to final scoring, can be automated, greatly improving the efficiency of large-scale, systematic testing of multimodal models. Attached Figure Description

[0017] To enable those skilled in the art to more clearly and comprehensively understand the technical solutions of the present invention, preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the accompanying drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the accompanying drawings: Figure 1 This is a functional block diagram of a multimodal large model evaluation system according to an embodiment of the present invention.

[0018] Figure 2 This is a flowchart of a multimodal large model evaluation method according to an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings: To make the objectives, technical solutions, and advantages of this invention clearer, several preferred embodiments of the invention will be described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely for explaining the invention and do not constitute any limitation on its scope of protection. Any modifications, equivalent substitutions, or improvements made based on the spirit and principles of this invention should be included within the scope of protection of this invention.

[0020] Example 1 Please see Figure 1This embodiment provides a multimodal large model evaluation system 100, which aims to perform a comprehensive and objective performance evaluation of an external, testable multimodal large model. In terms of physical implementation, the system 100 can be deployed on one or more cloud servers equipped with high-performance central processing units (CPUs), graphics processing units (GPUs), large-capacity memory (RAM), high-speed solid-state drives (SSDs), and network interfaces. The various logical devices of the system can be encapsulated using containerization technologies (such as Docker) and deployed and managed through a microservice architecture (such as orchestration using Kubernetes) to ensure high availability and scalability.

[0021] The system 100 logically includes a data storage 10, a first evaluation device 30, a second evaluation device 40, and a comprehensive scoring device 50.

[0022] Data storage 10 is the cornerstone of the entire evaluation system, and its core function is to store benchmark truth data used for evaluation. In this embodiment, data storage 10 can be a document database, as such databases are naturally suitable for storing flexible data structures, which perfectly matches the complex and diverse annotation information in this embodiment. For each traditional Chinese painting image used as test material, data storage 10 stores a corresponding structured data record. This record contains benchmark truth data in two core parts.

[0023] The first part is visual information about at least one local object in the image. This information is a precise description of the image's microscopic content. For example, for a local object, its annotation information may include a unique object ID, such as "obj001"; an object category, such as "birds"; a specific object name, such as "golden pheasant"; and its location information. This location information can be recorded as a bounding box, defined by the coordinate values ​​of its top-left and bottom-right vertices. For example, a set of coordinate values ​​could be (850, 220, 1250, 750), representing the minimum, minimum, maximum, and maximum values ​​along the x-axis and y-axis, respectively.

[0024] The second part is at least one global knowledge tag about the image. This part of the information is a summary of the macroscopic artistic features and background knowledge of the painting. The global knowledge tag is selected from at least one of the following: the theme depicted in the image, the technique used in the creation process of the image, or the historical period of the creation of the image. For example, for a traditional Chinese painting, its global knowledge tag may include multiple key-value pairs. One key is "theme", and its corresponding value is "flower-and-bird painting"; another key is "technique", and its corresponding value can be a list containing two elements, "meticulous brushwork" and "color application"; there is also a key "historical period of creation", and its value is "Northern Song Dynasty"; there is also a key "author", and its corresponding value can be the name of the painter, and there is also a key "school of painting", and its corresponding value is a list of the painter and the school of painting style. This annotation method that coexists micro-visual information and macro-knowledge information provides a solid data basis for subsequent dual-track evaluation.

[0025] The first evaluation device 30 is the core for performing the first-track evaluation. It is configured to: initiate a first query to a multi-modal large model to be tested based on visual information to obtain its first response to the visual information, and compare the first response with the visual information to generate a first evaluation result. Specifically, the device reads the visual information of a local object from the data storage 10, such as information about animals and people. Based on this information, it initiates a request to the multi-modal large model to be tested. The request will encapsulate the query content in a structured manner. For example, it specifies the URL address of the image to be processed, a field representing the query type, and a field containing specific query parameters. The query parameter field can contain an array of coordinates for defining the position, and is accompanied by a natural language question, such as "Please identify what the object in the specified area of the picture is?". After the model to be tested processes the request, it returns a response, which is the first response. For example, a well-performing model may return a structured data containing the answer, where one field is "object_name", and its value is "golden pheasant". The first evaluation device 30 performs a complete string match of the obtained "golden pheasant" with the "golden pheasant" in the ground truth. If the match is successful, this query is recorded as correct.

[0026] In a preferred embodiment, the visual information includes the location information of local objects, and the goal of the query is to enable the test model to locate the objects. In this case, the first query could be: "Please box the location of the 'pheasant' in the image." The first response returned by the test model is an array of coordinates. During comparison, the first evaluation device 30 is configured to calculate the Intersection over Union (IoU) between the bounding box returned by the test model and the baseline ground truth bounding box. The IoU is calculated as the ratio of the intersection area to the union area of ​​the two bounding boxes. An internal threshold is preset, for example, 0.75. If the calculated IoU value is greater than 0.75, the localization response is considered correct. In practice, this threshold is configurable; a higher threshold (e.g., 0.9) represents a more precise localization, while a lower threshold (e.g., 0.5) allows for a coarser localization. By querying and evaluating all local objects in the test set, the first evaluation device 30 ultimately calculates a first evaluation result, such as an "image understanding accuracy" of 90%.

[0027] The second evaluation device 40 is the core of the second-track evaluation. It is configured to: acquire unstructured text generated by the multimodal large model under test as an image; call a general large language model independent of the multimodal large model under test to instruct it to extract structured information corresponding to global knowledge tags from the unstructured text; and compare the structured information with the global knowledge tags to generate the second evaluation result.

[0028] The detailed workflow is as follows: First, the second evaluation device 40 sends a more open request to the model under test, providing only an image and requesting it to generate an analytical text. The model under test may return an unstructured text, such as: "This painting vividly depicts a colorful golden pheasant perched among blooming hibiscus flowers. The painter's brushwork is extremely meticulous, belonging to the category of meticulous brushwork, and the colors are gorgeous, reflecting the typical style of the Song Dynasty painting academy."

[0029] Next, the second evaluation device 40 does not attempt to parse this complex natural language itself. Instead, it uses this text as "raw material" to invoke an independent general-purpose large language model. This general-purpose large language model is a pre-trained, large-scale language model with general instruction-following capabilities, such as OpenAI's GPT-4o or DeepSeek. Its "independence" is reflected in its model structure, parameters, and training data being completely different from the model under test, thus avoiding the bias of "testing itself" and ensuring the fairness of the evaluation. The second evaluation device 40 sends a carefully crafted instruction or prompt to this general-purpose large language model. For example, the prompt first assigns a role to the general-purpose large language model, such as "a rigorous art history information extraction assistant." Then, the prompt clearly states the task: to extract specific information points from the subsequently provided text, such as "subject matter," "technique," and "historical period of creation." The prompt also includes constraints, such as returning a null value if the information is not found, and requiring the results to be returned in a specific structured format. Finally, the prompt appends the unstructured text generated by the model under test as the content to be processed.

[0030] After executing the above instructions, the general-purpose large language model returns structured information. For example, it identifies that the "golden pheasant" and "hibiscus" mentioned in the text belong to the subject matter of flower-and-bird painting, the phrase "extremely fine brushwork" corresponds to the meticulous brushwork technique, and "Song Dynasty Painting Academy" corresponds to the historical period of the Song Dynasty. This information extracted by the general-purpose large language model forms the basis for objective comparison. After receiving this structured information, the second evaluation device 40 compares it with the global knowledge tags in the data storage 10. In this example, "subject matter" and "technique" both match successfully, as does "historical period of creation." By performing the same information extraction and comparison on the appreciation texts generated from all images in the test set, the second evaluation device 40 finally calculates a second evaluation result, such as an "accuracy rate of appreciation text" of 85.0%.

[0031] The comprehensive scoring device 50 is the final aggregation point for the evaluation results. It is configured to calculate a comprehensive evaluation score based on the first and second evaluation results. The calculation method is typically a weighted average: Comprehensive Evaluation Score = w1 * First Evaluation Result + w2 * Second Evaluation Result, where w1 and w2 are preset weights, and w1 + w2 = 1. These weights can be set through a configuration file or dynamically adjusted through the system's management interface. For example, when the evaluation focuses on the model's fidelity, w1 = 0.7 and w2 = 0.3 can be set; when the focus is on knowledge richness, w1 = 0.3 and w2 = 0.7 can be set.

[0032] In a preferred embodiment, the system 100 further includes a question generation device 20. This device can be connected to the data storage 10 to automatically and in batches generate the required query statements and prompts for the first evaluation device 30 and the second evaluation device 40, thereby achieving a high degree of automation of the entire evaluation process. For example, it can read the label "golden pheasant" and automatically apply a template to generate the question "Please identify the golden pheasant in the picture".

[0033] Example 2: The following will be combined with Figure 2 The methods executed by the aforementioned system are described in detail. These methods embody the execution process of the aforementioned system 100.

[0034] The method begins with S100: providing baseline ground truth data. This step is the preparation phase for the evaluation. In a specific application scenario, this step includes: a team of professionals using specialized annotation tools processing a selected batch of traditional Chinese paintings. They carefully select key local objects in the paintings and assign attribute labels to them; simultaneously, they annotate the paintings with accurate global knowledge labels. All these annotation results are formatted and imported into data storage 10 for use in subsequent steps.

[0035] Next, the method enters a parallel evaluation process.

[0036] The first evaluation process is executed. This step, performed by the first evaluation device 30, aims to evaluate the underlying visual understanding capability of the model under test. It specifically includes a series of sub-operations: First, a first query is initiated. The system selects visual information of a local object from the data storage 10, such as the location coordinates of a "bridge" in the painting "Along the River During the Qingming Festival." Then, it sends a request containing the image and location coordinates to the model under test, querying for the name of the object at that location. Next, a first response is obtained. The system receives the response returned by the model under test, for example, the response being "bridge." Finally, a comparison and generation of the first evaluation result are performed. The system compares the response "bridge" with the baseline ground truth "rainbow bridge." Here, a fuzzy matching algorithm can be used, determining "bridge" as partially correct and awarding 0.5 points; or a more stringent complete match can be used, determining it as incorrect and awarding 0 points. In another scenario, the query requests the location of a "bridge," and the model under test returns a bounding box. In this case, the core of this step is calculating the IoU value between the returned bounding box and the baseline ground truth bounding box. If the IoU value is 0.82 and the preset threshold is 0.75, the answer is considered correct. The system will iterate through all test items and finally calculate the overall accuracy or average score as the first evaluation result.

[0037] The second evaluation process is executed. This step, performed by the second evaluation device 40, aims to objectively evaluate the knowledge accuracy of the text generated by the model under test. It also includes a series of sub-operations: First, unstructured text is acquired. The system sends an image of "Along the River During the Qingming Festival" to the model under test and requests it to generate an appreciation text. The system acquires a short natural language text of several hundred words describing the prosperous scene in the painting. Then, a general-purpose large language model is invoked to extract information. The system sends the acquired text, along with a carefully designed instruction to extract "subject matter," "technique," and "historical period of creation," to an independent and powerful general-purpose large language model. After processing, the general-purpose large language model returns structured information. Finally, the second evaluation result is compared and generated. The system compares each item in the returned structured information with the global knowledge tags about "Along the River During the Qingming Festival" in the data storage 10 (e.g., including "genre painting," "boundary painting," "Northern Song Dynasty," Zhang Zeduan, genre painting (or boundary painting), etc.). In this example, if the extracted information does not match the baseline truth, it is recorded as an error. The system records the correctness of each knowledge point and finally calculates the average knowledge point accuracy of all test samples as the second evaluation result.

[0038] S400: Calculate the comprehensive evaluation score. This step is performed by the comprehensive scoring device 50 and is the final stage of the evaluation. The system, based on preset weights (e.g., w1=0.6, w2=0.4), weights and sums the first evaluation result obtained from the first evaluation process (e.g., image understanding accuracy 88%) and the second evaluation result obtained from the second evaluation process (e.g., text analysis accuracy 75%) to obtain the final comprehensive evaluation score: 0.6 * 88 + 0.4 * 75 = 52.8 + 30 = 82.8 points. This score objectively and comprehensively reflects the overall performance of the model under test in this evaluation.

[0039] In a more optimized workflow, a question generation step performed by the question generation device 20 can be added before executing the first evaluation process. In this step, the system can automatically convert the label "bridge" into a multiple-choice question with distractors: "Which magnificent bridge is shown in the picture? A. Baodai Bridge B. Rainbow Bridge C. Seventeen-Arch Bridge," and mark "Rainbow Bridge" as the correct answer. Distractors A and C are automatically generated by querying a built-in "Ancient Chinese Bridges" knowledge base to find other famous bridges at the same level as "Rainbow Bridge." This method can test the model's discrimination ability in greater depth.

[0040] Finally, the present invention also protects a computer-readable storage medium, such as a server hard drive, cloud storage disk, USB flash drive, or optical disk. This medium stores computer program code that, when loaded and executed by a processor, implements the aforementioned evaluation method.

[0041] Those skilled in the art should understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. For example, the field of evaluation can be expanded from traditional Chinese painting to any other field of graphic and text generation requiring in-depth knowledge; these changes do not depart from the core technical ideas of the present invention.

Claims

1. A multimodal large model evaluation system, characterized in that, include: A data storage (10) stores reference truth data associated with at least one image, the reference truth data including: Visual information about at least one local object in the image; and at least one global knowledge label about the image; The first evaluation device (30) is configured to: initiate a first query to a multimodal large model to be tested based on the visual information, to obtain its first response to the visual information, and compare the first response with the visual information to generate a first evaluation result; The second evaluation device (40) is configured to: acquire unstructured text generated by the multimodal large model under test for the image, call a general large language model independent of the multimodal large model under test, instruct it to extract structured information corresponding to the global knowledge tag from the unstructured text, and compare the structured information with the global knowledge tag to generate a second evaluation result; The comprehensive scoring device (50) is configured to calculate a comprehensive evaluation score based on the first evaluation result and the second evaluation result.

2. The multimodal large model evaluation system according to claim 1, characterized in that, The visual information includes the location information of the local object; and the first evaluation device (30) is configured to: calculate the intersection-union ratio between the location indicated in the first response and the location information when comparing the first response with the visual information, and determine the correctness of the first response based on whether the intersection-union ratio reaches a preset threshold.

3. The multimodal large model evaluation system according to claim 1, characterized in that, The general large language model is a pre-trained large language model with general instruction following capabilities. Its parameter count and model structure are designed to be independent of the multimodal large model under test.

4. The multimodal large model evaluation system according to claim 1, characterized in that, The global knowledge tag is selected from at least one of the following: the subject matter depicted in the image, the techniques used in the creation of the image, the historical period of the creation of the image, the author of the image, or the school of painting to which the image belongs.

5. The multimodal large model evaluation system according to claim 1, characterized in that, It also includes a question generation device (20) configured to automatically generate a query statement for initiating the first query and prompt words for generating the unstructured text based on the visual information and the global knowledge tags.

6. A method for evaluating a multimodal large model, used to execute a multimodal large model evaluation system according to any one of claims 1-5, characterized in that, Includes the following steps: The data storage (10) provides reference truth data associated with at least one image, the reference truth data including visual information about at least one local object in the image, and at least one global knowledge label about the image; The first evaluation device (30) performs a first evaluation process, which includes: initiating a first query to a multimodal large model under test based on the visual information to obtain its first response to the visual information, and comparing the first response with the visual information to generate a first evaluation result; The second evaluation device (40) performs a second evaluation process, which includes: obtaining unstructured text generated by the multimodal large model under test for the image, calling a general large language model independent of the multimodal large model under test, instructing it to extract structured information corresponding to the global knowledge tag from the unstructured text, and comparing the structured information with the global knowledge tag to generate a second evaluation result; The comprehensive evaluation score (S400) is calculated by the comprehensive scoring device (50), and the calculation is based on the first evaluation result and the second evaluation result.

7. The multimodal large model evaluation method according to claim 6, characterized in that, The visual information includes the location information of the local object; Furthermore, the step of comparing the first response with the visual information in the first evaluation process specifically includes: calculating the intersection-union ratio (IUU) between the location indicated in the first response and the location information, and determining the correctness of the first response based on whether the IUU reaches a preset threshold.

8. The multimodal large model evaluation method according to claim 6, characterized in that, The global knowledge tag is selected from at least one of the following: the subject matter depicted in the image, the techniques used in the creation of the image, the historical period of the creation of the image, the author of the image, or the school of painting to which the image belongs.

9. The multimodal large model evaluation method according to claim 6, characterized in that, Before executing the first evaluation process, a question generation step is also included. This step automatically generates a query statement for initiating the first query and prompt words for generating the unstructured text based on the visual information and the global knowledge tags. The query statement can be constructed as a multiple-choice question containing distracting options.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the multimodal large model evaluation method as described in any one of claims 6 to 9.

Images