Evaluation test method of multi-modal chart question and answer large model, medium and equipment

Abstract

The invention relates to the field of large model evaluation, in particular to an evaluation test method of a multi-modal chart question and answer large model, a medium and equipment. Comprising the following steps: inputting a judgment class test data set into a to-be-evaluated large model to obtain a judgment class model output result; the question information in the judgment class question and answer pair comprises a question ontology text and prompt information that the answer information can only be front or back information. And carrying out statistics on accuracy information of all blank filling class model output results, selection class model output results and judgment class model output results, and generating execution evaluation information of the to-be-evaluated large model. In view of potential variability possibly existing in the following of the multi-modal large model to the instructions, in the evaluation of the low-order task performance, the used instruction types comprise three question types of judgment questions, gap filling questions and selection questions proposed from the positive and negative angles to ask the model to be evaluated; therefore, the to-be-evaluated large model is evaluated more comprehensively.

Description

Background Art

[0002] With the rapid development of deep learning technology, the scale of model parameters is getting larger and larger, from the initial millions of parameters to hundreds of billions or even trillions of parameters today. This growth has brought new challenges to computing resources, model training methods, and model performance evaluation. The application areas of large models are also constantly expanding, from natural language processing to computer vision, to recommendation systems, etc., covering almost all AI application areas. The differences in task characteristics and requirements in different fields make it difficult for a single evaluation standard to meet the needs of all scenarios. As models become more and more complex, how to improve their efficiency while ensuring model performance has become an important issue.

[0003] In recent years, with the emergence of GPT4o, multimodal large language models have demonstrated significant capabilities in multimodal understanding and generation. However, they have limited understanding of chart data. As an important research direction in the field of multimodal artificial intelligence, the multimodal chart question-answering task aims to return the answer that best matches the question description given a chart image and a question description, so as to better help users understand the content of the chart image or generate text descriptions related to the chart image. This task is committed to improving the understanding and analysis capabilities of chart tasks through the fusion of visual and language understanding of chart data. This type of task can be divided into two categories: subjective question answering and objective question answering. Objective question answering aims to extract the specific data point location, numerical value or classification problems in the chart image, such as data retrieval, extreme value, classification, etc.; subjective question answering aims to require the model to be able to generate deeper explanations and reasoning capabilities based on the content of the chart image, such as chart conversion, chart redrawing, chart viewpoint summary, etc.

[0004] Although existing benchmarking methods for multimodal graph question-answering models have made some progress in evaluating models, they lack comprehensive test method designs for subjective and objective task types, and the models show poor stability when facing a variety of specific task scenarios. In addition, there are a series of key problems such as the benchmark being too simple, the task evaluation dimensions being too few, and the evaluation indicators being too single. This makes it challenging to accurately evaluate the graph comprehension capabilities of large multimodal models. Summary of the invention

[0005] In view of the above technical problems, the technical solution adopted by the present invention is:

[0006] According to one aspect of the present invention, a method for evaluating and testing a multimodal graphical question-answering model is provided, the method comprising the following steps:

[0007] Obtain the fill-in-the-blank test data set, selection test data set, and judgment test data set corresponding to each type of low-level task. Each chart test image in the fill-in-the-blank test data set corresponds to a set of fill-in-the-blank type question-answer pairs. Each chart test image in the selection test data set corresponds to a set of selection type question-answer pairs. Each chart test image in the judgment test data set corresponds to a set of judgment type question-answer pairs. Low-level tasks are question-answering tasks that focus on specific, detail-oriented queries, seek or compare precise data points in charts, and involve direct factual information retrieval.

[0008] Each chart test image in the fill-in-the-blank test data set and the question information in the corresponding question-answer pair are input into the large model to be evaluated to obtain the fill-in-the-blank model output result corresponding to each chart test image.

[0009] The fill-in-the-blank model output results corresponding to each chart-type test image and the answer information in the corresponding question-answer pair are input into GPT4o for matching to generate the accuracy information of the fill-in-the-blank model output results corresponding to each chart-type test image.

[0010] Each chart test image and question information in the question-answer pair in the judgment test data set are input into the large model to be evaluated to obtain the judgment model output result corresponding to each chart test image. The question information in the question-answer pair in the judgment test data set includes prompt information for the question body text and the answer information can only be positive or negative information.

[0011] Each chart test image in the selection test data set and the question information in the question-answer pair are input into the large model to be evaluated to obtain the selection model output result corresponding to each chart test image.

[0012] Use regular expressions to obtain the selection model output results or judgment model output results corresponding to each chart test image and match them with the answer information in the corresponding question and answer pair to generate accuracy information of the selection model output results or judgment model output results corresponding to each chart test image.

[0013] The accuracy information of all the fill-in-the-blank model output results, selection model output results and judgment model output results corresponding to each type of low-level task is statistically analyzed to generate the execution evaluation information of the large model to be evaluated for each type of low-level task.

[0014] Furthermore, the method also includes:

[0015] Get a high-level test dataset corresponding to each type of high-level task. Each chart-type test image in the high-level test dataset corresponds to a set of high-level question-answer pairs. High-level tasks are tasks that focus on understanding the overall trend or pattern of the chart or context summary.

[0016] Each chart-type test image in the high-level test data set and the question information in the corresponding high-level question-answer pair are input into the large model to be evaluated to obtain the high-level model output result corresponding to each chart-type test image.

[0017] The high-level model output results corresponding to each chart-type test image and the answer information in the corresponding high-level question-answer pair are input into GPT4o for evaluation and scoring to generate the accuracy information of the high-level model output results corresponding to each chart-type test image. GPT4o evaluates and scores based on the fluency of the generated answers, the ability to follow instructions, the length of the generated text, and the completeness of the results.

[0018] The accuracy information of all high-order model output results corresponding to each type of high-order task is statistically analyzed to generate the execution evaluation information of the large model to be evaluated for each type of high-order task.

[0019] Based on the execution evaluation information of each type of high-level task and the execution evaluation information of each type of low-level task, the evaluation information of the large model to be evaluated is generated.

[0020] Furthermore, the test data set corresponding to each type of high-level task or low-level task is obtained, including:

[0021] Use the self-instruct model to generate multiple seed questions corresponding to each graph class test image.

[0022] Use the Qwen2.5-0.5b model to score each seed question, and take the seed question with the highest score as the intermediate question.

[0023] Use GPT4o to rewrite the intermediate questions according to the task type and generate multiple target questions corresponding to each chart type test image.

[0024] The initial annotated test data consisting of each target question and the corresponding chart-type test image is input into the Qwen2-VL-72b model to generate the answer information corresponding to each initial annotated test data to obtain the target test data.

[0025] Furthermore, before using the self-instruct model to generate a plurality of seed questions corresponding to each chart class test image, the method further includes:

[0026] The obtained initial chart-type test images are all converted into RGB format to generate a first test image set.

[0027] The low-quality images in the first test image set are removed to generate the second test image set. The low-quality images include low-resolution, over-compressed, blank or invalid images, and images with the same content repeated.

[0028] Median filtering is used to remove image noise in the second test image set to generate a chart-type test image set in the test data set.

[0029] Furthermore, before inputting each chart-type test image in the fill-in-the-blank test data set and the question information in the corresponding question-answer pair into the large model to be evaluated to obtain the fill-in-the-blank model output result corresponding to each chart-type test image, the method further includes:

[0030] Use the API interface access method or the model weight access method to obtain the large model to be evaluated.

[0031] Furthermore, the types of chart test images include stacked bar charts, complex line charts, scatter charts, pie charts, regular line charts, grouped bar charts, regular bar charts, three-dimensional bar charts, bubble charts and mixed charts.

[0032] Further, the categories of low-level tasks include reasoning, anomalies, distribution, correlation, ranging, sorting, filtering, data retrieval, extrema, and clustering.

[0033] Furthermore, the categories of high-level tasks include chart classification, opinion suggestions, chart redrawing, chart conversion, chart summary, chart filling assistance, text generation, assisted creation, multi-chart question and answer, multi-round chart dialogue, chart OCR recognition and information positioning.

[0034] According to a second aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned evaluation and testing method for a multimodal graphical question-and-answer large model.

[0035] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned evaluation and testing method for a multimodal graphical question and answer large model when executing the computer program.

[0036] The present invention has at least the following beneficial effects:

[0037] In the present invention, in the evaluation test of the modal diagram question-answering big model for low-level tasks (i.e., objective question-answering tasks), in order to quantitatively analyze the performance of the multimodal big model on the diagram question-answering task, a more reasonable evaluation is performed by setting multiple types of instructions and evaluation indicators. And in view of the potential variability in the compliance of the multimodal big model with the instructions, in the evaluation of the performance of low-level tasks, the instruction types used include three types of questions, namely, judgment questions, fill-in-the-blank questions, and multiple-choice questions, which are raised from positive and negative perspectives to ask questions to the evaluation model, thereby more comprehensively evaluating the evaluation big model through a richer type of questions.

[0038] In addition, in judging the consistency between the model output results and the corresponding annotated data, in order to avoid the interference of unnecessary information caused by the diverse expressions of the fill-in-the-blank model output results, GPT4o is used as a referee model to evaluate the consistency between the model output results and the annotated answer information. This can more accurately judge whether the model output results are accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of an evaluation and testing method for a multimodal graphical question-and-answer large model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0042] As a possible embodiment of the present invention, Figure 1 As shown, a method for evaluating and testing a multimodal graph question-answering large model is provided, and the method includes the following aspects:

[0043] S1: Get the test dataset.

[0044] In this embodiment, the chart question-and-answer tasks can be classified into three main categories: self-contained tasks, low-level tasks, and high-level tasks. Among them, self-contained tasks are the capabilities that the model itself should have. They are the main tasks or goals assigned during model design and training, and are the basic capabilities of the model. High-level tasks focus on questions that require understanding the overall context or summary of the chart, involving broader, goal-oriented queries aimed at understanding overall trends or patterns. In contrast, low-level tasks focus on specific, detail-oriented queries, seeking precise data points or comparisons in charts, and involve direct factual information retrieval.

[0045] Own tasks, high-order tasks and low-order tasks can be further classified based on their focus on chart question and answer. Own tasks can be divided into two subcategories: safety and self-cognition; low-order tasks can be divided into 10 subcategories: reasoning, anomalies, distribution, correlation, range determination, sorting, filtering, data retrieval, extreme values, and clustering; high-order tasks can be divided into 12 subcategories: chart classification, opinion suggestions, chart redrawing, chart conversion, chart summary, auxiliary chart filling, text generation, assisted creation, multi-chart Q&A, multi-round chart dialogue, chart OCR recognition and information positioning.

[0046] The evaluation method in this embodiment is mainly to evaluate the performance of the modal graph question and answer model in low-order tasks and high-order tasks. Therefore, when acquiring the test data set, the corresponding test data set is mainly acquired for the task categories included in the low-order tasks and high-order tasks.

[0047] S1 includes:

[0048] S100: Collection and acquisition of chart data. Specifically, the types of chart test pictures in this embodiment include stacked bar charts, complex line charts, scatter charts, pie charts, regular line charts, grouped bar charts, regular bar charts, three-dimensional bar charts, bubble charts and mixed charts.

[0049] For the image data in the chart question-answering task, the acquisition of image data can be divided into two categories, namely chart images and non-chart images.

[0050] The acquisition channels of chart images include collecting on the Internet through keywords, public data sets of chart question-answering tasks, and parsing chart images from papers or technical report documents. In this embodiment, two methods are adopted: public chart data sets and parsing from documents. The public chart data sets include ChartSumm, DVQA, PlotQA, ChartQA, FigureQA, ScigraphQA, TinyChart, Chart-to-Text, ChartBench, ChartLLama, MMC, ChartAssistant, OneChart, ChartX, and ChartInsights, a total of 15 public data sets. Parsing from documents is mainly parsed from papers and report documents in PDF format of financial reports. In this way, a sufficient number of chart images can be obtained.

[0051] The acquisition channels for non-graphic images are mainly domestic and foreign public image data, such as public recognition tasks, detection tasks, image generation tasks, and image data in multimodal instruction data (such as COCO, LAION, LLAVA-Instruct, etc.). In addition, for the images in our own tasks, we collect image data such as news or logos of related companies from different company websites.

[0052] After obtaining a large number of chart test images, you need to clean and deduplicate these data to select high-quality and diverse image data. The following are the corresponding steps:

[0053] S101: Convert all acquired initial chart test images into RGB format to generate a first test image set.

[0054] This step converts and unifies the format of the pictures, unifying pictures of different formats into a standard format. The standard format includes the storage format and size of the image, and converts all pictures into RGB format to avoid the influence of different color spaces.

[0055] S102: Remove low-quality pictures from the first test picture set to generate a second test picture set. Low-quality pictures include low-resolution, over-compressed, blank or invalid pictures, and pictures with the same content repeated.

[0056] S103: Use median filtering to remove image noise in the second test image set to generate a chart test image set in the test data set.

[0057] In S103, manual inspection can also be used to check whether some images contain extreme color values ​​(extreme color values ​​refer to colors of the image background that interfere with the icon expression), transition saturation and other abnormal conditions. Finally, the filtered images are deduplicated to remove duplicate images, thereby ensuring the quality, accuracy and diversity of the image data set and improving the efficiency and effect of the next step of instruction labeling.

[0058] After the above processing, the relevant chart-like images in the test data set have been obtained, and the subsequent instruction annotation information needs to be generated based on each chart-like image. Specifically, in the present embodiment, the instruction annotation information is to generate multiple question-answer pairs based on the content of the chart-like image. Each question-answer pair belongs to a set of annotation information, where multiple question-answer pairs can be annotation information corresponding to multiple types of question-answering tasks. For example, the question information in multiple question-answer pairs corresponding to a certain chart-like image may include: 1. What type of chart is this? What labels are there on the x-axis? 3. What are the data labels for each element? 4. Is the length of the bar chart for "Thursday" abnormal?

[0059] Specifically, when the instruction data is marked on the cleaned image data, the following steps can be followed:

[0060] S104: Generate multiple seed questions Q1 corresponding to each chart type test image using the self-instruct model.

[0061] In this step, you can also set corresponding prompt instruction information for the self-instruct model to standardize and improve the effectiveness of the model output information. For example, the prompt instruction information here can be [“You are a professional expert in understanding chart content. You can generate chart classification task questions based on the chart content. Be sure to analyze the key points of the content accurately and carefully: 1. The classification in the generated question is consistent with the chart type in the picture. 2. The generated questions are fluent and highly complex.”].

[0062] S105: Use the Qwen2.5-0.5b model to score each seed question, and obtain the seed question with the highest score as the intermediate question Q2.

[0063] After generating multiple seed questions corresponding to a certain type of task, it is necessary to select the best quality ones for subsequent expansion and rewriting. In this step, you can also set corresponding prompt instruction information for the Qwen2.5-0.5b model to standardize and improve the effectiveness of the model output information. For example, the prompt instruction information here can be [“You are a professional text content evaluation expert who can select the best quality text data based on the input text content. Be sure to evaluate it in three aspects: text length, text fluency, and text complexity.”].

[0064] S106: Use GPT4o to rewrite the intermediate questions according to the task type, and generate multiple target questions Q3 corresponding to each chart-type test image.

[0065] This step is to expand and rewrite the questions based on the best quality intermediate questions to improve the diversity and richness of the questions. In this step, you can also set corresponding prompt instruction information for the GPT4o model to standardize and improve the effectiveness of the model output information. For example, the prompt instruction information here can be [“You are a professional text content rewriting expert. Your task is to rewrite the text content according to diversity, complexity, and fluency. Please pay attention to the following points for the rewritten text content: 1. The text is more complex; 2. The text fluency is better; 3. The text is more diverse.”].

[0066] S107: Input the initial annotated test data consisting of each target question and the corresponding chart-type test image into the Qwen2-VL-72b model, generate answer information corresponding to each initial annotated test data, and obtain the target test data.

[0067] The initial labeled test data consisting of each Q1 and the corresponding chart-type test image is then input into the Qwen2-VL-72b question-answering model to generate the corresponding answer A1, thereby forming a complete test data including chart-type test images and question-answer pairs. In this way, a comprehensive and high-quality benchmark test dataset is constructed based on charts and task types.

[0068] Furthermore, for the generated annotated instruction data (i.e., test data), it is necessary to conduct quality inspection of the instruction data to ensure the accuracy, completeness, and consistency of the data. Since in the process of instruction data annotation, the generation of questions and answers are automatically generated using a large model, the large model may have problems such as mismatch between questions and image content, inconsistency between questions and answers, and poor quality of answers and questions when generating answers and questions. Therefore, it is necessary to conduct quality inspection of instruction data and remove unqualified data to ensure the accuracy and consistency of instruction data.

[0069] S2: Access the large model to be evaluated. S2 includes:

[0070] S201: Obtain the large model to be evaluated by using an API interface access method or a model weight access method.

[0071] By accessing through the API interface, the evaluator does not need to understand the internal implementation or complex configuration of the model, and only needs to call the interface to complete the interaction with the model. In addition, through API access, the model and inference code can be automatically updated and optimized by the provider, and the evaluator does not need to manually update or replace the local model, ensuring that the latest version is always used.

[0072] By accessing the model weights, the evaluator can use the same inference code to implement simultaneous evaluation of multiple models. In practical applications, model weight access is a common requirement. Through efficient model access methods, the evaluator can switch different models according to actual needs without completely changing the existing evaluation system architecture. In addition, through model access, multiple large models to be evaluated can be loaded locally, so that the consumption of system resources during the test of each large model to be evaluated can be more clearly obtained, which makes it easier to achieve load balancing between models and avoid system performance degradation due to overload of a certain model. Especially in high-concurrency scenarios, the reasonable allocation of tasks to different models can effectively disperse the computing pressure and improve the system's responsiveness.

[0073] S3: Evaluate and test the large model to be evaluated, including automated evaluation sets and manual evaluation.

[0074] S3 includes evaluation methods for low-level tasks. Specifically, the categories of low-level tasks include reasoning, anomalies, distribution, correlation, range determination, sorting, filtering, data retrieval, extreme values, and clustering. The following steps are specific steps for evaluating low-level tasks.

[0075] S301: Obtain the fill-in-the-blank test data set, selection test data set and judgment test data set corresponding to each type of low-level task. Each chart test image in the fill-in-the-blank test data set corresponds to a set of fill-in-the-blank type question-answer pairs. Each chart test image in the selection test data set corresponds to a set of selection type question-answer pairs. Each chart test image in the judgment test data set corresponds to a set of judgment type question-answer pairs. Low-level tasks focus on specific, detail-oriented queries, seek or compare precise data points in charts, and involve question-answering tasks that involve direct factual information retrieval. The low-level task test data corresponding to each type of chart includes three types of questions: fill-in-the-blank, judgment, and selection.

[0076] S302: Input each chart test image in the fill-in-the-blank test data set and the question information in the corresponding question-answer pair into the large model to be evaluated, so as to obtain the fill-in-the-blank model output result corresponding to each chart test image.

[0077] S303: Input the fill-in-the-blank model output results corresponding to each chart-type test image and the answer information in the corresponding question-answer pair into GPT4o for matching, so as to generate accuracy information of the fill-in-the-blank model output results corresponding to each chart-type test image.

[0078] The evaluation of low-level tasks requires that the model response can give accurate judgments or values. However, due to the lack of simplicity in the output of large multimodal models, precise character matching may not be able to accurately match the core fields in the answer, and thus is insufficient for evaluation. In order to avoid the interference of unnecessary information caused by the diverse expressions in the output results of fill-in-the-blank models, GPT4o is used as a referee model to evaluate the consistency between the model output results and the annotated answer information. By presetting the tolerance level, when the answer falls within the specified range, it will be considered correct. This allows for a more accurate judgment of whether the model output is accurate.

[0079] Specifically, you can also set corresponding prompt instruction information for the GPT4o model to standardize and improve the effectiveness of the model output information. For example, the prompt instruction information here can be [“You are a professional content analysis expert who can parse the answer-related content from the text. Please note the following points: 1. The parsing result should be the content that appears in the text. 2. The parsing result must be complete. 3. Only the complete answer after parsing is output, which should not contain other characters, punctuation marks, etc.”].

[0080] S304: Input each chart-type test image and the question information in the question-answer pair in the judgment-type test data set into the large model to be evaluated to obtain the judgment-type model output result corresponding to each chart-type test image. The question information in the question-answer pair in the judgment-type test data set includes prompt information for the question body text and the answer information can only be positive or negative information.

[0081] For the evaluation of judgment questions, if a vague or neutral numerical answer is given, it is equivalent to the model not giving a clear response, which has a certain impact on the accuracy test. Therefore, the present invention introduces a two-sided judgment design for each category of a given chart, explicitly requiring the multimodal large model to give a positive or negative answer to the judgment question, without any ambiguous expression.

[0082] Specifically, a prompt information indicating that the answer information can only be positive or negative information can be added to the question information of the judgment test data set. For example, the question information in the question-answer pair in the judgment test data set may include questions: "Let's answer the following questions one by one: 1. What type of chart is this? What is the label of the x-axis? 3. What is the data label of each element? 4. Is the length of the bar chart of "Thursday" abnormal?" and, note: "You only need to answer "yes" or "no"".

[0083] Or set a separate prompt command information for the large model to be evaluated. For example, the prompt command information can be [Commands such as: "You are a professional text content analysis expert who can choose the most appropriate answer from the text content. Please note the following points: 1. The answer selected should be the most relevant one among the options, and cannot be a label other than the option label. 2. For judgment questions, the specific results should be clearly given, and labels such as [uncertain, neutral, medium, neutral] should not appear."]

[0084] S305: Input each chart-type test image in the selection-type test data set and the question information in the question-answer pair into the large model to be evaluated, so as to obtain the selection-type model output result corresponding to each chart-type test image.

[0085] S306: Use regular expressions to obtain the selection model output result or the judgment model output result corresponding to each chart test image and match it with the answer information in the corresponding question and answer pair to generate accuracy information of the selection model output result or the judgment model output result corresponding to each chart test image.

[0086] Since the large model to be evaluated is limited in output simplicity when responding to selection or judgment questions, there is no need to use an additional language model to determine whether the output is a specific answer. Regular expression matching is used to determine whether the model's response is correct or not.

[0087] S307: The accuracy information of all the fill-in-the-blank model output results, selection model output results and judgment model output results corresponding to each type of low-level task is counted to generate the execution evaluation information of the large model to be evaluated for each type of low-level task.

[0088] Since there are multiple test data for each type of task, we can obtain multiple judgment information on the model output results, and finally we can evaluate the large model more objectively and accurately through statistical analysis.

[0089] Furthermore, S3 also includes evaluation methods for high-level tasks. Specifically, the categories of high-level tasks include chart classification, opinion suggestions, chart redrawing, chart conversion, chart summary, chart filling assistance, text generation, assisted creation, multi-chart question and answer, multi-round dialogue on charts, chart OCR recognition and information positioning. The following steps are the specific steps for evaluating high-level tasks.

[0090] S308: Obtain a high-level test data set corresponding to each type of high-level task. Each chart-type test image in the high-level test data set corresponds to a set of high-level question-answer pairs. High-level tasks are tasks that focus on understanding the overall trend or pattern of the chart or context summary.

[0091] S309: Input each chart-type test image in the high-level test data set and the question information in the corresponding high-level question-answer pair into the large model to be evaluated to obtain the high-level model output result corresponding to each chart-type test image.

[0092] S310: The high-level model output results corresponding to each chart-type test image and the answer information in the corresponding high-level question-answer pair are input into GPT4o for evaluation and scoring to generate the accuracy information of the high-level model output results corresponding to each chart-type test image. GPT4o evaluates and scores from the dimensions of the fluency of the generated answers, the ability to follow instructions, the length of the generated text, and the completeness of the results.

[0093] For high-level tasks, since the answers in high-level tasks are more subjective and lack fixed standard answers, it is necessary to evaluate the effect of the model to be evaluated by designing a referee model (i.e., GPT4o). In order to improve the evaluation accuracy of the referee model, the referee model is required to score the generated answers by setting different prompt instruction information according to the differences in each type of task. For example, the prompt instruction information can be [“You are a professional chart question and answer model evaluation expert. You need to evaluate the quality of the model’s answers in detail from different dimensions based on **charts**, **questions**, **human reference answers**, and **model answers**. * It is required to output the evaluation quality score, and the quality score corresponds to an integer value between [0-5]. The larger the score, the better the quality of the generated text, and the lower the score, the worse the quality of the generated text. ”]. The scoring range in this embodiment is [0,5], and the larger the score, the better the quality of the generated answer. The scoring criteria are: scoring according to the fluency of the generated answer, the ability to follow instructions, the length of the generated text, and the completeness of the results.

[0094] S311: Collect statistics on the accuracy information of all high-level model output results corresponding to each type of high-level task, and generate execution evaluation information of the large model to be evaluated for each type of high-level task.

[0095] S312: Generate evaluation information of the large model to be evaluated based on the execution evaluation information of each type of high-level task and the execution evaluation information of each type of low-level task.

[0096] In this embodiment, graph question answering is divided into high-level tasks and low-level tasks, and different evaluation methods are designed according to the task types, which can meet and comply with the evaluation requirements of actual tasks. This benchmarking method has good generalization.

[0097] Furthermore, the evaluation of the large model to be evaluated may also include manual evaluation.

[0098] Specifically, the manual evaluation method mainly evaluates the quality and accuracy of the model's output manually to evaluate its performance in the graph question answering task. This evaluation method is usually used to supplement the automated evaluation to ensure that the model's performance in complex, subjective or ambiguous scenarios can be reasonably evaluated.

[0099] The manual evaluation in the present invention adopts a direct evaluation method, and the manual scoring is performed by analyzing the quality of the text generated by the model. The scoring range is [0,5]. The larger the score, the better the quality of the generated text. The scoring standard is: 0 points for the reasoning process is wrong and the answer is wrong; 1 point for the reasoning process is accurate, the answer is wrong, and the instructions are not followed; 2 points for the reasoning process is wrong or the instructions are not followed, and the answer is accurate; 3 points for the answer is correct, the reasoning process is correct but the content of the reasoning process is incomplete, the text sentences are not fluent, the text description is too brief, and the answer does not follow the instructions; 4 points for the answer is correct, the reasoning process is correct and the content of the reasoning process is complete, the text sentences are fluent, and the instructions are generally followed; 5 points for the answer is correct, the reasoning process is correct and the content of the reasoning process is complete, the text sentences are fluent, the instructions are well followed, and the text quality is good.

[0100] The benchmarking method for multimodal graph question-answering large models aims to design and optimize testing methods to comprehensively evaluate the performance of the model when processing graph data and natural language questions. By combining multiple information sources such as images, text, and data, we explore how to effectively measure the model's understanding ability, reasoning ability, and text generation ability, including the accuracy of extraction, analysis, and reasoning inference of data in graphs, as well as accurate answers to natural language questions. Therefore, the present invention attempts to introduce a novel and comprehensive task evaluation method and test data set to evaluate the performance of multimodal large models on graph question-answering tasks, providing strong technical support for the performance improvement of multimodal large models in the field of graph question-answering.

[0101] This paper mainly relates to the benchmarking technology of multimodal graph question-answering large models, which is used to evaluate the performance of the model in graph understanding and question-answering tasks through a series of standardized test data sets and task testing methods. These benchmarks involve a variety of graph types (such as bar charts, line charts, pie charts, scatter plots, etc.) and related natural language questions. The ability of the test model lies in how to extract information from the graph, understand the context, make inferences, and finally generate accurate answers.

[0102] By combining the low-level tasks, high-level tasks and manual evaluation of the large model to be evaluated, a more comprehensive and accurate evaluation report on the processing capabilities of various tasks of the large model to be evaluated can be generated. This not only helps researchers reveal the advantages and limitations of the model, but also provides a theoretical basis and practical guidance for the further optimization of multimodal AI systems.

[0103] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0104] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0105] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0106] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0107] The electronic device according to this embodiment of the present invention is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0108] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).

[0109] The storage stores program codes, which can be executed by the processor, so that the processor executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0110] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0111] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0112] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0113] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0114] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0115] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0116] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0119] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0120] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0121] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0122] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An evaluation and testing method for a multimodal graph question-answering large model, characterized in that: The method comprises the following steps: Obtain a fill-in-the-blank test data set, a selection test data set, and a judgment test data set corresponding to each type of low-level task; each chart test image in the fill-in-the-blank test data set corresponds to a set of fill-in-the-blank type question-answer pairs; each chart test image in the selection test data set corresponds to a set of selection type question-answer pairs; each chart test image in the judgment test data set corresponds to a set of judgment type question-answer pairs; low-level tasks are question-answering tasks that focus on specific, detail-oriented queries, seek or compare precise data points in charts, and involve direct factual information retrieval; Input each chart test image and the question information in the corresponding question-answer pair in the fill-in-the-blank test data set into the large model to be evaluated, so as to obtain the fill-in-the-blank model output result corresponding to each chart test image; The output results of the fill-in-the-blank model corresponding to each chart test image and the answer information in the corresponding question-answer pair are input into GPT4o for matching to generate the accuracy information of the output results of the fill-in-the-blank model corresponding to each chart test image; Input each chart test image and question information in the question-answer pair in the judgment test data set into the large model to be evaluated to obtain the judgment model output result corresponding to each chart test image; The question information in the question-answer pair in the judgment test data set includes prompt information for the question body text and the answer information can only be positive or negative information; Input each chart test image and question information in the question-answer pair in the selection test data set into the large model to be evaluated to obtain the selection model output result corresponding to each chart test image; Use regular expressions to obtain the selection model output result or the judgment model output result corresponding to each chart test image and match it with the answer information in the corresponding question and answer pair to generate accuracy information of the selection model output result or the judgment model output result corresponding to each chart test image; The accuracy information of all the fill-in-the-blank model output results, selection model output results and judgment model output results corresponding to each type of low-level task is statistically analyzed to generate the execution evaluation information of the large model to be evaluated for each type of low-level task.

2. The method according to claim 1, characterized in that: The method further comprises: Obtain a high-level test data set corresponding to each type of high-level task; each chart-type test image in the high-level test data set corresponds to a set of high-level question-answer pairs; the high-level task is a task that focuses on understanding the overall trend or pattern of the chart or the context summary; Input each chart-type test image in the high-level test data set and the question information in the corresponding high-level question-answer pair into the large model to be evaluated, so as to obtain the high-level model output result corresponding to each chart-type test image; The high-level model output results corresponding to each chart-type test image and the answer information in the corresponding high-level question-answer pair are input into GPT4o for evaluation and scoring to generate the accuracy information of the high-level model output results corresponding to each chart-type test image; GPT4o evaluates and scores from the dimensions of the fluency of the generated answers, the ability to follow instructions, the length of the generated text, and the completeness of the results; The accuracy information of all high-level model output results corresponding to each type of high-level task is counted to generate the execution evaluation information of the large model to be evaluated for each type of high-level task; According to the execution evaluation information of each type of high-level task and the execution evaluation information of each type of low-level task, the evaluation information of the large model to be evaluated is generated.

3. The method according to claim 1, characterized in that Get the test data set corresponding to each type of high-level task or low-level task, including: Use the self-instruct model to generate multiple seed questions corresponding to each chart class test image; Use the Qwen2.5-0.5b model to score each seed question, and take the seed question with the highest score as the intermediate question; Use GPT4o to rewrite the intermediate questions according to the task type, and generate multiple target questions corresponding to each chart-type test image; The initial annotated test data consisting of each target question and the corresponding chart-type test image is input into the Qwen2-VL-72b model to generate the answer information corresponding to each initial annotated test data to obtain the target test data.

4. The method according to claim 3, characterized in that: Before using the self-instruct model to generate a plurality of seed questions corresponding to each chart class test image, the method further includes: Convert the acquired initial chart test images into RGB format to generate a first test image set; Removing low-quality pictures from the first test picture set to generate a second test picture set; the low-quality pictures include low-resolution, over-compressed, blank or invalid pictures, and pictures with repeated identical content; Median filtering is used to remove image noise in the second test image set to generate a chart-type test image set in the test data set.

5. The method according to claim 1, characterized in that Before inputting each chart-type test image in the fill-in-the-blank test data set and the question information in the corresponding question-answer pair into the large model to be evaluated to obtain the fill-in-the-blank model output result corresponding to each chart-type test image, the method further includes: Use the API interface access method or the model weight access method to obtain the large model to be evaluated.

6. The method according to claim 1, characterized in that The types of chart test images include stacked bar charts, complex line charts, scatter charts, pie charts, regular line charts, grouped bar charts, regular bar charts, three-dimensional bar charts, bubble charts and mixed charts.

7. The method according to claim 1, characterized in that The categories of low-level tasks include reasoning, anomalies, distribution, correlation, ranging, sorting, filtering, data retrieval, extrema, and clustering.

8. The method according to claim 2, characterized in that: Categories of high-level tasks include chart classification, opinion suggestions, chart redrawing, chart conversion, chart summary, chart filling assistance, text generation, assisted creation, multi-chart Q&A, multi-round chart dialogue, chart OCR recognition and information positioning.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements an evaluation and testing method for a multimodal graphical question-and-answer large model as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the evaluation and testing method of the multimodal graphical question and answer large model as described in any one of claims 1 to 8.

Images