Comprehensive evaluation method and device for chart question and answer model and medium

Abstract

The invention provides a comprehensive evaluation method and device for a chart question and answer model and a medium, and relates to the technical field of data processing.The method comprises the steps that a total set of initial chart data list sets is obtained according to a preset chart question and answer task type list and a target chart type list, obtaining a question text list corresponding to the initial chart data and a standard answer text corresponding to the question text according to the initial chart data, a definition text of a preset chart question and answer task type corresponding to the initial chart data and a target multi-mode large model, and obtaining a multi-thread processing mode according to a multi-thread processing mode, a preset configuration file and a chart question and answer model. And obtaining a target answer text corresponding to the question text in parallel, and obtaining an artificial score and an automatic score corresponding to the chart question and answer model according to the standard answer text and the target answer text corresponding to the question text to evaluate the chart question and answer model. The accuracy of the output result of the chart question and answer model under the condition of multiple chart question and answer task types can be evaluated, and diversified requirements of users can be met.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a comprehensive evaluation method, device and medium for a chart question-answering model. Background Art

[0002] With the rapid development of artificial intelligence technology, graph question answering has gradually become one of the important research directions in the field of natural language processing and computer vision. In information-intensive fields, such as business analysis, education, and scientific research, graphs have become an important tool for conveying complex data patterns and key information due to their intuitive and efficient characteristics. However, graph question answering models need to process image data and text information at the same time. At present, graph question answering models still face challenges in the deep integration of vision and language, resulting in insufficient cross-modal understanding capabilities and inability to effectively answer users' diverse questions. Therefore, it is necessary to comprehensively obtain the comprehensive score of the graph question answering model to evaluate the accuracy and consistency of the output results of the graph question answering model. Most of the existing technologies rely on graph question answering evaluations and existing evaluation systems to evaluate graph question answering systems and obtain the scores of graph question answering models.

[0003] However, the above method also has the following technical problems:

[0004] The task types covered by existing graph question and answer evaluation datasets are relatively single, especially the lack of Chinese graph data, and the existing evaluation system can only evaluate question and answer tasks of a single graph question and answer task type, such as extracting specific values, comparing data sizes, identifying data trends, and other question and answer task types. Therefore, the score obtained by the above method can only be used to evaluate the accuracy of the output results of the graph question and answer model in the case of a single graph question and answer task type, and cannot evaluate the accuracy of the output results of the graph question and answer model in the case of multiple graph question and answer task types, and cannot meet the diverse needs of users. Summary of the invention

[0005] In view of the above technical problems, the technical solution adopted by the present invention is:

[0006] According to a first aspect of the present invention, a comprehensive evaluation method for a graph question answering model is provided, the method comprising the following steps:

[0007] S1. According to the preset chart question and answer task type list and the target chart type list, obtain the total set of initial chart data list sets, wherein the preset chart question and answer task type list includes several preset chart question and answer task types corresponding to low-level question and answer tasks and several preset chart question and answer task types corresponding to high-level question and answer tasks, the target chart type list includes several target chart types, the total set of initial chart data list sets includes an initial chart data list set corresponding to each preset chart question and answer task type, the initial chart data list set includes an initial chart data list corresponding to each target chart type, the initial chart data list includes several initial chart data, the low-level question and answer task is a question and answer task with a fixed and unique answer, and the high-level question and answer task is a question and answer task that does not have a fixed or unique answer.

[0008] S2. Input the initial chart data and the definition text of the preset chart question-answering task type corresponding to the initial chart data into the target multimodal large model to obtain a question text list corresponding to the initial chart data, wherein the question text list includes a plurality of question texts.

[0009] S3. Obtain a standard answer text corresponding to the question text according to the initial chart data, the question text corresponding to the initial chart data and the target multimodal large model.

[0010] S4. According to the multi-threaded processing method, the preset configuration file and the chart question and answer model, the target answer text corresponding to each question text is obtained in parallel, wherein the preset configuration file includes each preset chart question and answer task type in the preset chart question and answer task type list and the model interface address corresponding to each preset chart question and answer task type, the model interface address points to a module in the chart question and answer model specifically used to process the chart question and answer task corresponding to the same preset chart question and answer task type as the preset chart question and answer task type corresponding to the model interface address, and the target answer text is the text output by the chart question and answer model.

[0011] S5. According to the standard answer text corresponding to the question text and the target answer text corresponding to the question text, obtain the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model, wherein the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are used to evaluate the chart question and answer model.

[0012] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the aforementioned method.

[0013] According to a third aspect of the present invention, there is provided an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the computer program.

[0014] The present invention has at least the following beneficial effects:

[0015] The present invention provides a comprehensive evaluation method, device and medium for a chart question-and-answer model. The method can obtain a total set of initial chart data list sets according to a preset chart question-and-answer task type list and a target chart type list, wherein the total set of initial chart data list sets includes an initial chart data list set corresponding to each preset chart question-and-answer task type, the initial chart data list set includes an initial chart data list corresponding to each target chart type, the initial chart data list includes a plurality of initial chart data, the initial chart data and the definition text of the preset chart question-and-answer task type corresponding to the initial chart data are input into a target multimodal large model to obtain a list of question texts corresponding to the initial chart data, and the standard chart data corresponding to the question texts are obtained according to the initial chart data, the question texts corresponding to the initial chart data and the target multimodal large model. Answer text, according to the multi-threaded processing method, the preset configuration file and the chart question and answer model, the target answer text corresponding to each question text is obtained in parallel, and the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are obtained according to the standard answer text corresponding to the question text and the target answer text corresponding to the question text. The manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are used to evaluate the chart question and answer model. It can be seen that the present invention is based on the standard answer text and the target answer text corresponding to the question text corresponding to the preset chart question and answer task type, obtains the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model, and is used to evaluate the chart question and answer model. It can be evaluated the accuracy of the output results of the chart question and answer model in various chart question and answer task types, and can meet the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a comprehensive evaluation method for a graphical question-answering model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar tasks, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged in appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] An embodiment of the present invention provides a comprehensive evaluation method for a graph question-answering model, the method comprising the following steps: Figure 1 As shown:

[0021] S1. According to the preset chart question and answer task type list and the target chart type list, obtain the total set of initial chart data list sets, wherein the preset chart question and answer task type list includes several preset chart question and answer task types corresponding to low-level question and answer tasks and several preset chart question and answer task types corresponding to high-level question and answer tasks, the target chart type list includes several target chart types, the total set of initial chart data list sets includes an initial chart data list set corresponding to each preset chart question and answer task type, the initial chart data list set includes an initial chart data list corresponding to each target chart type, and the initial chart data list includes several initial chart data.

[0022] Specifically, a low-level question-answering task is a question-answering task with a fixed and unique answer; it can be understood as a question-answering task with relatively direct question design and clear result output.

[0023] Furthermore, the preset chart question and answer task types corresponding to the low-level question and answer tasks include at least reasoning tasks, anomaly detection tasks, distribution analysis tasks, correlation analysis tasks, range determination tasks, sorting tasks, filtering tasks, data retrieval tasks, extreme value tasks and clustering tasks, and the above are only some of the examples given, and there may be other preset chart question and answer task types.

[0024] Specifically, high-level question-answering tasks are question-answering tasks that do not have fixed or unique answers; they can be understood as being more complex than low-level question-answering tasks, usually involving comprehensive analysis of multimodal data (such as text, images, charts, etc.), and requiring flexible generation of answers.

[0025] Furthermore, the preset chart question and answer task types corresponding to the advanced question and answer tasks include at least chart classification tasks, opinion suggestion tasks, chart redrawing tasks, chart conversion tasks, chart summary tasks, auxiliary chart filling tasks, text generation tasks, auxiliary creation tasks, multi-chart question and answer tasks, chart multi-round dialogue tasks, chart OCR recognition tasks and information positioning tasks, and the above are only some of the examples given, and there may be other preset chart question and answer task types.

[0026] Specifically, before step S1, the following steps S01-S04 are also included:

[0027] S01. Obtain a total set of open source data sets and a private data set, wherein the total set of open source data sets includes several open source data sets, the open source data sets include several chart data, and the private data sets include several preset Chinese chart data, wherein those skilled in the art know that the preset Chinese chart data is Chinese chart data pre-set by those skilled in the art according to actual needs, and any open source data set in the prior art belongs to the protection scope of the present invention, for example: DVQA, Chart2Text, ChartAssistant, ChartInsights, ChartLLama, ChartXiv, FigureQA, which will not be repeated here.

[0028] S02. Insert the private data set into the total set of open source data sets to obtain the total set of original chart data sets, where the total set of original chart data sets includes several original chart data sets, and the original chart data sets include several original chart data.

[0029] S03. Input the total set of original chart data sets into a chart classification model to obtain an initial chart type corresponding to each original chart data. The chart classification model is a machine learning or deep learning model used to identify and classify different types of charts (such as bar charts, line charts, pie charts, etc.).

[0030] S04. Manually screen the initial chart types corresponding to all original chart data in all original chart data sets to obtain a target chart type list.

[0031] Specifically, the target chart types include at least a conventional line chart, a complex line chart, a conventional bar chart, a grouped bar chart, a stacked bar chart, a pie chart and a scatter chart, and the above are only some of the examples given, and there may be other target chart types.

[0032] Furthermore, a conventional line chart contains a single line for describing the trend of a single variable; a complex line chart contains multiple lines for comparing or analyzing the correlation between multiple variables.

[0033] In a specific embodiment, the chart data is information data used to generate and display charts, including numerical data, legends, annotations, time series, etc.

[0034] In a specific embodiment, the chart data is an image that can fully present the chart.

[0035] In a specific embodiment, the chart data includes information data used to generate and display the chart and an image that can fully present the chart.

[0036] Through the above steps, the private dataset is inserted into the total set of open source datasets to obtain the total set of original chart datasets. The private dataset includes several preset Chinese chart data. The preset Chinese chart data is Chinese chart data pre-set by technical personnel in this field according to actual needs. It can be understood as Chinese chart data obtained after preprocessing and expanding the collected and sorted Chinese chart data covering various types such as bar charts, line charts, pie charts, etc. The total set of original chart datasets is input into the chart classification model to obtain the initial chart type corresponding to each original chart data. The initial chart types corresponding to all original chart data in all original chart datasets are manually screened to obtain a list of target chart types, which not only enriches the language and task diversity of the total set of original chart datasets, but also improves the accuracy of obtaining target chart types, thereby enhancing the applicability and comprehensiveness of the comprehensive evaluation method of the chart question and answer model.

[0037] Specifically, step S1 includes the following steps S11-S12:

[0038] S11. When the initial chart type corresponding to the original chart data in the original chart data set is the same as the target chart type, the original chart data is used as the intermediate chart data corresponding to the target chart type in the original chart data set corresponding to the original chart data, so as to obtain a list of intermediate chart data corresponding to the target chart type in the original chart data set, wherein the list of intermediate chart data includes a plurality of intermediate chart data; it can be understood that: the total set A of the original chart data set = {A1, A2, ..., A i , ..., A m}, A i ={A i1 , A i2 , ..., A ij , ..., A in(i)}, A iis the i-th original chart data set, i ranges from 1 to m, m is the number of original chart data sets, A ij A i The jth original chart data in A, where j ranges from 1 to n(i), and n(i) is i The number of original chart data in the target chart type list B = {B1, B2, ..., B e , ..., B f}, B e is the e-th target chart type, where e ranges from 1 to n, and n is the number of target chart types. 0 ij =B e When A ij As B e In A i The corresponding intermediate chart data in B e In A i The corresponding intermediate chart data list C ei ={C ei1 , C ei2 , ..., C eir , ..., C eis(ei)}, A 0 ij A ij The corresponding initial chart type, C eir For B e In A i The corresponding r-th intermediate chart data in the table, r ranges from 1 to s(ei), s(ei) is C ei The number of intermediate chart data in .

[0039] S12, obtaining a number of intermediate chart data from the intermediate chart data list corresponding to the target chart type in each original chart data set, and forming an initial chart data list corresponding to the target chart type in the initial chart data list set corresponding to the preset chart question-answering task type; it can be understood as: from C e1 , C e2 , ..., C ei , ..., C em In each intermediate chart data list in the , several intermediate chart data are obtained to form an initial chart data list set corresponding to a preset chart question-answering task type. e The corresponding initial chart data list.

[0040] Specifically, the total number of initial chart data corresponding to the preset chart question and answer task type is not less than 40 and not more than 60, and the total number of initial chart data corresponding to all preset chart question and answer task types is the same, wherein the total number of initial chart data corresponding to the preset chart question and answer task type is the sum of the numbers of initial chart data in all initial chart data lists in the initial chart data list set corresponding to the preset chart question and answer task type.

[0041] In a specific embodiment, a method of acquiring a plurality of intermediate chart data from the intermediate chart data list corresponding to the target chart type in each original chart data set is manual acquisition.

[0042] In a specific embodiment, a method of acquiring a plurality of intermediate chart data from the intermediate chart data list corresponding to the target chart type in each original chart data set is random acquisition.

[0043] Through the above steps, when the initial chart type corresponding to the original chart data in the original chart data set is the same as the target chart type, the original chart data is used as the intermediate chart data corresponding to the target chart type in the original chart data set corresponding to the original chart data, so as to obtain a list of intermediate chart data corresponding to the target chart type in the original chart data set, and a number of intermediate chart data are obtained from the intermediate chart data list corresponding to the target chart type in each original chart data set to form an initial chart data list corresponding to the target chart type in a set of initial chart data lists corresponding to a preset chart question and answer task type, and based on the standard answer text and target answer text corresponding to the question text corresponding to the preset chart question and answer task type, the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are obtained for evaluating the chart question and answer model, which can evaluate the accuracy of the output results of the chart question and answer model in the case of multiple chart question and answer task types, and can meet the diverse needs of users.

[0044] S2. Input the initial chart data and the definition text of the preset chart question and answer task type corresponding to the initial chart data into the target multimodal big model to obtain a question text list corresponding to the initial chart data, wherein the question text list includes several question texts, wherein those skilled in the art know that the target multimodal big model is a multimodal big model determined by those skilled in the art according to actual needs, and any multimodal big model in the prior art belongs to the protection scope of the present invention, for example: GPT-4o model, which will not be repeated here.

[0045] Specifically, the definition text of the preset chart question and answer task type is a text used to describe the definition of the preset chart question and answer task type.

[0046] Specifically, if the preset chart question and answer task type corresponding to the initial chart data is the preset chart question and answer task type corresponding to the low-level question and answer task, then the question types of the question text corresponding to the initial chart data include direct question and answer type, multiple-choice question type, fill-in-the-blank question type and error correction question type.

[0047] S3. Obtain a standard answer text corresponding to the question text according to the initial chart data, the question text corresponding to the initial chart data and the target multimodal large model.

[0048] Specifically, step S3 includes the following steps S31-S32:

[0049] S31. Input the initial chart data and the question text corresponding to the initial chart data into the target multimodal large model to obtain the intermediate answer text corresponding to the question text.

[0050] S32. Manually annotate the intermediate answer text corresponding to the question text according to the question text to obtain the standard answer text corresponding to the question text.

[0051] Through the above steps, since the question text and intermediate answer text generated by the target multimodal large model may be inaccurate or incomplete, the intermediate answer text corresponding to the question text is manually annotated according to the question text to obtain the standard answer text corresponding to the question text. The question text and intermediate answer text generated by the target multimodal large model are manually reviewed and corrected, and the accuracy and logic of the answer content are optimized to obtain the standard answer text corresponding to the question text, which is conducive to improving the accuracy of obtaining the standard answer text. Based on the standard answer text and target answer text corresponding to the question text corresponding to the preset chart question and answer task type, the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are obtained for evaluating the chart question and answer model. The accuracy of the output results of the chart question and answer model in various chart question and answer task types can be evaluated to meet the diverse needs of users.

[0052] S4. According to the multi-threaded processing method, the preset configuration file and the chart question and answer model, the target answer text corresponding to each question text is obtained in parallel, wherein the preset configuration file includes each preset chart question and answer task type in the preset chart question and answer task type list and the model interface address corresponding to each preset chart question and answer task type, the model interface address points to a module in the chart question and answer model that is specifically used to process the chart question and answer task corresponding to the same chart question and answer task type as the preset chart question and answer task type corresponding to the model interface address, and the target answer text is the text output by the chart question and answer model, which can be understood as the text presenting the output result of the chart question and answer model.

[0053] Specifically, in step S4, when the file format of the output file corresponding to the chart question and answer model is not in Excel format, the file format of the output file is converted into Excel format. The output file corresponding to the chart question and answer model is a file recording the text output by the chart question and answer model, which can be understood as a file recording the output results of the chart question and answer model.

[0054] Through the above steps, according to the multi-threaded processing method, the preset configuration file and the chart question and answer model, the target answer text corresponding to each question text is obtained in parallel, which can greatly shorten the overall running time for obtaining the target answer text, while ensuring the independence and accuracy of obtaining a single target answer text, which is conducive to improving the efficiency of obtaining the target answer text and saving resources.

[0055] S5. According to the standard answer text corresponding to the question text and the target answer text corresponding to the question text, obtain the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model, wherein the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are used to evaluate the chart question and answer model.

[0056] Specifically, step S5 includes the following steps S51-S54:

[0057] S51. Manually process the target answer text corresponding to the question text according to the standard answer text corresponding to the question text to obtain a first evaluation score of the target answer text.

[0058] S52. Take the average of the first evaluation scores of the target answer texts corresponding to all question texts corresponding to the preset chart question and answer task type as the first average score corresponding to the preset chart question and answer task type, and add the first average scores corresponding to all preset chart question and answer task types as the manual score corresponding to the chart question and answer model.

[0059] S53. Input the scoring instructions corresponding to the question text, the standard answer text corresponding to the question text, and the target answer text corresponding to the question text into the referee model to obtain a second evaluation score corresponding to the target answer text, wherein the visual language model is used as the referee model. Those skilled in the art know that any visual language model in the prior art belongs to the protection scope of the present invention, for example: Qwen2_vl_72B model, which will not be repeated here.

[0060] Specifically, the referee model refers to a machine learning or deep learning model used for judging or scoring in an automated evaluation system.

[0061] Specifically, when the referee model receives a scoring instruction corresponding to the question text, it determines a second evaluation score corresponding to the target answer text corresponding to the question text according to a preset scoring rule and a standard answer text corresponding to the question text.

[0062] Specifically, when the preset chart question and answer task type corresponding to the question text is the preset chart question and answer task type corresponding to the low-level question and answer task, the first preset scoring instruction is used as the scoring instruction corresponding to the question text, and the first preset scoring instruction is an instruction for determining the second evaluation score of the target answer text corresponding to the question text corresponding to the low-level question and answer task.

[0063] Further, when the preset chart question and answer task type corresponding to the question text is the preset chart question and answer task type corresponding to the advanced question and answer task, the second preset scoring instruction corresponding to the preset chart question and answer task type is used as the scoring instruction corresponding to the question text, and the second preset scoring instruction corresponding to the preset chart question and answer task type is an instruction for determining the second evaluation score of the target answer text corresponding to the question text corresponding to the preset chart question and answer task type.

[0064] Specifically, in the preset scoring rules, when the preset chart question and answer task type corresponding to the question text is the preset chart question and answer task type corresponding to the low-level question and answer task, determine whether the target answer text corresponding to the question text is the same as the standard answer text corresponding to the question text; if the target answer text corresponding to the question text is the same as the standard answer text corresponding to the question text, determine that the second evaluation score corresponding to the target answer text corresponding to the question text is 1; otherwise, determine that the second evaluation score corresponding to the target answer text corresponding to the question text is 0.

[0065] Further, when the preset chart question and answer task type corresponding to the question text is the preset chart question and answer task type corresponding to the advanced question and answer task, the second evaluation score corresponding to the target answer text corresponding to the question text is determined based on the semantic similarity between the standard answer text corresponding to the question text and the target answer text corresponding to the question text. When the semantic similarity is greater than the preset similarity, the second evaluation score corresponding to the target answer text corresponding to the question text is determined to be 1; otherwise, the second evaluation score corresponding to the target answer text corresponding to the question text is determined to be 0.

[0066] In a specific embodiment, when the preset chart question and answer task type corresponding to the question text is the preset chart question and answer task type corresponding to the advanced question and answer task, the referee model also outputs an explanatory text corresponding to the second evaluation score. When the second evaluation score is 1, the explanatory text corresponding to the second evaluation score is a text that represents that the answer presented by the target answer text corresponding to the question text is clear in expression, fluent in logic, and the chart question and answer model can complete the question and answer task. When the second evaluation score is 0, the explanatory text corresponding to the second evaluation score is a text that represents that the answer presented by the target answer text is incomplete, unclear in expression, not fluent in logic, lacks a reasonable thinking process, and the chart question and answer model cannot complete the question and answer task.

[0067] S54. The average of the second evaluation scores of the target answer texts corresponding to all question texts corresponding to the preset chart question and answer task type is taken as the second average score corresponding to the preset chart question and answer task type, and the sum of the second average scores corresponding to all preset chart question and answer task types is taken as the automatic score corresponding to the chart question and answer model.

[0068] Through the above steps, the target answer text corresponding to the question text is manually processed according to the standard answer text corresponding to the question text to obtain a first evaluation score of the target answer text, and further obtain a manual score corresponding to the chart question and answer model; the standard answer text corresponding to the question text and the target answer text corresponding to the question text are input into the referee model to obtain a second evaluation score corresponding to the target answer text, and further obtain an automatic score corresponding to the chart question and answer model. The manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are obtained to evaluate the chart question and answer model, which can evaluate the accuracy of the output results of the chart question and answer model in various chart question and answer task types, and can meet the diverse needs of users.

[0069] In a specific embodiment, after step S5, the following steps are also included:

[0070] When the score difference corresponding to the chart question and answer model is less than the preset score difference, the automatic score corresponding to the chart question and answer model is used as the comprehensive score corresponding to the chart question and answer model, which is used to evaluate the chart question and answer model, wherein the score difference corresponding to the chart question and answer model is the absolute value of the difference between the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model.

[0071] When the score difference corresponding to the graph question and answer model is not less than the preset score difference, the manual score corresponding to the graph question and answer model is used as the comprehensive score corresponding to the graph question and answer model to evaluate the graph question and answer model.

[0072] Specifically, the larger the comprehensive score, the higher the accuracy of the output results of the graph question answering model.

[0073] Through the above steps, when the score difference corresponding to the chart question and answer model is not less than the preset score difference, it means that the difference between the manual score obtained by manual processing and the automatic score obtained based on the referee model is large, and the result of the automatic evaluation is inconsistent with the result of the manual evaluation. The manual score corresponding to the chart question and answer model is used as the comprehensive score corresponding to the chart question and answer model, which is used to evaluate the chart question and answer model. When the score difference corresponding to the chart question and answer model is less than the preset score difference, it means that the difference between the manual score obtained based on manual processing and the automatic score obtained based on the referee model is small, and the result of the automatic evaluation is highly consistent with the result of the manual evaluation. The automatic score corresponding to the chart question and answer model is used as the comprehensive score corresponding to the chart question and answer model, which is used to evaluate the chart question and answer model, which can comprehensively and standardizedly evaluate the model performance of the chart question and answer model.

[0074] In a specific embodiment, when the score difference corresponding to the graph question and answer model is not less than the preset score difference, the scoring instructions input into the referee model are optimized and updated so that the score difference corresponding to the graph question and answer model is less than the preset score difference, and the automatic score corresponding to the graph question and answer model is used as the comprehensive score corresponding to the graph question and answer model; it can be understood as: the first preset scoring instructions and the second preset scoring instructions are optimized and updated so that the corresponding scoring instructions of the question text are optimized and updated, and the optimized and updated scoring instructions corresponding to the question text, the standard answer text corresponding to the question text, and the target answer text corresponding to the question text are input into the referee model to obtain the second evaluation score corresponding to the target answer text, and further obtain the automatic score corresponding to the graph question and answer model so that the score difference corresponding to the graph question and answer model is less than the preset score difference. At this time, the automatic score corresponding to the graph question and answer model is used as the comprehensive score corresponding to the graph question and answer model.

[0075] In a specific embodiment, when the score difference corresponding to the graph question and answer model is less than a preset score difference, when the comprehensive evaluation method of the graph question and answer model is used again, there is no need to obtain the manual score corresponding to the graph question and answer model, but only the automatic score corresponding to the graph question and answer model needs to be obtained, and the automatic score corresponding to the graph question and answer model is used as the comprehensive score corresponding to the graph question and answer model for evaluating the graph question and answer model.

[0076] Through the above steps, when the score difference corresponding to the chart question and answer model is not less than the preset score difference, the scoring instructions input into the referee model are optimized and updated so that the score difference corresponding to the chart question and answer model is less than the preset score difference, and the automatic score corresponding to the chart question and answer model is used as the comprehensive score corresponding to the chart question and answer model, so that the results of the automatic evaluation and the results of the manual evaluation reach a high degree of consistency. When the results of the automatic evaluation and the results of the manual evaluation reach a high degree of consistency, it indicates that the automatic evaluation and the manual evaluation results are highly consistent, and the ability to rely entirely on automatic evaluation is possessed. Therefore, when the comprehensive evaluation method of the chart question and answer model is used again, there is no need to obtain the manual score corresponding to the chart question and answer model, but only the automatic score corresponding to the chart question and answer model, and the automatic score corresponding to the chart question and answer model is used as the comprehensive score corresponding to the chart question and answer model for evaluating the chart question and answer model, which can significantly reduce the investment in manual evaluation and reduce the evaluation cost.

[0077] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store a computer program related to a method in a method embodiment, and the computer program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0078] An embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the above embodiment when executing the computer program.

[0079] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0080] The present invention provides a comprehensive evaluation method, device and medium for a chart question-and-answer model. The method can obtain a total set of initial chart data list sets according to a preset chart question-and-answer task type list and a target chart type list, wherein the total set of initial chart data list sets includes an initial chart data list set corresponding to each preset chart question-and-answer task type, the initial chart data list set includes an initial chart data list corresponding to each target chart type, the initial chart data list includes a plurality of initial chart data, the initial chart data and the definition text of the preset chart question-and-answer task type corresponding to the initial chart data are input into a target multimodal large model to obtain a list of question texts corresponding to the initial chart data, and the standard chart data corresponding to the question texts are obtained according to the initial chart data, the question texts corresponding to the initial chart data and the target multimodal large model. Answer text, according to the multi-threaded processing method, the preset configuration file and the chart question and answer model, the target answer text corresponding to each question text is obtained in parallel, and the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are obtained according to the standard answer text corresponding to the question text and the target answer text corresponding to the question text. The manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are used to evaluate the chart question and answer model. It can be seen that the present invention is based on the standard answer text and the target answer text corresponding to the question text corresponding to the preset chart question and answer task type, obtains the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model, and is used to evaluate the chart question and answer model. It can be evaluated the accuracy of the output results of the chart question and answer model in various chart question and answer task types, and can meet the diverse needs of users.

[0081] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A comprehensive evaluation method for a graph question answering model, characterized in that: The method comprises the following steps: S1. According to the preset chart question and answer task type list and the target chart type list, a total set of initial chart data list sets is obtained, wherein the preset chart question and answer task type list includes several preset chart question and answer task types corresponding to low-level question and answer tasks and several preset chart question and answer task types corresponding to high-level question and answer tasks, the target chart type list includes several target chart types, the total set of initial chart data list sets includes an initial chart data list set corresponding to each preset chart question and answer task type, the initial chart data list set includes an initial chart data list corresponding to each target chart type, the initial chart data list includes several initial chart data, the low-level question and answer task is a question and answer task with a fixed and unique answer, and the high-level question and answer task is a question and answer task without a fixed or unique answer; S2, inputting the initial chart data and the definition text of the preset chart question-answering task type corresponding to the initial chart data into the target multimodal large model to obtain a question text list corresponding to the initial chart data, wherein the question text list includes a plurality of question texts; S3, obtaining a standard answer text corresponding to the question text according to the initial chart data, the question text corresponding to the initial chart data, and the target multimodal large model; S4. According to the multi-threaded processing mode, the preset configuration file and the chart question and answer model, the target answer text corresponding to each question text is obtained in parallel, wherein the preset configuration file includes each preset chart question and answer task type in the preset chart question and answer task type list and the model interface address corresponding to each preset chart question and answer task type, the model interface address points to the chart question and answer model The module is specifically used to process the chart question and answer task corresponding to the same preset chart question and answer task type as the preset chart question and answer task type corresponding to the model interface address in the chart question and answer model, and the target answer text is the text output by the chart question and answer model; S5. According to the standard answer text corresponding to the question text and the target answer text corresponding to the question text, obtain the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model, wherein the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model are used to evaluate the chart question and answer model.

2. The comprehensive evaluation method of the diagram question-answering model according to claim 1, characterized in that: Before step S1, the following steps S01-S04 are also included: S01. Obtain a total set of open source data sets and a private data set, wherein the total set of open source data sets includes several open source data sets, the open source data sets include several chart data, and the private data sets include several preset Chinese chart data; S02, inserting the private data set into the total set of open source data sets to obtain the total set of original chart data sets, where the total set of original chart data sets includes a plurality of original chart data sets, and the original chart data sets include a plurality of original chart data; S03, inputting the total set of original chart data sets into the chart classification model to obtain an initial chart type corresponding to each original chart data; S04. Manually screen the initial chart types corresponding to all original chart data in all original chart data sets to obtain a target chart type list.

3. The comprehensive evaluation method of the diagram question-answering model according to claim 2, characterized in that: Step S1 includes the following steps S11-S12: S11, when the initial chart type corresponding to the original chart data in the original chart data set is the same as the target chart type, the original chart data is used as the intermediate chart data corresponding to the target chart type in the original chart data set corresponding to the original chart data, so as to obtain an intermediate chart data list corresponding to the target chart type in the original chart data set, wherein the intermediate chart data list includes a plurality of intermediate chart data; S12. Obtain a number of intermediate chart data from the intermediate chart data list corresponding to the target chart type in each original chart data set to form an initial chart data list corresponding to the target chart type in an initial chart data list set corresponding to a preset chart question and answer task type.

4. The comprehensive evaluation method of the diagram question answering model according to claim 1, characterized in that: Step S3 includes the following steps S31-S32: S31, inputting the initial chart data and the question text corresponding to the initial chart data into the target multimodal large model to obtain the intermediate answer text corresponding to the question text; S32. Manually annotate the intermediate answer text corresponding to the question text according to the question text to obtain the standard answer text corresponding to the question text.

5. The comprehensive evaluation method of the diagram question-answering model according to claim 1, characterized in that: Step S5 includes the following steps S51-S54: S51, manually processing the target answer text corresponding to the question text according to the standard answer text corresponding to the question text to obtain a first evaluation score of the target answer text; S52, taking the average of the first evaluation scores of the target answer texts corresponding to all the question texts corresponding to the preset chart question and answer task type as the first average score corresponding to the preset chart question and answer task type, and adding the first average scores corresponding to all the preset chart question and answer task types as the manual score corresponding to the chart question and answer model; S53, inputting the scoring instruction corresponding to the question text, the standard answer text corresponding to the question text, and the target answer text corresponding to the question text into the referee model to obtain a second evaluation score corresponding to the target answer text, wherein the visual language model is used as the referee model; S54. The average of the second evaluation scores of the target answer texts corresponding to all question texts corresponding to the preset chart question and answer task type is taken as the second average score corresponding to the preset chart question and answer task type, and the sum of the second average scores corresponding to all preset chart question and answer task types is taken as the automatic score corresponding to the chart question and answer model.

6. The comprehensive evaluation method of the diagram question-answering model according to claim 5, characterized in that: After step S5, the following steps are also included: When the score difference corresponding to the chart question and answer model is less than the preset score difference, the automatic score corresponding to the chart question and answer model is used as the comprehensive score corresponding to the chart question and answer model, which is used to evaluate the chart question and answer model, wherein the score difference corresponding to the chart question and answer model is the absolute value of the difference between the manual score corresponding to the chart question and answer model and the automatic score corresponding to the chart question and answer model.

7. The comprehensive evaluation method of the diagram question-answering model according to claim 6, characterized in that: When the score difference corresponding to the graph question and answer model is not less than the preset score difference, the scoring instructions input into the referee model are optimized and updated so that the score difference corresponding to the graph question and answer model is less than the preset score difference.

8. The comprehensive evaluation method of the diagram question-answering model according to claim 1, characterized in that: Target chart types include at least regular line chart, complex line chart, regular bar chart, grouped bar chart, stacked bar chart, pie chart, and scatter chart.

9. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the comprehensive evaluation method of the graphical question-answering model as described in any one of claims 1-8.

10. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the comprehensive evaluation method for the graphical question-answering model as described in any one of claims 1 to 8 when executing the computer program.

Images