Multi-turn dialogue-based visual evaluation benchmark construction method for multi-type problems
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUIZHOU UNIV
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-05
Smart Images
Figure CN122153019A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and data visualization technology, specifically involving a method for constructing a multi-turn conversational visualization evaluation benchmark for multiple types of questions. It is particularly suitable for conversational natural language to visualization tasks driven by large language models, and can systematically evaluate the model's recognition ability, visualization generation accuracy and text response quality on multiple types of questions, including answerable, fuzzy, unanswerable and irrelevant questions. Background Technology
[0002] Natural Language to Visualization (NL2VIS) technology aims to automatically transform users' natural language queries into structured visual charts (such as bar charts and line charts), serving as an important means to lower the barrier to data analysis and empower non-professional users. In recent years, with the rapid development of Large Language Models (LLMs), LLM-based NL2VIS methods have significantly improved intent understanding and chart generation capabilities. For example, systems like Chat2VIS and LIDA can directly generate executable Vega-Lite visualization specifications based on a single-turn user query. However, most of these existing methods are based on single-turn, complete, and explicit question settings, assuming that users can accurately express their analytical needs in one go. This fails to adequately address the complexities common in real-world dialogue scenarios, such as semantic omissions, ambiguous intent, missing information, and topic deviations.
[0003] In real-world multi-turn conversational visualization interactions, user-posed questions are not always clear and actionable. Real-world queries typically fall into four categories: First, answerable questions, where the user explicitly expresses their visualization intent and the required data fields are complete. Second, ambiguous questions, where the user offers multiple reasonable interpretations of the chart type, data fields, or aggregation method, requiring system clarification. Third, unanswerable questions, where the user query involves information not found in the relevant data table or constraints beyond the data range. Fourth, irrelevant questions, where the user's input is unrelated to the current visualization task, such as casual conversation. Existing NL2VIS benchmarks (such as NVBen and NL4DV) primarily target single-turn, answerable queries, lacking systematic annotation for multiple question types, including ambiguous, unanswerable, and irrelevant ones, and failing to assess the model's ability to identify question types and provide differentiated responses in multi-turn contexts.
[0004] The lack of benchmarks for evaluating multi-turn dialogue visualizations across various question types significantly limits the performance evaluation of large language models (LLMs) on this task. On one hand, researchers typically use only metrics such as overall accuracy, failing to analyze in detail the model's bottlenecks in identifying fuzzy or unanswerable minority classes. On the other hand, existing evaluation methods often assume the model only needs to generate visualizations, without requiring it to provide appropriate textual responses (such as clarifying the question or explaining why it cannot be executed) when the question type is incorrectly identified. This incompleteness in evaluation makes it difficult to objectively measure the true robustness of different LLMs in complex conversational visualization scenarios, hindering targeted improvements to related technologies.
[0005] Therefore, the technical problem to be solved by this invention is: how to construct a multi-turn conversational visualization evaluation benchmark for multiple types of questions, which can systematically cover four types of user queries: answerable, fuzzy, unanswerable, and irrelevant, and support fine-grained evaluation of the model's type recognition ability, visualization generation accuracy, and text response quality in a multi-turn interaction context. Summary of the Invention
[0006] The main objective of this invention is to provide a method for constructing a multi-turn conversational visual evaluation benchmark for multiple types of questions. The aim is to build a multi-turn conversational visual evaluation benchmark that can systematically cover four types of user queries: answerable, fuzzy, unanswerable, and irrelevant, in order to solve the problem that existing evaluation benchmarks lack multi-type annotation and multi-turn context support.
[0007] Based on the first main aspect of the present invention, a method for constructing a multi-turn dialogic visual evaluation benchmark for multiple types of problems is provided, comprising the following steps performed by a computer system:
[0008] Obtain a single-round fuzzy query dataset, wherein the single-round fuzzy query contains at least one query statement with multiple candidate visualization schemes;
[0009] For each single-round fuzzy query, a large language model is used to reconstruct it into a multi-round dialogue sample containing at least two rounds of clarification interaction. The first round is the fuzzy request put forward by the user, and the subsequent rounds are clarification question-answer pairs used to disambiguate.
[0010] For the multi-turn dialogue samples, context-aware dialogue extension is performed using the thought chain method to generate an extended dialogue sequence that is semantically coherent with the original visual target.
[0011] Type validation and sentence structure optimization are performed on the generated multi-turn dialogue data to construct a multi-turn dialogue visualization evaluation benchmark that includes four types of question labels: answerable, ambiguous, unanswerable, and irrelevant.
[0012] The above approach transforms a single-turn fuzzy query, which lacks interactive processes, into a multi-turn dialogue sequence with a complete clarification chain and semantic coherence, ultimately constructing an evaluation benchmark labeled with four question categories. This method is the first to systematically label and evaluate four categories of questions—answerable, fuzzy, unanswerable, and irrelevant—in multi-turn dialogue scenarios, providing a standardized data foundation for objectively measuring the comprehensive capabilities of large language models in complex conversational visualization tasks.
[0013] As a further preferred embodiment, in the aforementioned method, the reconstructing of the sample into a multi-turn dialogue containing at least two rounds of clarification interaction using a large language model specifically includes:
[0014] For each ambiguous question, a target chart is pre-selected from multiple candidate visualization solutions, and a large language model is used to automatically generate a two-round interactive dialogue that includes the user's ambiguous question, the system's clarification question, and the user's supplementary explanation. This expands the original sample from a single ambiguous need to a complete process of clarifying the need.
[0015] The above solution pre-selects a target chart as the ideal result from multiple candidate visualization solutions for each ambiguous question. Then, using a large language model, it automatically generates a two-round interactive dialogue, including the user's ambiguous question, the system's clarification question, and the user's supplementary explanation. This expands the original single-round ambiguous requirement into a complete process of gradually clarifying the requirement through clarification. The effect is that the generated clarification dialogue has a clear goal orientation, enabling the ambiguous question to converge to a unique executable visualization solution after interaction, avoiding semantic drift caused by a lack of goal constraints.
[0016] As a further preferred embodiment, in the aforementioned method, the context-aware dialogue expansion via the thought chain method further includes:
[0017] A structured randomness mechanism is introduced to control the direction of expansion by randomly sampling a contextual relationship and a question type. The contextual relationships include three types: constraint enhancement, topic exploration, and participant switching, which are used to simulate typical contextual patterns in real multi-turn dialogues.
[0018] The above approach extracts key data fields representing the core intent from the generated clarification dialogue as anchor points. Then, it randomly samples a contextual relationship (including constraint enhancement, topic exploration, and participant switching) and a question type, expanding the question-and-answer content round-by-round around the anchor point according to the selected relationship and type. This approach effectively simulates typical interaction patterns in real multi-turn dialogues, such as users gradually revising their needs, switching analytical perspectives, or changing comparison objects. Simultaneously, random sampling avoids the uniformity of generated sample patterns, improving the logical coherence, semantic relevance, and expressive diversity of the dialogue sequence to a certain extent, making the evaluation benchmark closer to real-world application scenarios.
[0019] As a further preferred embodiment, in the aforementioned method, the type verification includes:
[0020] The generated question-answer pairs are checked one by one using a large language model to see if they conform to the preset question type definition. For samples with uncertain judgment results, the checks are repeated multiple times under non-zero temperature parameters. Only when the question type judgment results of multiple checks are completely consistent are the samples judged as verified and retained; otherwise, the samples are marked as unqualified and removed from the evaluation benchmark.
[0021] The "uncertain samples" refer to samples whose confidence level for the problem type output by the model is lower than a preset threshold or whose probability distributions are similar across multiple types during the initial check.
[0022] The above scheme utilizes a large language model to repeatedly check the generated question-answer pairs under non-zero temperature parameters. For uncertain samples with low confidence in the model output during the initial check or with similar probability distributions across multiple types, only those with completely consistent question type determinations across multiple checks are retained; otherwise, they are discarded. By introducing a multiple sampling and consensus voting mechanism, the type labeling fluctuations caused by randomness and illusion issues in the large language model are suppressed, thereby enhancing the stability and reliability of question type labels in the evaluation benchmark and providing high-quality ground truth annotations for subsequent model evaluation.
[0023] As a further preferred embodiment, in the aforementioned method, the sentence structure optimization includes:
[0024] Referring to the language features of the natural language visualization corpus, the user questions in the verified question-answer pairs are rewritten according to multiple preset sentence types using a large language model, so as to generate multiple variant samples of the same semantic content under different sentence expressions.
[0025] The various sentence types include at least three types: imperative, interrogative, and declarative, which are used to expand the coverage of question sentence types in the evaluation benchmark and improve the naturalness and flexibility of expression.
[0026] After completing the type verification, referencing the language expression features reflected in existing natural language visualization corpora (such as NLV Corpus), the user questions in the verified question-answer pairs are rewritten according to multiple preset sentence types using a large language model, thereby generating multiple variant samples with different sentence expressions for the same semantic content.
[0027] In practical implementation, sentence structure templates can be built as prompt examples for the large language model. For example, the imperative template can be designed as "verb (e.g., draw / generate / display) + analysis goal (including dimensions and measures) + chart type", such as "summarize sales by region and draw a bar chart". The interrogative template can be designed as "interrogative word (e.g., can / can) + verb + analysis goal + chart type + question mark", such as "can you summarize sales by region and generate a bar chart?" The declarative template can be designed as "subject (e.g., I / user) + want to view / need + analysis goal + chart type", such as "I want to view bar charts of sales by region".
[0028] By using the templates described above to guide the large language model in batch rewriting, each original question is expanded into three variations: imperative, interrogative, and declarative (and potentially more sentence types in practice). This expands the range of sentence types covered in the evaluation benchmark, avoids overfitting due to a single sentence type, and improves the naturalness and expressive flexibility of the benchmark. Consequently, it more accurately reflects the robustness and generalization ability of the large language model under different natural language expression habits.
[0029] As a further preferred embodiment, in the aforementioned method, the evaluation benchmark further includes a macro-average F1 score indicating the ability to identify problem types, which is calculated as follows:
[0030] For each of the four categories of questions—answerable, fuzzy, unanswerable, and irrelevant—the precision and recall of the model are calculated. The precision and recall are then harmonic averaged to obtain the F1 score for that category. Finally, the F1 scores of the four categories are arithmetically averaged.
[0031] Precision represents the proportion of samples that the model predicts as belonging to that class, while recall represents the proportion of samples that actually belong to that class that the model correctly predicts as belonging to that class.
[0032] This indicator is used to comprehensively evaluate the model's ability to identify balance across various types of problems, avoiding evaluation bias caused by an imbalance in the number of samples in each category.
[0033] Because the sample size varies across different question types (e.g., answerable questions account for 57.1%, while vague, unanswerable, and irrelevant questions together account for over 40%), using only the overall accuracy rate could easily lead to evaluation results dominated by the category with the larger sample size. Therefore, this metric employs the macro-average F1 method, as shown in the formula below:
[0034] ;
[0035] ;
[0036] in, This represents the F1 value for the macro average problem type. This represents the total number of question types. Precision(c) and Recall(c) characterize the prediction accuracy and coverage of LLM on the c-th type of question, respectively. C represents the set of question types. ={answerable, ambiguous, unanswerable, irrelevant}. c represents a specific question type in set C. F1(c) represents the F1 score of the c-th type of question.
[0037] ;
[0038] ;
[0039] Here, TP represents the number of samples correctly predicted as class c, FP represents the number of samples incorrectly predicted as class c but not actually belonging to that class, and FN represents the number of samples not predicted as class c but actually belonging to that class. The effect of this metric is to treat each class of questions fairly, avoid evaluation bias caused by sample imbalance, and thus accurately reflect the model's ability to identify difficult categories such as fuzzy and unanswerable questions.
[0040] As a further preferred embodiment, in the aforementioned method, the evaluation benchmark further includes the chart matching accuracy (CEM), which is statistically analyzed only when the model correctly identifies the question type for the current round and the type belongs to either answerable or fuzzy visualization tasks; its calculation method is as follows:
[0041] The proportion of rounds that meet the above conditions is where the visualization specifications generated by the model are completely consistent with the pre-annotated standard visualization specifications in terms of chart type, field mapping, encoding channels, aggregation transformation, sorting and filtering key elements.
[0042] The comparison ignores differences in the order of key-value pairs in the visualization specification;
[0043] This metric is used to evaluate the model's ability to generate executable and semantically accurate visualization specifications in scenarios where visualization results are required.
[0044] As an example, this metric is only valid if the type prediction is correct and it falls under the visualization task. Time calculation:
[0045] ;
[0046] in, This indicates the total number of interaction rounds contained in a complete dialogue. This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise. The model represents the first The types of problems predicted by the round of user questions. Indicates the first The actual problem types of user questions (manually labeled). This represents a set of visualization tasks. For exact matching functions, Indicates the first The wheels are pre-annotated with standard visualization specifications (such as JSON objects in Vega-Lite format). This represents the visualization specifications generated by the model. A value of 1 is assigned when two visualization specifications are completely identical in key elements such as chart type, field mapping and encoding channels, aggregation / transformation, sorting, and filtering; otherwise, a value of 0. To reduce interference from implementation details, differences in JSON order are ignored during comparison.
[0047] As a further preferred embodiment, in the aforementioned method, the evaluation benchmark further includes a Dialogue Quality Score (DQS), which is used for comprehensive evaluation on a per-dialogue-round basis. The score for a single round is calculated as follows:
[0048] The system determines whether the model correctly identifies the problem type for that round. If the identification is incorrect, the score for that round is zero; if the identification is correct, different scoring rules are applied based on the actual problem type for that round.
[0049] For answerable types, the score is a Boolean value indicating whether the visualization specification is fully matched; for ambiguous types, the score is the product of the Boolean value indicating a full match of the visualization specification and the quality of the text response; for unanswerable and irrelevant types, the score is the text response quality score.
[0050] The text response quality score is calculated by an automatic evaluator driven by a large language model, which scores the model-generated text response from four dimensions: accuracy, completeness, clarity, and relevance, and then takes the average value.
[0051] The final dialogue quality score is the arithmetic mean of the scores of each round of dialogue, and is used to comprehensively evaluate the overall performance of the model in simultaneously identifying question types and generating correct responses in multi-round dialogues.
[0052] As an example, the dialogue quality score is based on type recognition, integrating chart and text quality. When the question type is incorrectly identified, a response meeting the task requirements cannot be generated in that round, and the score is 0; when the question type is correctly identified, a score is awarded based on the quality of the generated response for that corresponding question type. Single-round score. Defined as:
[0053] ;
[0054] Ultimately, the overall DQS score is:
[0055]
[0056] in, This indicates the total number of interaction rounds contained in a complete dialogue. Indicates an indicator function. Indicates the first The actual problem types of user issues. For exact matching functions, Indicates the first Standard visualization specifications for pre-annotated wheels. This refers to the visualization specifications for model generation. This indicates the text quality score.
[0057] Based on a second key aspect of the present invention, a multi-turn dialogic visual evaluation benchmark construction system for implementing the aforementioned method and oriented towards multiple types of problems is provided, comprising:
[0058] The data acquisition unit is used to acquire a single-round fuzzy query dataset.
[0059] A clarification dialogue generation unit is used to reconstruct the single-turn fuzzy query into a multi-turn dialogue sample containing a clarification process using a large language model;
[0060] The dialogue expansion unit is used to extract data fields representing the core intent from the clarifying dialogue based on the thinking chain method, and expand the question and answer content around the anchor point in turn to generate a semantically coherent multi-turn dialogue sequence.
[0061] The data validation optimization unit is used to perform type consistency validation and sentence diversity rewriting on the generated multi-turn dialogue data using a large language model, and to generate an evaluation benchmark dataset containing four categories of labels: answerable, ambiguous, unanswerable, and irrelevant.
[0062] Based on a third key aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that, when the program is executed, it implements the aforementioned method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems.
[0063] Compared with existing technologies, this invention solves the problems of existing benchmarks lacking multi-type annotations, multi-turn contexts, and differentiated evaluation metrics by clarifying dialogue generation, expanding thought chain dialogues, and data verification and optimization processes. It provides a standardized data foundation and evaluation framework for objectively evaluating the comprehensive capabilities of large language models in conversational visualization tasks. This is mainly reflected in the following aspects:
[0064] First, this invention is the first in the field of conversational visualization to systematically construct a multi-turn dialogue evaluation benchmark covering four categories of questions: answerable, fuzzy, unanswerable, and irrelevant, breaking through the limitations of existing benchmarks that only target single-turn, answerable queries. Through a series of processes including clarifying dialogue generation, expanding thought chain dialogues, and type verification and sentence structure optimization, this invention transforms single-turn fuzzy queries into multi-turn dialogue sequences with clearly defined four categories of labels, enabling the evaluation benchmark to truly reflect the robustness of large language models in complex dialogue scenarios. This improvement redefines the evaluation scope of conversational visualization at the task definition level.
[0065] Second, this invention introduces a multi-round repeated verification mechanism and a sentence diversity optimization method based on a large language model, solving the technical problems of poor consistency in type labeling and limited expression forms in existing benchmarks. Addressing the "illusion" problem and random fluctuations in data generated by large language models, this invention employs a multiple repeated check and consistency voting strategy under a non-zero temperature parameter, retaining samples only when multiple judgments are completely consistent, thereby significantly improving the stability and reliability of labeling. Simultaneously, by rewriting user questions into various sentence structures such as imperative, interrogative, and declarative, the language coverage of the benchmark is expanded, avoiding evaluation bias caused by the model overfitting to specific sentence structures. These techniques work synergistically to improve the quality and generalization ability of the benchmark data.
[0066] Third, this invention designs a fine-grained, multi-dimensional evaluation index system (macro-average F1, chart matching accuracy (CEM), and dialogue quality score (DQS)). This system comprehensively measures model performance from three levels: question type identification, visualization specification generation, and text response quality, overcoming the shortcomings of existing evaluation indices that are singular and unable to distinguish error types. Specifically, the macro-average F1 avoids evaluation distortion caused by imbalanced samples by arithmetically averaging the F1 scores of the four question types. CEM calculates the specification perfect match rate only when the type is correct and belongs to a visualization task, decoupling type judgment from generation quality. DQS, through segmented scoring rules, takes type correctness as a prerequisite for subsequent responses, comprehensively reflecting the model's overall practicality in multi-turn dialogues.
[0067] This invention achieves significant innovations in three aspects: task definition, data construction method, and evaluation system. It not only fills the gap in multi-round conversational visualization evaluation benchmarks, but also lays a solid foundation for standardized evaluation in this field. Attached Figure Description
[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.
[0069] Figure 1 The following is an execution flowchart of a multi-turn dialogic visual evaluation benchmark construction method for multiple types of problems, according to one embodiment of the present invention;
[0070] Figure 2 An example diagram of the MTPCoVis task is shown in one embodiment of the present invention. Detailed Implementation
[0071] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the following embodiments are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.
[0072] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.
[0073] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.
[0074] In the following embodiments, the technical terms that may be involved and their meanings are as follows:
[0075] LLM (Large Language Models): refers to natural language processing models with a large number of parameters trained on massive amounts of text data. They are capable of understanding and generating natural language text and have the ability to perform context modeling, reasoning, and generation.
[0076] NL2VIS (Natural Language to Visualization): refers to the technology that automatically converts users' natural language queries into structured visual representations (such as chart specifications).
[0077] Vega-Lite: A declarative visualization syntax specification that uses JSON format to describe the type of chart, data fields, encoding channels (such as X-axis, Y-axis, color), and other attributes, which can be parsed and rendered into statistical charts by computers.
[0078] VQL (Visualization Query Language): An intermediate representation language that combines SQL-like data manipulation syntax with visualization structure information (such as chart types, grouping, binning, etc.) to transform natural language queries into structured visualization templates.
[0079] CoT (Chain of Thought): A cue-based learning method that guides large language models to complete complex tasks step by step by displaying the step-by-step reasoning process in the input, thereby improving the logicality and interpretability of the generated results.
[0080] MTPCoVis (Multi-Type Problem Conversational Visualization): This is the task name defined in this invention, referring to the visualization generation task that simultaneously handles four types of user queries—answerable, ambiguous, unanswerable, and irrelevant—in a multi-turn dialogue environment.
[0081] MTPCoVisEval: The name of a multi-turn conversational visual evaluation benchmark for multiple types of questions constructed in one embodiment of the present invention, which includes four types of question labels and corresponding dialogue data.
[0082] MTPCoVisAgent: A dialogic visual agent name based on a large language model proposed in one embodiment of the present invention, used to identify question types and generate corresponding responses in multi-turn dialogues.
[0083] CEM (Chart Exact Match): One of the evaluation metrics defined in this invention, used to measure the proportion of visualization specifications generated by the model that are completely consistent with standard specifications in key elements.
[0084] TQS (Text Quality Score): One of the evaluation metrics defined in this invention, which is the average of the scores given by an automatic evaluator on the text response generated by the model from four dimensions: accuracy, completeness, clarity, and relevance.
[0085] DQS (Dialogue Quality Score): One of the evaluation metrics defined in this invention, which comprehensively considers the correctness of the model's question type judgment and the quality of response generation in multi-turn dialogues, and is used to evaluate the overall dialogue performance of the model.
[0086] like Figure 1 As shown, in one embodiment of the present invention, a method for constructing a multi-turn dialogic visual evaluation benchmark for multiple types of problems includes the following steps S100-S400 executed by a computer system:
[0087] S100, Obtain a single-round fuzzy query dataset, wherein the single-round fuzzy query contains at least one query statement with multiple candidate visualization schemes;
[0088] S200: For each single-round fuzzy query, a large language model is used to reconstruct it into a multi-round dialogue sample containing at least two rounds of clarification interaction. The first round is the fuzzy request put forward by the user, and the subsequent rounds are clarification question-answer pairs used to eliminate ambiguity.
[0089] S300, For the multi-turn dialogue sample, the context-aware dialogue is extended using the thought chain method to generate an extended dialogue sequence that is semantically coherent with the original visual target.
[0090] S400 performs type validation and sentence structure optimization on the generated multi-turn dialogue data, and constructs a multi-turn dialogue visualization evaluation benchmark that includes four types of question labels: answerable, ambiguous, unanswerable, and irrelevant.
[0091] In one of the following embodiments, to further illustrate the above steps, evaluate the ability of LLMs to simultaneously complete question type identification and visualization generation in multi-turn dialogue scenarios, and overcome the limitations of the existing NL2VIS benchmark, which is mainly oriented towards single-turn, single-question type, this invention constructs a multi-turn dialogue-based visualization evaluation benchmark named MTPCoVisEval that is oriented towards multiple question types.
[0092] In this embodiment, the MTPCoVis task is defined first. MTPCoVis (Multi-Type Problem Conversational Visualization) is designed for visualization generation tasks in multi-turn dialogues. Its core is to simultaneously model query type discrimination and response generation to adapt to the dynamic changes in user intent during the interaction process.
[0093] like Figure 2 As shown, the question-and-answer content is merely illustrative and does not limit the scope of the invention. Figure 2 In the context of the current query Historical context With data table LLM requires output of the problem type. and corresponding response :
[0094] ;
[0095] Based on semantic integrity and executability, queries are divided into four mutually exclusive types.
[0096] Answerable: User questions contain a complete and explicit visualization intent and provide the key information needed to generate charts. These queries require the direct generation of syntactically correct and executable visualization specifications (such as Vega-Lite).
[0097] Ambiguous: User questions may have multiple reasonable interpretations in terms of field selection, chart type, or data transformation method. It is necessary to eliminate ambiguity by generating clarifying questions and converge to a clear expression in the next round of dialogue.
[0098] Unanswerable: When a user's question involves information that does not exist in the data table or constraints that are outside the data range, an explanatory text should be returned explaining why the task cannot be performed.
[0099] Irrelevant: User questions are not directly related to the data visualization task, such as casual conversation. In such cases, a general natural language response that fits the context of the conversation should be generated to maintain the continuity of the interaction.
[0100] exist Figure 2 Using a student record data table containing student ID, name, department, and total credits as the basic input, the system addresses the limitations of traditional Natural Language to Visualization (NL2VIS) technology, which can only handle single, answerable queries and cannot adapt to complex scenarios such as ambiguity, unanswerable, and irrelevant questions in real multi-turn dialogues. It fully demonstrates the complete processing logic for four typical user requests and clarifies the implementation principle of classifying questions first and then providing differentiated responses.
[0101] For the implementation process of the first type of answerable questions, when a user makes a clear request to "create a bar chart that displays the total credits by department, where the total credits are greater than 50", the model first completes the question type determination: confirming that the user's intent is clear and unambiguous, and that the data table contains all the required fields such as department (dept_name) and total credits (tot_cred), which meets the criteria for determining an answerable question.
[0102] The model then executes a visualization generation process, accurately analyzing the user's core requirements such as chart type, aggregation rules, and filtering conditions, and generating executable code that conforms to the Vega-Lite specification. This fully implements all requirements, including bar chart type, department classification axis, credit summation aggregation, and credit threshold filtering, ensuring that the output results accurately match the user's expectations.
[0103] In the implementation process for the second type of fuzzy problem, when a user raises an ambiguous query such as "how many students are represented in the pie chart", the model first determines that the problem type is fuzzy: the user only mentions "pie chart" without specifying the statistical dimension (whether it is classified by department or displayed by individual), the required information is missing, and it is impossible to directly generate an effective visualization.
[0104] Unlike traditional models that ignore ambiguity and forcibly generate invalid pie chart code, this model executes a clarifying response process, generating targeted guiding questions to confirm the statistical dimensions of the pie chart with the user. Through multiple rounds of interaction, it eliminates ambiguity in the user's needs, laying the foundation for generating accurate visualizations and ensuring the effectiveness and practicality of the interaction.
[0105] Regarding the implementation process for the third type of unanswerable questions, when a user requests a "bar chart showing the sum of graduation years by department", the model first determines the question type: confirming that there is no "graduation year (graduation_year)" related field in the data table, the user's request exceeds the coverage of the dataset, and it is impossible to generate a logically consistent and effective visualization.
[0106] Instead of forcibly generating invalid code with logical errors, the model executes an explanatory response process, generating clear text descriptions that clearly inform users of the core reason why visualization is not possible (the dataset lacks corresponding fields), thus avoiding misleading users and ensuring the credibility and professionalism of the interaction.
[0107] Regarding the implementation process for the fourth type of irrelevant question, when a user makes a casual request such as "Thank you for your help," the model first determines that the question type is irrelevant: the request is not directly related to the visualization task, and no visualization content needs to be generated. The model executes a general natural language response process, generating a polite reply that conforms to the dialogue logic, maintaining the coherence and friendliness of the dialogue, and adapting to the non-task-related dialogue needs in real-world interaction scenarios.
[0108] Overall, Figure 2 The examples in the document demonstrate the practical value of the MTPCoVis task through the complete implementation of four types of scenarios, and solve the adaptability defects of traditional technologies in real-world dialogue scenarios.
[0109] The following embodiments further illustrate the specific implementation process of steps S100-S400 of the present invention.
[0110] In practical implementation, firstly, single-round fuzzy query data is obtained from the existing NL2VIS dataset, corresponding to the scheme in step S100. In one specific embodiment, the NVBench 2.0 dataset is used as the original data source. This dataset contains a large number of fuzzy query statements with multiple candidate visualization schemes. For example, an original query is "display sales figures." This query does not explicitly specify the chart type, grouping field, or aggregation method, and there are multiple reasonable interpretations (such as a bar chart summarizing by region, a line chart summarizing by time trend, etc.). The system uses this type of query as the starting point for constructing the evaluation benchmark.
[0111] The purpose of step S200 is to reconstruct multi-turn dialogue samples using a large language model. For each single-turn fuzzy query, the system calls a large language model (such as GPT-o4-mini) to generate a clarifying dialogue. The specific process is as follows:
[0112] Pre-selected target chart: For each fuzzy question, a target chart is pre-selected manually or automatically from multiple candidate visualization options as the expected visualization result for that data. For example, for "display sales revenue", the pre-selected target is "a bar chart summarizing sales revenue by region".
[0113] Generate clarifying dialogue: Using a large language model as the generator, and inputting prompts including the user's vague question, the data table structure, and a pre-selected target chart, automatically generate a two-round interactive dialogue:
[0114] Round 1: The system generates clarifying questions, such as "What type of chart would you like to see for sales figures? A bar chart summarizing by region, or a line chart by time?"
[0115] Round 2: Simulate user supplementary instructions, such as "Please give me a bar chart summarizing by region".
[0116] Through the above process, the original single-round vague requirement is expanded into a complete dialogue segment that includes "user vague question → system clarification → user explicit statement".
[0117] Step 300 primarily utilizes the thought chain method to expand the dialogue with a context-aware approach. After obtaining two rounds of basic clarification dialogue, the system further expands the dialogue through multiple rounds to generate longer, semantically coherent dialogue sequences. The specific steps are as follows:
[0118] Anchor point extraction: Randomly extract key data fields that can represent the core intent from the generated dialogue as anchor points, such as discrete fields like "region" or time fields like "year".
[0119] Random sampling of context relationships and question types: The system maintains a structured randomness mechanism, randomly sampling a combination from three preset context relationships (constraint enhancement, topic exploration, participant switching) and four question types. For example, the sampling result is "constraint enhancement" + "answerable".
[0120] Step-by-step expansion: Based on the sampled relationships and types, subsequent question-and-answer pairs are generated around the anchor point. For example, if the previous round generated a "bar chart summarizing sales by region" and sampled an "enhanced constraint" relationship, the next round of user questions can be generated: "Only display regions with sales exceeding 1 million." The system uses a thought chain approach to progressively reason and generate question-and-answer pairs that conform to the contextual logic, ensuring that the entire dialogue sequence remains consistent with the original visualization goal.
[0121] Through the above expansion, an initial two-round dialogue can be extended to 6-8 rounds, and the question type for each round is labeled according to the sampling results.
[0122] Step 400 primarily involves type validation and sentence structure optimization. After generating multi-turn dialogue data, the system performs type validation and sentence structure optimization to improve data quality.
[0123] Type Validation: The system calls the large language model to perform multiple (e.g., 5) independent question type determinations for each question-answer pair under a non-zero temperature parameter (e.g., 0.7). Samples with low model output confidence (e.g., below 0.85) or similar probability distributions across multiple types during the initial check are considered uncertain samples. Only when the results of multiple checks are completely consistent is the sample deemed valid and retained; otherwise, it is discarded.
[0124] Sentence structure optimization: The system references the natural language expression features in the NLV Corpus and uses a large language model to rewrite the validated user questions into three sentence structures: imperative, interrogative, and declarative.
[0125] For example, the original question "Sales revenue by region" can be generated as:
[0126] Imperative: "Draw a bar chart summarizing sales by region"
[0127] Question type: "Can you summarize sales figures by region and generate a bar chart?"
[0128] Statement: "I want to see a bar chart of sales figures for each region."
[0129] Each original question generates three variant samples, thus enriching the sentence diversity of the evaluation benchmark.
[0130] In the following embodiments, to characterize the complexity of the MTPCoVis multi-turn dialogue visualization task in real-world interaction scenarios, the MTPCoVisEval dataset was constructed. This dataset contains 3858 dialogues and 23466 turns, with an average of 6.082 turns per dialogue, which can well reflect the dynamic behavior and question evolution of users in multi-turn interactions.
[0131] In this embodiment, in terms of question type distribution, there were 13,403 rounds of answerable questions, accounting for 57.1%; 3,709 rounds of ambiguous questions, accounting for 15.8%; 2,673 rounds of unanswerable questions, accounting for 11.4%; and 3,681 rounds of irrelevant questions, accounting for 15.7%. This indicates that more than 40% of the questions are not directly answerable. In addition to generating accurate visualization results, LLM also needs to identify complex situations such as semantic uncertainty, missing information, and contextual deviations to support robust interaction.
[0132] In terms of analysis tasks, the dataset covers a variety of common aggregation operations, including 6730 mean values, 5524 maximum values, 4845 count values, 2327 minimum values, and 1601 summation values, indicating that it contains a relatively rich set of statistical analysis requirements, which helps to comprehensively test the ability of LLM in statistical semantic understanding and visualization.
[0133] In terms of chart type distribution, the rounds requiring visualization specifications cover a variety of common graph types, including 5445 bar charts (31.8%), 4118 pie charts (24.1%), 3058 line charts (17.9%), and 2470 point charts (14.4%). Additionally, there are 1120 heatmaps (6.5%) and 904 boxplots (5.3%). This distribution indicates that the dataset covers both the basic statistical graphs that dominate actual analysis and a certain proportion of complex chart types, thus enhancing the comprehensiveness of the evaluation.
[0134] To evaluate the comprehensive capabilities of different methods in the MTPCoVis task, a fine-grained evaluation framework was constructed in the following embodiments. Assume a complete dialogue contains N rounds of interaction, the... The real problem type of the wheel is denoted as The problem types predicted by LLM are denoted as ,in .
[0135] When the When a task requires generating charts, the real visualization specification and the inference generation specification are respectively denoted as... and When the task in this round is a text response, its text quality score is recorded as follows: Based on the above definitions, this embodiment evaluates LLM performance from four dimensions: question type recognition capability, chart generation accuracy, text response quality, and overall dialogue quality, with corresponding indicators being question type F1, chart exact match (CEM), text quality score (TQS), and dialogue quality score (DQS).
[0136] Because the sample size varies across different question types, using only overall metrics such as accuracy can easily lead to evaluation results being dominated by the category with the larger sample size, thus masking the ability of LLM to identify categories with fewer samples. To mitigate the impact of class imbalance, this embodiment uses the macro-average question type F as the basic evaluation metric:
[0137] ;
[0138] ;
[0139] In the above formula, Precision(c) and Recall(c) respectively characterize the prediction accuracy and coverage capability of LLM on the c-th type of problem.
[0140] ;
[0141] ;
[0142] In the formula, TP represents the number of samples correctly predicted as class c, and FP represents the number of samples incorrectly predicted as class c but not actually belonging to that problem type.
[0143] In most implementations, chart matching accuracy (CEM) is only measured when the type prediction is correct and the task falls under visualization. Time calculation:
[0144] ;
[0145] in, The exact match function sets a value of 1 when two visualization specifications are completely identical in key elements such as chart type, field mapping and encoding channels, aggregation / transformation, sorting and filtering; otherwise, it sets a value of 0. To reduce interference from implementation details, differences in JSON order are ignored during comparison.
[0146] In most embodiments, the Text Quality Score (TQS) is only applied when the text is type-correct and belongs to a text task. Time calculation:
[0147] ;
[0148] in, Represents a set of text tasks. Indicates the first The quality score of the text response generated by the wheel model is calculated by averaging the scores given by an automated evaluator driven by GPT-o4-mini across four dimensions: accuracy, completeness, clarity, and relevance. Each dimension ranges from 0 to 5. The meanings of the remaining parameters are consistent with other formulas.
[0149] To further verify the reliability of the automatic evaluation, this embodiment invited three graduate students in the field of visualization to manually score 100 randomly selected text response samples and take the average value. At the same time, the correlation coefficients of Pearson, Spearman and Kendall's Tau were calculated.
[0150] Table 1. Correlation between automatic and manual evaluation of large language models (by question type)
[0151]
[0152] The correlation results are shown in Table 1. The automatic scoring and manual scoring are significantly positively correlated under different question types, indicating that the automatic evaluation method has good stability and consistency.
[0153] In most embodiments, the Dialogue Quality Score (DQS) is based on type recognition and integrates graph and text quality. When the question type is incorrectly identified, a response meeting the task requirements cannot be generated in that round, and the score is 0; when the question type is correctly identified, a score is awarded based on the quality of the generated response for the corresponding question type. Single-round score. Defined as:
[0154] ;
[0155] Ultimately, the overall DQS score is:
[0156]
[0157] in, This represents the index number of the interaction round in a multi-turn dialogue (i.e., which round of the dialogue), and its value ranges from 1 to N. N represents the total number of interaction rounds in a complete dialogue.
[0158] The four indicators mentioned above correspond to different levels of evaluation objectives. Question type F1 is used to evaluate the reliability of question type identification; CEM and TQS, assuming correct question type identification, respectively test the accuracy of chart generation and text generation; DQS further evaluates the comprehensive ability of the method in multi-turn dialogues from the overall dialogue perspective. The collaborative design of multi-dimensional indicators enables the conclusions to more comprehensively reflect the real performance in multi-turn dialogue visualization scenarios.
[0159] In the following examples, the performance of eight representative general-purpose LLMs on the MTPCoVis task was evaluated, covering inference LLMs (DeepSeek-V3.1, DeepSeek-R1) and non-inference LLMs (Qwen3-235B, Ernie-4.5-300B, Kimi-k2, and Llama-3 series 405B, 70B, 8B).
[0160] Table 2 summarizes the performance of various LLMs on four metrics: Question Type F1, CEM, TQS, and DQS. Overall, TQS is generally high across different LLMs, ranging from 0.7310 to 0.9474. This indicates that, assuming correct question type identification, most LLMs can generate well-structured and fluent text responses. In contrast, CEM is significantly lower overall, ranging from 0.1439 to 0.5490. This suggests that even with good text quality, LLMs still have limitations in generating executable visual specifications.
[0161] Table 2. Capabilities of different general-purpose large language models on the MTPovis task
[0162]
[0163] To further pinpoint the key bottlenecks in question type identification, this embodiment statistically analyzes the Precision, Recall, and F1 scores of different LLMs across four question types. Overall, the performance of each LLM is relatively stable in the answerable and irrelevant question types, with generally high F1 scores. In contrast, their identification ability in the ambiguous and unanswerable question types is significantly insufficient, with generally low F1 scores and a greater likelihood of significant imbalances between Precision and Recall.
[0164] The overall F1 score for most LLM question types is not high, further limiting the improvement potential of DQS. The overall performance exhibits a structural characteristic: relatively strong text generation capabilities, weaker chart execution capabilities, and insufficient stability in question type recognition.
[0165] It should be noted that the program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0166] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0167] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this invention does not impose any limitations on them.
[0168] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.
[0169] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems, characterized in that, This includes the following steps performed by the computer system: Obtain a single-round fuzzy query dataset, wherein the single-round fuzzy query contains at least one query statement with multiple candidate visualization schemes; For each single-round fuzzy query, a large language model is used to reconstruct it into a multi-round dialogue sample containing at least two rounds of clarification interaction. The first round is the fuzzy request put forward by the user, and the subsequent rounds are clarification question-answer pairs used to disambiguate. For the multi-turn dialogue samples, context-aware dialogue extension is performed using the thought chain method to generate an extended dialogue sequence that is semantically coherent with the original visual target. Type validation and sentence structure optimization are performed on the generated multi-turn dialogue data to construct a multi-turn dialogue visualization evaluation benchmark that includes four types of question labels: answerable, ambiguous, unanswerable, and irrelevant.
2. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 1, characterized in that, The process of reconstructing the large language model into a multi-turn dialogue sample containing at least two rounds of clarification interaction specifically includes: For each ambiguous question, a target chart is pre-selected from multiple candidate visualization solutions, and a large language model is used to automatically generate a two-round interactive dialogue that includes the user's ambiguous question, the system's clarification question, and the user's supplementary explanation. This expands the original sample from a single ambiguous need to a complete process of clarifying the need.
3. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 1, characterized in that, The context-aware dialogue extension using the thought chain method further includes: A structured randomness mechanism is introduced to control the direction of expansion by randomly sampling a contextual relationship and a question type. The contextual relationships include three types: constraint enhancement, topic exploration, and participant switching, which are used to simulate typical contextual patterns in real multi-turn dialogues.
4. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 1, characterized in that, The type validation includes: The generated question-answer pairs are checked one by one using a large language model to see if they conform to the preset question type definition. For samples with uncertain judgment results, the checks are repeated multiple times under non-zero temperature parameters. Only when the question type judgment results of multiple checks are completely consistent are the samples judged as verified and retained; otherwise, the samples are marked as unqualified and removed from the evaluation benchmark. The "uncertain samples" refer to samples whose confidence level for the problem type output by the model is lower than a preset threshold or whose probability distributions are similar across multiple types during the initial check.
5. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 1, characterized in that, The sentence structure optimization includes: Referring to the language features of the natural language visualization corpus, the user questions in the verified question-answer pairs are rewritten according to multiple preset sentence types using a large language model, so as to generate multiple variant samples of the same semantic content under different sentence expressions. The various sentence types include at least three types: imperative, interrogative, and declarative.
6. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 4 or 5, characterized in that, The evaluation benchmark further includes a macro-average F1 score for the ability to identify problem types, which is calculated as follows: For each of the four categories of questions—answerable, fuzzy, unanswerable, and irrelevant—the precision and recall of the model are calculated. The precision and recall are then harmonic averaged to obtain the F1 score for that category. Finally, the F1 scores of the four categories are arithmetically averaged. Precision represents the proportion of samples that the model predicts as belonging to that class, while recall represents the proportion of samples that actually belong to that class, which are correctly predicted by the model.
7. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 4 or 5, characterized in that, The evaluation benchmark further includes Chart Matching Accuracy (CEM), which is only counted when the model correctly identifies the question type for the current round and the type belongs to either answerable or fuzzy visualization tasks; its calculation method is as follows: The proportion of rounds that meet the above conditions is where the visualization specifications generated by the model are completely consistent with the pre-annotated standard visualization specifications in terms of chart type, field mapping, encoding channels, aggregation transformation, sorting and filtering key elements. The order of key-value pairs in the visualization specification is ignored during the comparison.
8. The method for constructing multi-turn dialogic visual evaluation benchmarks for multiple types of problems according to claim 4 or 5, characterized in that, The evaluation criteria further include the Dialogue Quality Score (DQS), which is a comprehensive evaluation based on dialogue rounds. The score for each round is calculated as follows: The system determines whether the model correctly identifies the problem type for that round. If the identification is incorrect, the score for that round is zero; if the identification is correct, different scoring rules are applied based on the actual problem type for that round. For answerable types, the score is a Boolean value indicating whether the visualization specification is fully matched; for ambiguous types, the score is the product of the Boolean value indicating a full match of the visualization specification and the quality of the text response; for unanswerable and irrelevant types, the score is the text response quality score. The text response quality score is calculated by an automatic evaluator driven by a large language model, which scores the model-generated text response from four dimensions: accuracy, completeness, clarity, and relevance, and then takes the average value. The final dialogue quality score is the arithmetic mean of the scores for each round of dialogue.
9. A system for constructing a multi-turn dialogic visual evaluation benchmark for multi-type problems to implement the method as described in claim 1, characterized in that, include: The data acquisition unit is used to acquire a single-round fuzzy query dataset. A clarification dialogue generation unit is used to reconstruct the single-turn fuzzy query into a multi-turn dialogue sample containing a clarification process using a large language model; The dialogue expansion unit is used to extract data fields representing the core intent from the clarifying dialogue based on the thinking chain method, and expand the question and answer content around the anchor point in turn to generate a semantically coherent multi-turn dialogue sequence. The data validation optimization unit is used to perform type consistency validation and sentence diversity rewriting on the generated multi-turn dialogue data using a large language model, and to generate an evaluation benchmark dataset containing four categories of labels: answerable, ambiguous, unanswerable, and irrelevant.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the multi-turn dialogic visual evaluation benchmark construction method for multi-type problems as described in claim 1.