Chart understanding model diagnosis method and device based on legend counterfactual intervention

Abstract

The application discloses a chart understanding model diagnosis method and device based on legend counterfactual intervention, and belongs to the field of multi-modal model evaluation, specifically: a first chart is rendered according to the chart specification and the legend specification in the first sample unit, and the legend question and the question answer are used as the matching question and answer metadata; counterfactual intervention is implemented on the legend specification to construct a second sample unit, and a second chart is generated according to the second sample unit; wherein when the mapping relationship corresponding to the legend specification changes, the legend question and the question answer are updated; the first chart and the corresponding legend question are input into a target model to obtain a first output, and the second chart and the corresponding legend question are input into the target model to obtain a second output; based on the first output, the second output and the question answers corresponding thereto, a model diagnosis result is generated. Therefore, by implementing the application, causal diagnosis of the model legend understanding ability can be realized.

Description

Technical Field

[0001] This invention relates to the field of multimodal model evaluation, and in particular to a diagnostic method and apparatus for graph understanding models based on legend counterfactual intervention. Background Technology

[0002] Chart-based question answering and chart understanding are important applications of multimodal large models. Existing models typically need to extract values ​​from visualizations such as line charts, bar charts, scatter plots, area charts, pie charts, or radar charts, identify series correspondences, and perform statistical inference. Legends are the core semantic bridge in multi-series charts, defining the mapping relationship between "visual attributes" and "data series," such as which series a color, dot shape, or line type corresponds to. As the application of multimodal large models expands in scientific charts, business dashboards, and automated analysis scenarios, accurately evaluating the chart understanding capabilities of these models has become a key issue in the field of multimodal model evaluation.

[0003] Current mainstream chart-based question-answering benchmarks, such as FigureQA, DVQA, PlotQA, ChartQA, and MultiChartQA, typically evaluate models based on static samples of "image + question + answer." These evaluation methods only report the model's overall question-answering accuracy on the test set, failing to pinpoint the specific reasons for errors: whether it's a failure in legend parsing, a failure to bind the legend to graphic elements within the chart, a failure in subsequent inference based on the legend, or a failure to correctly reject an answer when legend information is missing. Diagnostic evaluation methods focusing on visual encoding or real-world charts, such as EncQA, ChartMuseum, and ChartInsights, are beginning to address the impact of encoding type, text shortcuts, or low-level reading issues on model performance. However, these methods still primarily evaluate isolated samples, lacking a systematic task decomposition for the crucial legend component, and also lacking a group-based comparative evaluation mechanism that only changes the legend while keeping other chart conditions largely unchanged. This makes it difficult to verify whether the model's output can reasonably change with variations in the legend.

[0004] In addition, existing visual attribution, alignment, or counterfactual evaluation methods can observe whether the model pays attention to relevant areas in the chart. However, these methods are usually observational or non-chart-native analysis methods and have not formed a systematic diagnostic system for the four types of tasks of chart legend analysis, binding, reasoning, and rejection. They cannot yet perform step-by-step diagnosis of the model's subdivided abilities such as legend analysis, legend binding, legend reasoning, or legend rejection. In summary, existing model evaluation methods have the following shortcomings: First, in existing static chart-based question-and-answer benchmarks, the legend and the chart are generated as a whole, and the legend specifications cannot be independently modeled and modified. This makes it impossible to implement counterfactual interventions based solely on the legend, and also lacks a mechanism for automatically updating the questions and answers synchronously after changes in the legend's mapping relationship. Second, existing methods lack a mechanism for calculating paired diagnostic indicators by intervention type, making it impossible to assess the model's binding-maintained class consistency and binding-change-class sensitivity through grouped control samples. This makes it difficult to verify whether the model output truly depends on the mapping relationship defined by the legend. Third, existing methods cannot systematically decompose and fine-grainedly diagnose the model's capabilities based on the legend across four tasks: parsing, binding, reasoning, and abandonment of answers, making it difficult to pinpoint the specific points of model failure. Summary of the Invention

[0005] This invention provides a diagnostic method and apparatus for a diagram understanding model based on legend counterfactual intervention, which can realize causal diagnosis of the model's ability to understand legends.

[0006] This invention provides a diagnostic method for a chart understanding model based on legend counterfactual intervention, including: Multiple first sample units are constructed, and corresponding first charts are generated for each first sample unit. Each first sample unit includes chart specifications, legend specifications, legend questions, and question answers. The legend specifications are used to define the mapping relationship between visual attributes and data series. The legend specifications can be modified independently of the chart specifications. Counterfactual interventions are applied to each of the aforementioned legend specifications. Based on the intervention results, multiple second sample units are constructed, and corresponding second charts are generated for each second sample unit. For any second sample unit, if the corresponding mapping relationship changes, the corresponding legend question and answer are updated, while the updated question intent remains fixed. The types of counterfactual interventions include binding preservation, binding change, and visibility. The chart specifications and underlying data of the second charts are the same as those of the first chart. Each of the first charts and the corresponding legend questions are input into the target model for chart understanding, resulting in multiple first outputs. Similarly, each of the second charts and the corresponding legend questions are input into the target model for chart understanding, resulting in multiple second outputs. When the counterfactual intervention is of the binding-maintaining or binding-changing type, the consistency index and sensitivity index of the target model are calculated based on each of the first outputs and the corresponding question answers, and each of the second outputs and the corresponding question answers, and a model diagnostic result is generated based on the consistency index and the sensitivity index; when the counterfactual intervention is of the visibility type, the answerability index and unanswerable index of the target model are calculated based on each of the second outputs and the corresponding question answers, and a model diagnostic result is generated based on the answerability index and the unanswerable index; wherein, the model diagnostic result also includes diagnostic results output by the legend parsing task, legend binding task, legend conditional reasoning task and legend perception unanswerable task respectively.

[0007] This invention constructs multiple first sample units, each containing chart specifications and legend specifications, and generates corresponding first charts. The legend specifications are independent of the chart specifications and can be modified independently, providing a base chart for subsequent generation of control samples. By implementing binding-preservation, binding-change, and visibility-based counterfactual interventions on the legend specifications and constructing corresponding second charts, and updating the question-and-answer mechanism while maintaining the question intent when the mapping relationship changes, while keeping the chart specifications and underlying data of the second chart identical to the first chart, it is possible to obtain a set of control charts by only changing the legend's mapping relationship or visibility. By inputting the set of first and second charts into the target model to obtain corresponding outputs, it is possible to obtain the model's performance in the legend mapping... This invention provides differentiated outputs under three conditions: unchanged mapping relationship, changed mapping relationship, and legend not visible. When the intervention type is binding-maintaining or binding-changing, consistency and sensitivity indices are calculated based on the model output and corresponding answers, respectively. This quantifies the model's output stability when the legend mapping relationship remains unchanged and its output sensitivity when the mapping relationship changes. When the intervention type is visibility-based, answerable and rejection indices are calculated based on the model output and corresponding answers. This quantifies the model's ability to answer answerable questions and correctly reject unanswerable questions when the legend is not visible. Model diagnostic results are generated based on these indices, allowing verification of whether the model output changes reasonably due to legend changes and whether correct rejection occurs when the legend is not visible. Compared to existing technologies that struggle to verify whether model output changes reasonably due to legend changes, this application enables causal diagnosis of the model's legend understanding ability.

[0008] Further, the construction of multiple first sample units and the generation of corresponding first charts based on each first sample unit includes: Acquire chart data and generate chart specifications based on the chart data and preset chart configuration; Assign visual attributes to each data series included in the chart specification to obtain the legend specification; Based on the chart specifications and the legend specifications, generate legend questions and corresponding answers; wherein, the legend questions belong to one of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task; The chart specifications, the legend specifications, the legend questions, and the question answers are combined into a first sample unit; The first chart is generated by rendering based on the chart specifications and the legend specifications.

[0009] This invention, through parameterized definition of chart data, visual attribute allocation, and legend task generation, and rendering the chart after combining them into sample units, can generate standardized test samples with legend mapping relationships.

[0010] Furthermore, the step of implementing counterfactual intervention on each of the aforementioned legend specifications, constructing multiple second sample units based on the intervention results, and generating corresponding second charts according to each second sample unit includes: For any legend specification, a counterfactual intervention is performed on the legend specification to obtain a variant legend specification; Compare the mapping relationship defined in the variant legend specification with the mapping relationship defined in the legend specification; If the comparison results are inconsistent, the legend question and the answer are updated according to the mapping relationship defined by the variant legend specification, and the chart specification, the variant legend specification, the updated legend question and the updated answer are combined into a second sample unit; If the comparison results are consistent, the chart specifications, the variant legend specifications, the legend question, and the question answer will be combined into a second sample unit. A second chart is generated based on the chart specifications and the variant legend specifications.

[0011] This invention, through intervention on the legend specifications and comparison with the original mapping relationship, selects to update or retain the question and answer based on the comparison results and constructs variant units. This can distinguish the intervention type and ensure that the variant chart is synchronized with the current legend mapping relationship.

[0012] Furthermore, the step of inputting each of the first charts and the corresponding legend questions into the target model for chart understanding yields multiple first outputs, including: For any first chart, convert the legend question in the corresponding first sample unit into text format to obtain the first question text; Input the first chart and the first question text into the target model, and receive the first output generated by the target model.

[0013] This invention, by converting the legend of the first chart into text and inputting it into the model along with the chart, can standardize the reasoning process of the basic samples and provide a reference benchmark for subsequent comparisons.

[0014] Furthermore, the step of inputting each of the second charts and the corresponding legend questions into the target model for chart understanding yields multiple second outputs, including: For any second chart, convert the legend question in the corresponding second sample unit into text format to obtain the second question text; Input the second chart and the second question text into the target model, and receive the second output generated by the target model.

[0015] This invention enables the acquisition of inference results for variant samples under legend intervention conditions by converting the legend question of the second chart into text and inputting it into the model along with the chart.

[0016] Further, the step of calculating the consistency index and sensitivity index of the target model based on each of the first outputs and the corresponding question answers, and each of the second outputs and the corresponding question answers, includes: Any first sample unit and its corresponding second sample unit are used as a set of control samples; wherein, if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are consistent, then the control sample is the first control sample; if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are inconsistent, and the answer to the question corresponding to the first sample unit is different from the answer to the question corresponding to the second sample unit, then the control sample is the second control sample. For any control sample, the corresponding first output is compared with the corresponding question answer to obtain the first comparison result, and the corresponding second output is compared with the corresponding question answer to obtain the second comparison result; If the control sample is the first control sample, then when both the first comparison result and the second comparison result are consistent, the control sample is determined to have passed the consistency test; If the control sample is the second control sample, then when the first comparison result and the second comparison result are both consistent, the control sample is determined to have passed the sensitivity test; The proportion of groups that passed the consistency test to the total number of groups in the first control sample was calculated to obtain the consistency index. The proportion of groups that passed the sensitivity test to the total number of groups in the second control sample was calculated to obtain the sensitivity index.

[0017] This invention categorizes control samples based on whether the mapping relationship is consistent, and calculates the proportion of both classes of samples where the outputs are correct in both instances, thereby quantifying the output stability and output sensitivity of the model.

[0018] Further, generating model diagnostic results based on the consistency index and the sensitivity index includes: If the consistency index is lower than the preset consistency threshold, the generated model diagnosis result is that the target model has a legend binding stability defect; wherein, the legend binding stability defect indicates that the target model outputs inconsistently when the mapping relationship between visual attributes and data series has not changed; If the sensitivity index is lower than the preset sensitivity threshold, the generated model diagnosis result is that the target model has a legend binding sensitivity defect; wherein, the legend binding sensitivity defect indicates that the target model fails to correctly adjust the output as the mapping relationship between visual attributes and data series changes.

[0019] By comparing consistency and sensitivity indicators with preset thresholds, this invention can identify failure types where the model outputs are unstable when the legend remains unchanged or fails to adjust correctly when the legend changes.

[0020] Further, the step of calculating the answerability index and the abandonment index of the target model based on each of the second outputs and the corresponding question answers includes: For any second sample unit, determine whether the legend question corresponding to the second sample unit can still be answered when the legend is partially or completely invisible; If it is still possible to answer, the second sample unit is marked as an answerable sample. The corresponding second output is compared with the corresponding question answer. The proportion of the number of answerable samples with the same comparison result to the total number of answerable samples is calculated to obtain the answerability index. If an answer cannot be given, the second sample unit is marked as an unanswerable sample. The corresponding second output is compared with the expected rejection behavior. The rejection accuracy and rejection recall are calculated based on the comparison results. The harmonic mean of the rejection accuracy and the rejection recall is used as the rejection index. The expected rejection behavior refers to the preset rejection text output by the target model when it is unable to answer based on the legend information.

[0021] This invention categorizes samples under visibility-based interventions into answerable and unanswerable samples based on whether the question is still answerable. For answerable samples, the proportion of outputs matching the answers is calculated to obtain an answerability index. For unanswerable samples, the outputs are compared with the expected rejection behavior, and the harmonic mean of the rejection accuracy and rejection recall is calculated as the rejection index. This allows for the quantification of the model's reliability in answering questions that are still answerable under conditions where the legend is not visible, as well as its ability to correctly reject questions that are truly unanswerable, providing a detailed quantitative basis for diagnosing the model's rejection behavior.

[0022] Furthermore, the model diagnostic results also include diagnostic results output for the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task, respectively, specifically: Based on the comparison results between each first output and the corresponding question answer, the accuracy rate is calculated for each of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task. The corresponding diagnostic results are then output based on the accuracy rate of each type of task.

[0023] This invention calculates the accuracy rate and outputs the corresponding diagnostic results for each of the following tasks: legend parsing, legend binding, legend conditional reasoning, and legend perception abandonment task. This allows the model's legend comprehension ability to be decomposed into four specific task stages for independent evaluation, enabling precise localization of model failure points.

[0024] Another embodiment of the present invention provides a diagnostic device for a chart understanding model based on legend counterfactual intervention, comprising: a chart generation module, a counterfactual intervention module, a model reasoning module, and a result generation module; The chart generation module is used to construct multiple first sample units and generate corresponding first charts based on each first sample unit; wherein, each first sample unit includes chart specifications, legend specifications, legend questions and question answers, the legend specifications are used to define the mapping relationship between visual attributes and data series, and the legend specifications can be modified independently of the chart specifications; The counterfactual intervention module is used to implement counterfactual intervention on each of the legend specifications, construct multiple second sample units based on the intervention results, and generate corresponding second charts according to each second sample unit; wherein, for any second sample unit, if the corresponding mapping relationship changes, the corresponding legend question and question answer are updated; The model reasoning module is used to input each of the first charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple first outputs accordingly. It also inputs each of the second charts and the corresponding legend questions into the target model for chart understanding, and obtains multiple second outputs accordingly. The updated question intent remains fixed. The counterfactual intervention types include binding preservation, binding change, and visibility. The chart specifications and underlying data of the second charts are the same as those of the first charts. The result generation module is configured to, when the counterfactual intervention is of the binding-maintaining or binding-changing type, calculate the consistency index and sensitivity index of the target model based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers, and generate a model diagnostic result based on the consistency index and the sensitivity index; when the counterfactual intervention is of the visibility type, calculate the abandonment index of the target model based on each of the second outputs and their corresponding question answers, and generate the model diagnostic result based on the abandonment index; wherein, the model diagnostic result also includes diagnostic results output by the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task respectively. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating an embodiment of the diagram understanding model diagnostic method based on legend counterfactual intervention provided by the present invention. Figure 2 This is a schematic diagram of one embodiment of the diagram understanding model diagnostic device based on legend counterfactual intervention provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0028] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0031] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0032] See Figure 1 To address the problem that existing technologies struggle to verify whether model outputs can reasonably change due to legend variations, an embodiment of the present invention provides a diagnostic method for chart understanding models based on legend counterfactual intervention, comprising steps S101 to S104: Step S101: Construct multiple first sample units and generate corresponding first charts based on each first sample unit; wherein, each first sample unit includes chart specifications, legend specifications, legend questions and question answers, the legend specifications are used to define the mapping relationship between visual attributes and data series, and the legend specifications can be modified independently of the chart specifications.

[0033] It should be noted that constructing multiple first sample units and generating corresponding first charts based on each first sample unit means: for the target model to be diagnosed, multiple first sample units are pre-constructed, with each first sample unit corresponding to a basic test case. For each first sample unit, the chart type, underlying data, and number of series are determined according to its chart specifications. Visual attributes such as color, dot shape, line type, or texture are assigned to each data series according to its legend specifications, and the mapping relationship between visual attributes and data series is defined. Simultaneously, it is determined which type of task the legend question corresponding to the first sample unit belongs to: legend parsing task, legend binding task, legend conditional reasoning task, or legend perception abandonment task, as well as the corresponding question answer. Based on this, a corresponding first chart image is rendered and generated according to the chart specifications and legend specifications. This first chart serves as a base sample for subsequent group comparison with a second chart obtained through counterfactual intervention.

[0034] Preferably, the step of constructing multiple first sample units and generating corresponding first charts based on each first sample unit includes: Acquire chart data and generate chart specifications based on the chart data and preset chart configuration; Assign visual attributes to each data series included in the chart specification to obtain the legend specification; Based on the chart specifications and the legend specifications, generate legend questions and corresponding answers; wherein, the legend questions belong to one of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task; The chart specifications, the legend specifications, the legend questions, and the question answers are combined into a first sample unit; The first chart is generated by rendering based on the chart specifications and the legend specifications.

[0035] In one embodiment, each first sample unit can be represented as ,in For chart specifications, For illustration specifications, For the illustration problem, This is the answer to the question. Chart specifications should at least include chart type, underlying data, number of series, number of points, theme, and whether to display numerical labels; wherein, the chart type is selected from the point primitive family, line primitive family, and area primitive family to facilitate horizontal comparison under the shared primitive mechanism. Legend specifications should at least include visual encoding channels, encoding mode, color scheme, point shape, line type, texture scheme, legend position, legend size, column layout, and visibility.

[0036] Furthermore, based on real chart data sources, synthetic data sources, data that has been redrawn in a unified style from real chart data sources, or mixed data combining synthetic and real chart data sources, basic charts such as scatter plots, line charts, radar charts, bar charts, area charts, or pie charts (or any visualization object that uses legends to identify different data series, categories, or semantic dimensions) are generated according to preset configurations, ensuring that the basic charts have definite data semantics before legend intervention is added; among them, synthetic data sources preferably generate series names with entity indicator semantics to improve the rationality of legend labels.

[0037] Furthermore, a legend channel is specified for the base chart. For example, line charts use color or color + dot encoding, and bar charts use color or color + texture encoding. Simultaneously, visual attributes are assigned to each data series in the base chart according to the chart type to obtain the first chart. These visual attributes include one or more of color, dot shape, line type, and texture, and support single-channel encoding, dual-channel redundant encoding, and combined encoding.

[0038] Furthermore, the system automatically generates legend questions and corresponding answers for four types of tasks: legend parsing (e.g., "How many labels are there in this legend?", "What visual attribute does a certain label correspond to?"), legend binding (e.g., "What is the value of a certain series in a certain year?"), legend conditional reasoning (e.g., "Which two curves intersect?", "What is the average value of the first three time points?"), and legend awareness abandonment task (e.g., "Can this question still be answered when the legend is occluded?"). Legend question generation is not limited to direct instantiation of fixed templates; it can also be constructed using rule generation, parameter filling, template combination, semantic constraint generation, or generation assisted by a large language model followed by program verification. Visual vocabulary cues can be added to the templates, indicating only which visual channels are used without directly revealing the mapping from the legend to the data series, thus balancing answerability and diagnostics. Legend questions can be expressed in question-and-answer, judgment, selection, matching, or fill-in-the-blank formats, as long as their essence is to test the model's ability to parse and bind legends, reason based on legends, or make judgments when legend information is missing.

[0039] Step S102: Implement counterfactual interventions on each of the legend specifications, construct multiple second sample units based on the intervention results, and generate corresponding second charts according to each second sample unit; wherein, for any second sample unit, if the corresponding mapping relationship changes, update the corresponding legend question and question answer, and keep the updated question intent fixed; the types of counterfactual interventions include binding preservation, binding change, and visibility; the chart specifications and underlying data of the second chart are the same as those of the first chart.

[0040] It should be noted that implementing counterfactual interventions on each of the aforementioned legend specifications, constructing multiple second sample units based on the intervention results, and generating corresponding second charts based on each second sample unit means: for the legend specifications in each first sample unit, performing at least one counterfactual intervention operation to obtain variant legend specifications. This counterfactual intervention operation is performed at the legend specification level, maintaining the non-legendary visual implementation, underlying data, series values, axes, chart type, and question intent in the corresponding chart specifications unchanged. After obtaining the variant legend specifications, if the mapping relationship between its defined visual attributes and data series changes, the corresponding legend question and question answer are updated according to the new mapping relationship, ensuring that the data series pointed to by the updated legend question remains consistent with the mapping relationship in the variant legend specifications; subsequently, the chart specifications, variant legend specifications, updated legend questions, and updated question answers are combined into a second sample unit. If the mapping relationship remains unchanged, the chart specifications, variant legend specifications, original legend questions, and original question answers are directly combined into a second sample unit. Finally, the corresponding second chart is rendered and generated based on the chart specifications and variant legend specifications. The second chart serves as a variant sample, forming a control sample with the corresponding first chart, and is used to subsequently verify whether the output of the target model changes reasonably due to the change in the legend.

[0041] Preferably, the step of implementing counterfactual intervention on each of the legend specifications, constructing multiple second sample units based on the intervention results, and generating corresponding second charts according to each second sample unit includes: For any legend specification, a counterfactual intervention is performed on the legend specification to obtain a variant legend specification; Compare the mapping relationship defined in the variant legend specification with the mapping relationship defined in the legend specification; If the comparison results are inconsistent, the legend question and the answer are updated according to the mapping relationship defined by the variant legend specification, and the chart specification, the variant legend specification, the updated legend question and the updated answer are combined into a second sample unit; If the comparison results are consistent, the chart specifications, the variant legend specifications, the legend question, and the question answer will be combined into a second sample unit. A second chart is generated based on the chart specifications and the variant legend specifications.

[0042] In one embodiment, counterfactual interventions are categorized into binding preservation (i.e., only changing the legend display order or legend position without altering the correspondence between legend entries and sequences in the figure), binding change (i.e., swapping the bindings between legend entries and sequences), and visibility (i.e., hiding the entire legend, retaining only the target item, retaining only the distractor item, or hiding the target item). The binding preservation category is used to test stability, the binding change category is used to test causal sensitivity, and the visibility category is used to test the ability to reject answers. Specifically, these include, but are not limited to, legend rearrangement, legend entry position swapping, legend transposition, legend shrinking, position relocation, similar color replacement, partial occlusion, full occlusion, retaining the target item, and retaining the distractor item.

[0043] Furthermore, if the mapping relationship corresponding to the legend changes after counterfactual intervention, the visual description or label pointing involved in the legend question will be automatically updated, and the question answer will be updated synchronously, so that the second sample unit has the same analytical intent and similar difficulty as the first sample unit.

[0044] Step S103: Input each of the first charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple first outputs. Input each of the second charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple second outputs.

[0045] It should be noted that inputting each first chart and its corresponding legend question into the target model for chart understanding, resulting in multiple first outputs, and inputting each second chart and its corresponding legend question into the target model for chart understanding, resulting in multiple second outputs, means that: for each control sample, the first chart and its corresponding legend question are first input into the target model to be diagnosed. The target model extracts visual features from the first chart and performs multimodal understanding and reasoning in conjunction with the legend question, generating the first output corresponding to the first sample unit. Then, the second chart and its corresponding legend question from the same control sample are input into the same target model. The target model extracts visual features from the second chart and performs multimodal understanding and reasoning in conjunction with the legend question, generating the second output corresponding to the second sample unit. The first output serves as the basic reference, and the second output serves as the variant reference; the two constitute a pair of outputs that can be compared and analyzed. By traversing all groups of control samples, all first outputs and all second outputs are obtained, providing a data foundation for subsequent calculation of consistency and sensitivity indices.

[0046] Preferably, the step of inputting each of the first charts and the corresponding legend questions into the target model for chart understanding yields multiple first outputs, including: For any first chart, convert the legend question in the corresponding first sample unit into text format to obtain the first question text; Input the first chart and the first question text into the target model, and receive the first output generated by the target model.

[0047] Preferably, the step of inputting each of the second charts and the corresponding legend questions into the target model for chart understanding yields multiple second outputs, including: For any second chart, convert the legend question in the corresponding second sample unit into text format to obtain the second question text; Input the second chart and the second question text into the target model, and receive the second output generated by the target model.

[0048] In one embodiment, for any set of control samples, the first chart image from the base sample side and the corresponding legend question text are simultaneously fed into the target model to be diagnosed. The target model extracts visual features from the first chart image, identifies the visual attributes and spatial distribution of each graphic element in the chart, and understands the semantics of the legend question text. Based on the mapping relationship between the visual attributes and data series established by the legend, it points the visual descriptions involved in the question to the corresponding data series, completes numerical reading or statistical inference, and generates the first output. Next, the second chart image from the variant sample side of the same set of control samples and the corresponding legend question text are fed into the same target model. The target model extracts visual features from the second chart image and performs multimodal understanding and inference based on the legend question text. If the variant sample belongs to the binding-preserving intervention, the legend question text is the same as the base sample. The target model needs to re-parse the legend and complete the inference under the legend layout or visual attribute presentation after the intervention to generate the second output. If the variant sample belongs to the binding-changing intervention, the legend question text has been automatically updated according to the mapping relationship after the variant. The target model needs to identify the target data series and complete the inference according to the updated legend mapping relationship to generate the second output.

[0049] Step S104: When the counterfactual intervention is of the binding-maintaining or binding-changing type, the consistency index and sensitivity index of the target model are calculated based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers. A model diagnostic result is generated based on the consistency index and the sensitivity index. When the counterfactual intervention is of the visibility type, the answerability index and unanswerable index of the target model are calculated based on each of the second outputs and their corresponding question answers. A model diagnostic result is generated based on the answerability index and the unanswerable index. The model diagnostic result also includes diagnostic results output by the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception unanswerable task, respectively.

[0050] It should be noted that when the counterfactual intervention is of the type of binding maintenance or binding change, the consistency index and sensitivity index of the target model are calculated based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers. The generation of model diagnostic results based on the consistency index and the sensitivity index means that, for each control sample under the binding maintenance or binding change intervention, the first output of the base sample is compared with its corresponding question answer, and the second output of the variant sample is compared with its corresponding question answer. For the binding maintenance intervention, if both the first and second outputs are consistent with their respective question answers, the group is deemed to have passed the consistency test. The consistency index is obtained by summarizing the results and calculating the proportion of the number of groups that passed the test to the total number of groups in the binding maintenance category. This index reflects the output stability of the model when the legend mapping relationship remains unchanged. For the binding change intervention, if both the first and second outputs are consistent with their respective question answers, the group is deemed to have passed the sensitivity test. The sensitivity index is obtained by summarizing the results and calculating the proportion of the number of groups that passed the test to the total number of groups in the binding change category. This index reflects whether the model can correctly adjust its output when the legend mapping relationship changes. Based on the values ​​of the consistency index and the sensitivity index, model diagnostic results are generated to verify whether the model output changes reasonably due to changes in the legend mapping relationship, and to diagnose whether the model truly relies on the legend for causal reasoning.

[0051] Furthermore, when the counterfactual intervention is of the visibility type, based on each second output and the corresponding question answer, the answerability index and the abandonment index of the target model are calculated. The generation of the model diagnostic result based on the answerability index and the abandonment index refers to: for each second sample unit under the visibility type intervention, determining whether its legend question is still answerable when the legend is partially or completely invisible. If it is still answerable, the second sample unit is marked as an answerable sample, and the corresponding second output is compared with the question answer. The proportion of consistent comparison results is calculated to obtain the answerability index, which reflects the model's ability to answer answerable questions under the condition that the legend is invisible. If it is not answerable, the second sample unit is marked as an unanswerable sample, and the corresponding second output is compared with the expected rejection behavior. The abandonment accuracy rate and abandonment recall rate are calculated based on the comparison results, and the harmonic mean of the two is used as the abandonment index, which reflects the model's ability to correctly reject unanswerable questions under the condition that the legend is invisible. Based on the values ​​of the responsive and unresponsive indicators, model diagnostic results are generated to verify whether the model has the correct ability to judge unresponsiveness when the legend is not visible.

[0052] Preferably, the step of calculating the consistency index and sensitivity index of the target model based on each of the first outputs and the corresponding question answers, and each of the second outputs and the corresponding question answers, includes: Any first sample unit and its corresponding second sample unit are used as a set of control samples; wherein, if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are consistent, then the control sample is the first control sample; if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are inconsistent, and the answer to the question corresponding to the first sample unit is different from the answer to the question corresponding to the second sample unit, then the control sample is the second control sample. For any control sample, the corresponding first output is compared with the corresponding question answer to obtain the first comparison result, and the corresponding second output is compared with the corresponding question answer to obtain the second comparison result; If the control sample is the first control sample, then when both the first comparison result and the second comparison result are consistent, the control sample is determined to have passed the consistency test; If the control sample is the second control sample, then when the first comparison result and the second comparison result are both consistent, the control sample is determined to have passed the sensitivity test; The proportion of groups that passed the consistency test to the total number of groups in the first control sample was calculated to obtain the consistency index. The proportion of groups that passed the sensitivity test to the total number of groups in the second control sample was calculated to obtain the sensitivity index.

[0053] Preferably, generating model diagnostic results based on the consistency index and the sensitivity index includes: If the consistency index is lower than the preset consistency threshold, the generated model diagnosis result is that the target model has a legend binding stability defect; wherein, the legend binding stability defect indicates that the target model outputs inconsistently when the mapping relationship between visual attributes and data series has not changed; If the sensitivity index is lower than the preset sensitivity threshold, the generated model diagnosis result is that the target model has a legend binding sensitivity defect; wherein, the legend binding sensitivity defect indicates that the target model fails to correctly adjust the output as the mapping relationship between visual attributes and data series changes.

[0054] Preferably, the step of calculating the answerability index and the abandonment index of the target model based on each of the second outputs and the corresponding question answers includes: For any second sample unit, determine whether the legend question corresponding to the second sample unit can still be answered when the legend is partially or completely invisible; If it is still possible to answer, the second sample unit is marked as an answerable sample. The corresponding second output is compared with the corresponding question answer. The proportion of the number of answerable samples with the same comparison result to the total number of answerable samples is calculated to obtain the answerability index. If an answer cannot be given, the second sample unit is marked as an unanswerable sample. The corresponding second output is compared with the expected rejection behavior. The rejection accuracy and rejection recall are calculated based on the comparison results. The harmonic mean of the rejection accuracy and the rejection recall is used as the rejection index. The expected rejection behavior refers to the preset rejection text output by the target model when it is unable to answer based on the legend information.

[0055] Preferably, the model diagnostic results also include diagnostic results output for the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task, respectively, specifically: Based on the comparison results between each first output and the corresponding question answer, the accuracy rate is calculated for each of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task. The corresponding diagnostic results are then output based on the accuracy rate of each type of task.

[0056] In one embodiment, the consistency metric measures whether the model performs consistently across binding-preserving variants (i.e., where the mapping remains unchanged), and is calculated as: Consistency = (Number of correct answers in binding-preserving variants) / (Total number of answers). The sensitivity metric measures whether the model is sensitive to binding-changing variants (i.e., where the mapping changes), and is calculated as: Sensitivity = (Number of correct answers in binding-changing variants) / (Total number of answers). In addition, the following metrics can be calculated to achieve a comprehensive diagnosis of the target model, including: CF1, the harmonic mean of consistency and sensitivity metrics, calculated as: CF1 = 2 × (consistency × sensitivity) / (consistency + sensitivity); Answerable accuracy (CAA), the answerable metric, representing the proportion of times the model answers correctly among answerable questions, calculated as: CAA = (number of times the model answers correctly among answerable questions) / (total number of answerable questions); and Abandoned answer F1 score (AF1), the abandoned answer metric, representing the proportion of times the model correctly rejects unanswerable questions, calculated as: AF1 = 2 × (abandoned answer precision × abandoned answer recall) / (abandoned answer precision + abandoned answer recall).

[0057] In one embodiment, if the first comparison results are inconsistent and the corresponding legend question is a legend parsing task, then the legend parsing is diagnosed as a failure, meaning the model cannot correctly identify the labels or visual attributes in the legend; if the corresponding legend question is a legend binding task, then the legend binding is diagnosed as a failure, meaning the model cannot correctly bind the visual description in the question to the corresponding data series; if the corresponding legend question is a legend conditional reasoning task, then the legend conditional reasoning is diagnosed as a failure; if the corresponding legend question is a legend perception rejection task and the model should reject the answer but answers instead, then the legend perception rejection is diagnosed as a failure; if the model should answer but rejects the answer, then the legend perception rejection is diagnosed as a misjudgment.

[0058] In one embodiment, if the first comparison result is consistent while the second comparison result is inconsistent, it indicates that the model can answer correctly on the original image but makes a mistake on the variant sample. In this case, if the set of comparison samples is the first comparison sample, it is diagnosed that the model has insufficient stability in the legend understanding and correspondence stage, that is, although the model has basic capabilities, it cannot maintain correct output when the legend presentation method changes; if the set of comparison samples is the second comparison sample, it is diagnosed that the model has insufficient sensitivity in the legend understanding and correspondence stage, that is, the model fails to perceive changes in the legend mapping relationship and adjust the output accordingly, indicating that the model does not truly rely on the legend mapping relationship for reasoning.

[0059] This invention constructs multiple first sample units, each containing chart specifications and legend specifications, and generates corresponding first charts. The legend specifications are independent of the chart specifications and can be modified independently, providing a base chart for subsequent generation of control samples. By implementing binding-preservation, binding-change, and visibility-based counterfactual interventions on the legend specifications and constructing corresponding second charts, and updating the question-and-answer mechanism while maintaining the question intent when the mapping relationship changes, while keeping the chart specifications and underlying data of the second chart identical to the first chart, it is possible to obtain a set of control charts by only changing the legend's mapping relationship or visibility. By inputting the set of first and second charts into the target model to obtain corresponding outputs, it is possible to obtain the model's performance in the legend mapping... This invention provides differentiated outputs under three conditions: unchanged mapping relationship, changed mapping relationship, and legend not visible. When the intervention type is binding-maintaining or binding-changing, consistency and sensitivity indices are calculated based on the model output and corresponding answers, respectively. This quantifies the model's output stability when the legend mapping relationship remains unchanged and its output sensitivity when the mapping relationship changes. When the intervention type is visibility-based, answerable and rejection indices are calculated based on the model output and corresponding answers. This quantifies the model's ability to answer answerable questions and correctly reject unanswerable questions when the legend is not visible. Model diagnostic results are generated based on these indices, allowing verification of whether the model output changes reasonably due to legend changes and whether correct rejection occurs when the legend is not visible. Compared to existing technologies that struggle to verify whether model output changes reasonably due to legend changes, this application enables causal diagnosis of the model's legend understanding ability.

[0060] Optionally, in this embodiment of the invention, the step of constructing a plurality of first sample units and generating a corresponding first chart based on each of the first sample units includes: Acquire chart data and generate chart specifications based on the chart data and preset chart configuration; Assign visual attributes to each data series included in the chart specification to obtain the legend specification; Based on the chart specifications and the legend specifications, generate legend questions and corresponding answers; wherein, the legend questions belong to one of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task; The chart specifications, the legend specifications, the legend questions, and the question answers are combined into a first sample unit; The first chart is generated by rendering based on the chart specifications and the legend specifications.

[0061] This invention, through parameterized definition of chart data, visual attribute allocation, and legend task generation, and rendering the chart after combining them into sample units, can generate standardized test samples with legend mapping relationships.

[0062] Optionally, in this embodiment of the invention, the step of implementing counterfactual intervention on each of the legend specifications, constructing multiple second sample units based on the intervention results, and generating corresponding second charts according to each of the second sample units includes: For any legend specification, a counterfactual intervention is performed on the legend specification to obtain a variant legend specification; Compare the mapping relationship defined in the variant legend specification with the mapping relationship defined in the legend specification; If the comparison results are inconsistent, the legend question and the answer are updated according to the mapping relationship defined by the variant legend specification, and the chart specification, the variant legend specification, the updated legend question and the updated answer are combined into a second sample unit; If the comparison results are consistent, the chart specifications, the variant legend specifications, the legend question, and the question answer will be combined into a second sample unit. A second chart is generated based on the chart specifications and the variant legend specifications.

[0063] This invention, through intervention on the legend specifications and comparison with the original mapping relationship, selects to update or retain the question and answer based on the comparison results and constructs variant units. This can distinguish the intervention type and ensure that the variant chart is synchronized with the current legend mapping relationship.

[0064] Optionally, in this embodiment of the invention, the step of inputting each of the first charts and the corresponding legend questions into the target model for chart understanding results in multiple first outputs, including: For any first chart, convert the legend question in the corresponding first sample unit into text format to obtain the first question text; Input the first chart and the first question text into the target model, and receive the first output generated by the target model.

[0065] This invention, by converting the legend of the first chart into text and inputting it into the model along with the chart, can standardize the reasoning process of the basic samples and provide a reference benchmark for subsequent comparisons.

[0066] Optionally, in this embodiment of the invention, the step of inputting each of the second charts and the corresponding legend questions into the target model for chart understanding results in multiple second outputs, including: For any second chart, convert the legend question in the corresponding second sample unit into text format to obtain the second question text; Input the second chart and the second question text into the target model, and receive the second output generated by the target model.

[0067] This invention enables the acquisition of inference results for variant samples under legend intervention conditions by converting the legend question of the second chart into text and inputting it into the model along with the chart.

[0068] Optionally, in this embodiment of the invention, calculating the consistency index and sensitivity index of the target model based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers, includes: Any first sample unit and its corresponding second sample unit are used as a set of control samples; wherein, if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are consistent, then the control sample is the first control sample; if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are inconsistent, and the answer to the question corresponding to the first sample unit is different from the answer to the question corresponding to the second sample unit, then the control sample is the second control sample. For any control sample, the corresponding first output is compared with the corresponding question answer to obtain the first comparison result, and the corresponding second output is compared with the corresponding question answer to obtain the second comparison result; If the control sample is the first control sample, then when both the first comparison result and the second comparison result are consistent, the control sample is determined to have passed the consistency test; If the control sample is the second control sample, then when the first comparison result and the second comparison result are both consistent, the control sample is determined to have passed the sensitivity test; The proportion of groups that passed the consistency test to the total number of groups in the first control sample was calculated to obtain the consistency index. The proportion of groups that passed the sensitivity test to the total number of groups in the second control sample was calculated to obtain the sensitivity index.

[0069] This invention categorizes control samples based on whether the mapping relationship is consistent, and calculates the proportion of both classes of samples where the outputs are correct in both instances, thereby quantifying the output stability and output sensitivity of the model.

[0070] Optionally, in this embodiment of the invention, generating model diagnostic results based on the consistency index and the sensitivity index includes: If the consistency index is lower than the preset consistency threshold, the generated model diagnosis result is that the target model has a legend binding stability defect; wherein, the legend binding stability defect indicates that the target model outputs inconsistently when the mapping relationship between visual attributes and data series has not changed; If the sensitivity index is lower than the preset sensitivity threshold, the generated model diagnosis result is that the target model has a legend binding sensitivity defect; wherein, the legend binding sensitivity defect indicates that the target model fails to correctly adjust the output as the mapping relationship between visual attributes and data series changes.

[0071] By comparing consistency and sensitivity indicators with preset thresholds, this invention can identify failure types where the model outputs are unstable when the legend remains unchanged or fails to adjust correctly when the legend changes.

[0072] Further, the step of calculating the answerability index and the abandonment index of the target model based on each of the second outputs and the corresponding question answers includes: For any second sample unit, determine whether the legend question corresponding to the second sample unit can still be answered when the legend is partially or completely invisible; If it is still possible to answer, the second sample unit is marked as an answerable sample. The corresponding second output is compared with the corresponding question answer. The proportion of the number of answerable samples with the same comparison result to the total number of answerable samples is calculated to obtain the answerability index. If an answer cannot be given, the second sample unit is marked as an unanswerable sample. The corresponding second output is compared with the expected rejection behavior. The rejection accuracy and rejection recall are calculated based on the comparison results. The harmonic mean of the rejection accuracy and the rejection recall is used as the rejection index. The expected rejection behavior refers to the preset rejection text output by the target model when it is unable to answer based on the legend information.

[0073] This invention categorizes samples under visibility-based interventions into answerable and unanswerable samples based on whether the question is still answerable. For answerable samples, the proportion of outputs matching the answers is calculated to obtain an answerability index. For unanswerable samples, the outputs are compared with the expected rejection behavior, and the harmonic mean of the rejection accuracy and rejection recall is calculated as the rejection index. This allows for the quantification of the model's reliability in answering questions that are still answerable under conditions where the legend is not visible, as well as its ability to correctly reject questions that are truly unanswerable, providing a detailed quantitative basis for diagnosing the model's rejection behavior.

[0074] Furthermore, the model diagnostic results also include diagnostic results output for the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task, respectively, specifically: Based on the comparison results between each first output and the corresponding question answer, the accuracy rate is calculated for each of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task. The corresponding diagnostic results are then output based on the accuracy rate of each type of task.

[0075] This invention calculates the accuracy rate and outputs the corresponding diagnostic results for each of the following tasks: legend parsing, legend binding, legend conditional reasoning, and legend perception abandonment task. This allows the model's legend comprehension ability to be decomposed into four specific task stages for independent evaluation, enabling precise localization of model failure points.

[0076] like Figure 2 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; An embodiment of the present invention provides a diagnostic device for a chart understanding model based on legend counterfactual intervention, comprising: a chart generation module 201, a counterfactual intervention module 202, a model reasoning module 203, and a result generation module 204; The chart generation module 201 is used to construct multiple first sample units and generate corresponding first charts according to each first sample unit; wherein, each first sample unit includes chart specifications, legend specifications, legend questions and question answers, the legend specifications are used to define the mapping relationship between visual attributes and data series, and the legend specifications can be modified independently of the chart specifications; The counterfactual intervention module 202 is used to implement counterfactual intervention on each of the legend specifications, construct multiple second sample units based on the intervention results, and generate corresponding second charts according to each second sample unit; wherein, for any second sample unit, if the corresponding mapping relationship changes, the corresponding legend question and question answer are updated, and the updated question intent remains fixed; the types of counterfactual intervention include binding retention, binding change, and visibility; the chart specifications and underlying data of the second chart are the same as those of the first chart; The model reasoning module 203 is used to input each of the first charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple first outputs accordingly; and to input each of the second charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple second outputs accordingly. The result generation module 204 is configured to, when the counterfactual intervention is of the binding-maintaining or binding-changing type, calculate the consistency index and sensitivity index of the target model based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers, and generate a model diagnostic result based on the consistency index and the sensitivity index; when the counterfactual intervention is of the visibility type, calculate the answerability index and the unanswerable index of the target model based on each of the second outputs and their corresponding question answers, and generate the model diagnostic result based on the answerability index and the unanswerable index; wherein, the model diagnostic result also includes diagnostic results output by the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception unanswerable task respectively.

[0077] Optionally, in this embodiment of the invention, the chart generation module 201 includes: a chart specification generation submodule, a legend specification generation submodule, a legend question and answer generation submodule, a first sample unit construction submodule, and a first chart rendering submodule; The chart specification generation submodule is used to acquire chart data and generate chart specifications based on the chart data and preset chart configuration. The legend specification generation submodule is used to assign visual attributes to each data series contained in the chart specification to obtain the legend specification. The legend question-and-answer generation submodule is used to generate legend questions and corresponding answers based on the chart specifications and the legend specifications; wherein, the legend questions belong to one of the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task; The first sample unit construction submodule is used to combine the chart specifications, the legend specifications, the legend questions, and the question answers into the first sample unit; The first chart rendering submodule is used to render and generate a first chart according to the chart specifications and the legend specifications.

[0078] This invention, through parameterized definition of chart data, visual attribute allocation, and legend task generation, and rendering the chart after combining them into sample units, can generate standardized test samples with legend mapping relationships.

[0079] Optionally, in this embodiment of the invention, the counterfactual intervention module 202 includes: a variant legend specification generation submodule, a mapping relationship comparison submodule, a legend question and answer update submodule, a second sample unit construction submodule, and a second chart rendering submodule; The variant legend specification generation submodule is used to perform counterfactual intervention on any legend specification to obtain a variant legend specification; The mapping comparison submodule is used to compare the mapping relationship defined in the variant legend specification with the mapping relationship defined in the legend specification. The legend question and answer update submodule is used to update the legend question and the question answer according to the mapping relationship defined by the variant legend specification if the comparison result is inconsistent, and to combine the chart specification, the variant legend specification, the updated legend question and the updated question answer into a second sample unit. The second sample unit construction submodule is used to combine the chart specification, the variant legend specification, the legend question, and the question answer into a second sample unit if the comparison results are consistent. The second chart rendering submodule is used to render and generate a second chart based on the chart specifications and the variant legend specifications.

[0080] This invention, through intervention on the legend specifications and comparison with the original mapping relationship, selects to update or retain the question and answer based on the comparison results and constructs variant units. This can distinguish the intervention type and ensure that the variant chart is synchronized with the current legend mapping relationship.

[0081] Optionally, in this embodiment of the invention, the model reasoning module 203 includes: a first question text generation submodule and a first model reasoning submodule; The first question text generation submodule is used to convert the legend question in the corresponding first sample unit into text format for any first chart, so as to obtain the first question text; The first model reasoning submodule is used to input the first chart and the first question text into the target model, and to receive the first output generated by the target model.

[0082] This invention, by converting the legend of the first chart into text and inputting it into the model along with the chart, can standardize the reasoning process of the basic samples and provide a reference benchmark for subsequent comparisons.

[0083] Optionally, in this embodiment of the invention, the model reasoning module 203 further includes: a second question text generation submodule and a second model reasoning submodule; The second question text generation submodule is used to convert the legend question in the corresponding second sample unit into text format for any second chart, so as to obtain the second question text; The second model reasoning submodule is used to input the second chart and the second question text into the target model, and to receive the second output generated by the target model.

[0084] This invention enables the acquisition of inference results for variant samples under legend intervention conditions by converting the legend question of the second chart into text and inputting it into the model along with the chart.

[0085] Optionally, in this embodiment of the invention, the result generation module 204 includes: a control sample construction submodule, a result comparison submodule, a consistency test submodule, a sensitivity test submodule, and an index calculation submodule; The control sample construction submodule is used to take any first sample unit and its corresponding second sample unit as a set of control samples; wherein, if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are consistent, then the control sample is a first control sample; if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are inconsistent, and the answer to the question corresponding to the first sample unit is different from the answer to the question corresponding to the second sample unit, then the control sample is a second control sample. The result comparison submodule is used to compare the corresponding first output with the corresponding question answer for any control sample to obtain a first comparison result, and to compare the corresponding second output with the corresponding question answer to obtain a second comparison result; The consistency test submodule is used to determine that the control sample passes the consistency test when the first comparison result and the second comparison result are consistent if the control sample is the first control sample. The sensitivity test submodule is used to determine that the control sample passes the sensitivity test if the control sample is the second control sample and both the first comparison result and the second comparison result are consistent. The index calculation submodule is used to calculate the proportion of the number of groups that passed the consistency test to the total number of the first control sample groups, to obtain the consistency index, and to calculate the proportion of the number of groups that passed the sensitivity test to the total number of the second control sample groups, to obtain the sensitivity index.

[0086] This invention categorizes control samples based on whether the mapping relationship is consistent, and calculates the proportion of both classes of samples where the outputs are correct in both instances, thereby quantifying the output stability and output sensitivity of the model.

[0087] Optionally, in this embodiment of the invention, the result generation module 204 further includes: a first result generation submodule and a second result generation submodule; The first result generation submodule is used to generate a model diagnosis result indicating that the target model has a legend binding stability defect if the consistency index is lower than a preset consistency threshold; wherein, the legend binding stability defect indicates that the target model outputs inconsistently when the mapping relationship between visual attributes and data series has not changed; The second result generation submodule is used to generate a model diagnosis result indicating that the target model has a legend binding sensitivity defect if the sensitivity index is lower than a preset sensitivity threshold; wherein, the legend binding sensitivity defect indicates that the target model fails to correctly adjust its output as the mapping relationship between visual attributes and data series changes.

[0088] By comparing consistency and sensitivity indicators with preset thresholds, this invention can identify failure types where the model outputs are unstable when the legend remains unchanged or fails to adjust correctly when the legend changes.

[0089] Optionally, in this embodiment of the invention, the result generation module 204 further includes: a sample judgment submodule, an answerable index calculation submodule, and an abandoned answer index calculation submodule; The sample judgment submodule is used to determine, for any second sample unit, whether the legend question corresponding to the second sample unit can still be answered when the legend is partially or completely invisible. The answerability index calculation submodule is used to mark the second sample unit as an answerable sample if it is still answerable, compare the corresponding second output with the corresponding question answer, calculate the proportion of the number of answerable samples with the same comparison result to the total number of answerable samples, and obtain the answerability index. The abandoned response index calculation submodule is used to mark the second sample unit as an unanswerable sample if it cannot be answered, compare the corresponding second output with the expected rejection behavior, calculate the abandoned response precision and abandoned response recall based on the comparison result, and use the harmonic mean of the abandoned response precision and the abandoned response recall as the abandoned response index; wherein, the expected rejection behavior refers to the preset rejection text output by the target model when it cannot answer based on the legend information.

[0090] This invention categorizes samples under visibility-based interventions into answerable and unanswerable samples based on whether the question is still answerable. For answerable samples, the proportion of outputs matching the answers is calculated to obtain an answerability index. For unanswerable samples, the outputs are compared with the expected rejection behavior, and the harmonic mean of the rejection accuracy and rejection recall is calculated as the rejection index. This allows for the quantification of the model's reliability in answering questions that are still answerable under conditions where the legend is not visible, as well as its ability to correctly reject questions that are truly unanswerable, providing a detailed quantitative basis for diagnosing the model's rejection behavior.

[0091] Optionally, in this embodiment of the invention, the result generation module 204 further includes: a phased result generation submodule; The phased result generation submodule is used to calculate the accuracy rate according to the comparison results between each first output and the corresponding question answer, for the legend parsing task, legend binding task, legend conditional reasoning task and legend perception abandonment task, and output the corresponding diagnostic results based on the accuracy rate of each type of task.

[0092] This invention calculates the accuracy rate and outputs the corresponding diagnostic results for each of the following tasks: legend parsing, legend binding, legend conditional reasoning, and legend perception abandonment task. This allows the model's legend comprehension ability to be decomposed into four specific task stages for independent evaluation, enabling precise localization of model failure points.

[0093] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the diagram understanding model diagnosis method based on legend counterfactual intervention provided by any of the above-described method embodiments of the present invention.

[0094] In this embodiment of the invention, the chart generation module 201 constructs multiple first sample units containing chart specifications and legend specifications and generates corresponding first charts. The legend specifications are independent of the chart specifications and can be modified independently, providing a base chart for subsequent generation of control samples. The counterfactual intervention module 202 applies binding-preservation, binding-change, and visibility counterfactual interventions to the legend specifications and constructs corresponding second charts. When the mapping relationship changes, the question and answer are updated while maintaining the question intent, and the chart specifications and underlying data of the second chart are kept identical to those of the first chart. This allows for the generation of grouped control charts by changing only the legend's mapping relationship or visibility. The model inference module 203 inputs the grouped first and second charts into the target model to obtain corresponding outputs, enabling the generation of model control samples. The model provides differentiated outputs under three conditions: unchanged legend mapping, changed legend mapping, and legend not visible. When the intervention type is binding maintenance or binding change, the result generation module 204 calculates consistency and sensitivity indices based on the model output and corresponding answers, respectively. This quantifies the model's output stability when the legend mapping remains unchanged and its output sensitivity when the mapping changes. When the intervention type is visibility, the result generation module 204 calculates answerable and rejection indices based on the model output and corresponding answers. This quantifies the model's ability to answer answerable questions and correctly reject unanswerable questions when the legend is not visible. Based on these indices, a model diagnostic result is generated, which can verify whether the model output changes reasonably due to legend changes and whether it correctly rejects questions when the legend is not visible. Compared to existing technologies that struggle to verify whether model output changes reasonably due to legend changes, this application enables causal diagnosis of the model's legend understanding ability.

[0095] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0096] Based on the above embodiment of a diagram understanding model diagnostic method based on legend counterfactual intervention, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a diagram understanding model diagnostic method based on legend counterfactual intervention according to any embodiment of the present invention.

[0097] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0098] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0099] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0100] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the diagram understanding model diagnostic method based on legend counterfactual intervention as described in any of the above-described method embodiments of the present invention.

[0101] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0102] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A diagnostic method for a chart understanding model based on legend counterfactual intervention, characterized in that, include: Multiple first sample units are constructed, and corresponding first charts are generated for each first sample unit; wherein, the first sample unit includes chart specifications, legend specifications, legend questions and question answers, the legend specifications are used to define the mapping relationship between visual attributes and data series, and the legend specifications can be modified independently of the chart specifications; Counterfactual interventions are applied to each of the aforementioned legend specifications. Based on the intervention results, multiple second sample units are constructed, and corresponding second charts are generated for each second sample unit. For any second sample unit, if the corresponding mapping relationship changes, the corresponding legend question and answer are updated, while the updated question intent remains fixed. The types of counterfactual interventions include binding preservation, binding change, and visibility. The chart specifications and underlying data of the second charts are the same as those of the first chart. Each of the first charts and the corresponding legend questions are input into the target model for chart understanding, resulting in multiple first outputs. Similarly, each of the second charts and the corresponding legend questions are input into the target model for chart understanding, resulting in multiple second outputs. When the counterfactual intervention is of the binding-maintaining or binding-changing type, the consistency index and sensitivity index of the target model are calculated based on each of the first outputs and the corresponding question answers, and each of the second outputs and the corresponding question answers, and a model diagnostic result is generated based on the consistency index and the sensitivity index; when the counterfactual intervention is of the visibility type, the answerability index and unanswerable index of the target model are calculated based on each of the second outputs and the corresponding question answers, and a model diagnostic result is generated based on the answerability index and the unanswerable index; wherein, the model diagnostic result also includes diagnostic results output by the legend parsing task, legend binding task, legend conditional reasoning task and legend perception unanswerable task respectively.

2. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The construction of multiple first sample units and the generation of corresponding first charts based on each first sample unit includes: Acquire chart data and generate chart specifications based on the chart data and preset chart configuration; Assign visual attributes to each data series included in the chart specification to obtain the legend specification; Based on the chart specifications and the legend specifications, generate legend questions and corresponding answers; wherein, the legend questions belong to one of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task; The chart specifications, the legend specifications, the legend questions, and the question answers are combined into a first sample unit; The first chart is generated by rendering based on the chart specifications and the legend specifications.

3. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The process involves implementing counterfactual intervention on each of the aforementioned legend specifications, constructing multiple second sample units based on the intervention results, and generating corresponding second charts for each of the second sample units, including: For any legend specification, a counterfactual intervention is performed on the legend specification to obtain a variant legend specification; Compare the mapping relationship defined in the variant legend specification with the mapping relationship defined in the legend specification; If the comparison results are inconsistent, the legend question and the answer are updated according to the mapping relationship defined by the variant legend specification, and the chart specification, the variant legend specification, the updated legend question and the updated answer are combined into a second sample unit; If the comparison results are consistent, the chart specifications, the variant legend specifications, the legend question, and the question answer will be combined into a second sample unit. A second chart is generated based on the chart specifications and the variant legend specifications.

4. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The process involves inputting each of the first charts and its corresponding legend into the target model for chart understanding, resulting in multiple first outputs, including: For any first chart, convert the legend question in the corresponding first sample unit into text format to obtain the first question text; Input the first chart and the first question text into the target model, and receive the first output generated by the target model.

5. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The process involves inputting each of the second charts and the corresponding legend questions into the target model for chart understanding, resulting in multiple second outputs, including: For any second chart, convert the legend question in the corresponding second sample unit into text format to obtain the second question text; Input the second chart and the second question text into the target model, and receive the second output generated by the target model.

6. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The step of calculating the consistency index and sensitivity index of the target model based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers, includes: Any first sample unit and its corresponding second sample unit are used as a set of control samples; wherein, if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are consistent, then the control sample is the first control sample; if the mapping relationship corresponding to the first sample unit and the mapping relationship corresponding to the second sample unit are inconsistent, and the answer to the question corresponding to the first sample unit is different from the answer to the question corresponding to the second sample unit, then the control sample is the second control sample. For any control sample, the corresponding first output is compared with the corresponding question answer to obtain the first comparison result, and the corresponding second output is compared with the corresponding question answer to obtain the second comparison result; If the control sample is the first control sample, then when both the first comparison result and the second comparison result are consistent, the control sample is determined to have passed the consistency test; If the control sample is the second control sample, then when the first comparison result and the second comparison result are both consistent, the control sample is determined to have passed the sensitivity test; The proportion of groups that passed the consistency test to the total number of groups in the first control sample was calculated to obtain the consistency index. The proportion of groups that passed the sensitivity test to the total number of groups in the second control sample was calculated to obtain the sensitivity index.

7. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The step of generating model diagnostic results based on the consistency index and the sensitivity index includes: If the consistency index is lower than the preset consistency threshold, the generated model diagnosis result is that the target model has a legend binding stability defect; wherein, the legend binding stability defect indicates that the target model outputs inconsistently when the mapping relationship between visual attributes and data series has not changed; If the sensitivity index is lower than the preset sensitivity threshold, the generated model diagnosis result is that the target model has a legend binding sensitivity defect; wherein, the legend binding sensitivity defect indicates that the target model fails to correctly adjust the output as the mapping relationship between visual attributes and data series changes.

8. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The step of calculating the answerability index and the abandonment index of the target model based on each of the second outputs and the corresponding question answers includes: For any second sample unit, determine whether the legend question corresponding to the second sample unit can still be answered when the legend is partially or completely invisible; If it is still possible to answer, the second sample unit is marked as an answerable sample. The corresponding second output is compared with the corresponding question answer. The proportion of the number of answerable samples with the same comparison result to the total number of answerable samples is calculated to obtain the answerability index. If an answer cannot be given, the second sample unit is marked as an unanswerable sample. The corresponding second output is compared with the expected rejection behavior. The rejection accuracy and rejection recall are calculated based on the comparison results. The harmonic mean of the rejection accuracy and the rejection recall is used as the rejection index. The expected rejection behavior refers to the preset rejection text output by the target model when it is unable to answer based on the legend information.

9. The diagnostic method for a chart understanding model based on legend counterfactual intervention as described in claim 1, characterized in that, The model diagnostic results also include diagnostic results output separately for the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task, specifically: Based on the comparison results between each first output and the corresponding question answer, the accuracy rate is calculated for each of the following tasks: legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task. The corresponding diagnostic results are then output based on the accuracy rate of each type of task.

10. A diagnostic device for a chart understanding model based on legend counterfactual intervention, characterized in that, include: The module includes a chart generation module, a counterfactual intervention module, a model reasoning module, and a results generation module. The chart generation module is used to construct multiple first sample units and generate corresponding first charts based on each first sample unit; wherein, each first sample unit includes chart specifications, legend specifications, legend questions and question answers, the legend specifications are used to define the mapping relationship between visual attributes and data series, and the legend specifications can be modified independently of the chart specifications; The counterfactual intervention module is used to implement counterfactual intervention on each of the legend specifications, construct multiple second sample units based on the intervention results, and generate corresponding second charts according to each second sample unit. For any second sample unit, if the corresponding mapping relationship changes, the corresponding legend question and question answer are updated, and the updated question intent remains fixed. The types of counterfactual intervention include binding preservation, binding change, and visibility. The chart specifications and underlying data of the second chart are the same as those of the first chart. The model reasoning module is used to input each of the first charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple first outputs accordingly; and to input each of the second charts and the corresponding legend questions into the target model for chart understanding, and obtain multiple second outputs accordingly. The result generation module is configured to, when the counterfactual intervention is of the binding-maintaining or binding-changing type, calculate the consistency index and sensitivity index of the target model based on each of the first outputs and their corresponding question answers, and each of the second outputs and their corresponding question answers, and generate a model diagnostic result based on the consistency index and the sensitivity index; when the counterfactual intervention is of the visibility type, calculate the abandonment index of the target model based on each of the second outputs and their corresponding question answers, and generate the model diagnostic result based on the abandonment index; wherein, the model diagnostic result also includes diagnostic results output by the legend parsing task, legend binding task, legend conditional reasoning task, and legend perception abandonment task respectively.

Images