A method and system for constructing counterfactual question-answering based on self-generated graphs

Abstract

The present invention discloses a method and system for constructing counterfactual question and answer of a chart based on bootstrap generation, including extracting visual description and metadata from a chart image; based on the chart visual description text, generating an initial counterfactual hypothesis proposal corresponding to the chart visual description text through a few-sample prompting method of a large language model, and generating a proposal reserve that meets the quantity requirements in a bootstrap manner; based on the chart metadata and the initial question answer pair, sampling a counterfactual hypothesis proposal sample from the hypothesis proposal reserve, and generating a counterfactual question answer pair through a few-sample prompting method of a large language model; after collecting the counterfactual question answer pairs that meet the quantity requirements, verifying the text and numerical value of the counterfactual question answer pair. The present invention can construct a chart question and answer dataset with counterfactual logic, can generate a challenging test set for the performance evaluation of a visual question and answer model, and can provide a diverse training corpus for the training of a visual text model.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal question answering and natural language processing, and in particular to a method and system for constructing counterfactual question answering of a graph based on bootstrapped generation. Background Art

[0002] Chart Question Answering (CQA) is an important task in the field of multimodal learning research. It not only requires parsing the numerical and visual information of the chart at the visual level, but also requires understanding the meaning and logic of the text questions in order to give correct answers.

[0003] Currently, graph question answering is often performed using a pre-trained Visual Question Answering (VQA) model. The role of a graph question answering dataset is to serve as a training dataset and evaluation indicator for the visual question answering model. However, previous graph question answering datasets often have simple question structures and lack complex logic. In recent years, large multimodal models have become increasingly popular, and the performance indicators on previous graph question answering datasets have tended to be saturated. Previous graph datasets lack challenges for large multimodal models, and it is difficult to distinguish models with different capabilities with obvious gaps.

[0004] Counterfactual thinking is a psychological reasoning mode that hypothesizes results by changing actual conditions or backgrounds. It is widely used to simulate complex situations and analyze potential causal relationships. By introducing this thinking mode into the construction of the graph question answering dataset, new counterfactual question contexts can be generated based on existing questions and answers to improve the logical complexity of the dataset.

[0005] There are two popular methods for constructing chart datasets: human-authored methods and template-based methods. Human-authored methods, such as ChartQA, require a large number of annotators to design questions, which is time-consuming and resource-intensive. Template-based methods, such as PlotQA, DVQA, and LEAF-QA, have the disadvantages of insufficient question logic diversity and a single question grammatical structure.

[0006] Large language models, such as the closed-source models of the GPT-4, Claude, and O1 series and the open-source models of the Qwen, LLaMA, and DeepSeek series, have excellent natural language understanding and generation capabilities, and have shown emerging logical reasoning capabilities in the fields of scientific question answering and common sense reasoning; the method of constructing chart datasets based on large language models is still in the exploratory stage. Currently, there is a method of constructing datasets that uses simple one-time prompts to call large language models, but this method relies on prompt design, lacks integration with chart image information, and lacks diversity in generating repeated questions. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and to propose a method and system for constructing counterfactual questions and answers based on self-generated graphs.

[0008] The object of the present invention is achieved through the following technical solution: a method for constructing counterfactual questions and answers based on a graph generated by bootstrapping, the method comprising:

[0009] S1. Randomly sample the initial chart question answering dataset containing question pairs related to chart images and extract visual descriptions and metadata from chart images.

[0010] S2. Based on the visual description text of the chart, generate the initial counterfactual hypothesis proposal corresponding to the visual description text of the chart through the few-shot prompt method of the large language model, and generate a proposal reserve that meets the quantity requirements in a bootstrap manner;

[0011] S3. Based on the graph metadata and the initial question-answer pair, counterfactual hypothesis proposal samples are sampled from the hypothesis proposal reserve, and counterfactual question-answer pairs are generated through the few-sample prompting method of the large language model;

[0012] S4. After generating and collecting the counterfactual question-answer pairs that meet the quantity requirement, verify the text and numerical value of the counterfactual question-answer pairs.

[0013] Furthermore, the extraction of visual descriptions and metadata from the chart image is accomplished by a visual language model, the visual language model comprising an image encoder and a text decoder, and the processing process comprises:

[0014] ;

[0015] ;

[0016] Among them, E is the image feature representation output by the image encoder, T is the output text of the text decoder, is the patched representation of the input graph image, is the i-th patch of the image, and m is the number of patches;

[0017] Indicates that given the image feature E and the previous The probability of generating the tth token under the condition of tokens, A series of tokens for output text, is the i-th mark.

[0018] Furthermore, the image encoder is a bidirectional Transformer architecture neural network, and the text decoder is a unidirectional Transformer architecture neural network; the bidirectional Transformer neural network is composed of L layers of Transformer blocks stacked together, each block inputs m embedding vectors, which pass through the multi-head attention layer, residual connection and normalization layer, forward MLP layer, residual connection and normalization layer in sequence, and finally outputs m vectors. A casual mask is added to the multi-head attention layer in the unidirectional Transformer neural network to create a unidirectional attention mechanism.

[0019] Furthermore, the large language model in S2 adopts an open source text generation model based on Qwen2.5 and LLaMA3.1, which is cleaned and screened based on the existing instruction data set. After selecting instructions related to the graphic document field, QLora fine-tuning method and AdamW are used to fine-tune the instruction data set.

[0020] Furthermore, the counterfactual content logic described by the initial counterfactual hypothesis proposal corresponds to the input visual description text of the chart, and the generation logic of the initial counterfactual hypothesis proposal corresponds to the example in the few-sample prompt.

[0021] Furthermore, the step of generating a proposal reserve that meets the quantity requirement in a bootstrapping manner includes:

[0022] The hypothesis reserve is expanded iteratively in a bootstrapped manner, and the screened high-quality hypothesis proposals are added to the proposal reserve. If the total number of proposals in the reserve does not reach the preset threshold, the dynamic sampling and generation cycle continues. Otherwise, the process is terminated and the final proposal reserve is output. The final hypothesis proposal reserve is classified and organized according to logical patterns and chart types. A visual proposal distribution statistical chart is provided.

[0023] Furthermore, sampling counterfactual hypothesis proposal samples from the hypothesis proposal reserve includes: based on the diversity evaluation of KL divergence or cosine similarity, giving priority to samples with greater differences from the existing proposals in the current reserve.

[0024] Furthermore, the generating of counterfactual question-answer pairs by the small sample prompting method of the large language model includes:

[0025] The input contains visual descriptions and metadata in natural language text form, and the output is a counterfactual question-answer pair in natural language text form;

[0026] The output counterfactual question-answer pair refers to a natural language text containing a counterfactual question and a corresponding answer. The counterfactual question is a new question obtained by adding the counterfactual hypothesis logic described in the hypothesis proposal sample to the original question, where the first half of the question is the counterfactual hypothesis and the second half is the original question. The output counterfactual question-answer pairs are combined to obtain a graphical counterfactual question and answer dataset.

[0027] On the other hand, the present invention specification also provides a system for constructing counterfactual questions and answers based on a bootstrapped generated chart, the system comprising: a chart understanding module, a hypothesis proposal generation module, and a counterfactual question and answer generation module;

[0028] The chart understanding module is used to extract visual descriptions and metadata from chart images;

[0029] The hypothesis proposal generation module is used to generate an initial counterfactual hypothesis proposal corresponding to the visual description text of the chart, and to bootstrap a hypothesis proposal reserve based on the initial hypothesis proposal;

[0030] The counterfactual question and answer generation module is used to sample counterfactual hypothesis proposal samples from the hypothesis proposal reserve, and generate counterfactual question and answer pairs through the few-sample prompting method of the large language model.

[0031] On the other hand, the present invention specification also provides a device for constructing counterfactual questions and answers of graphs based on bootstrap generation, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method for constructing counterfactual questions and answers of graphs based on bootstrap generation.

[0032] Beneficial effects of the present invention:

[0033] 1. The present invention constructs an automated counterfactual chart dataset construction system by using a bootstrapped text generation method combined with chart information extraction and visual understanding modules, thereby solving the defects of existing chart dataset construction methods, such as the labor-intensive labeling and the lack of question logical difficulty and question grammatical diversity.

[0034] 2. The present invention introduces counterfactual logic into the graph question and answer dataset, which improves the diversity and difficulty of the graph question and answer dataset, and improves the application value of the graph question and answer dataset in application scenarios such as multimodal large model pre-training and multimodal algorithm evaluation that have rigid requirements on dataset diversity and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a structural schematic diagram of a system for constructing counterfactual question and answer based on a self-generated chart provided by the present invention.

[0036] Figure 2 It is a schematic diagram of the structure of a chart understanding module in a system for building counterfactual questions and answers based on self-generated charts provided by the present invention.

[0037] Figure 3 It is a structural schematic diagram of a hypothesis proposal generation module in a counterfactual question-answering construction system based on a bootstrapped generated graph provided by the present invention.

[0038] Figure 4 This is a visualization example of a hypothesis proposal reserve result generated by a counterfactual question-answering construction system based on a bootstrapped generated graph provided by the present invention.

[0039] Figure 5 It is a schematic diagram of the structure of a counterfactual question and answer generation module in a counterfactual question and answer construction system based on a self-generated chart provided by the present invention.

[0040] Figure 6 It is a flowchart of an embodiment of constructing counterfactual question and answer based on a self-generated chart provided by the present invention.

[0041] Figure 7 It is a schematic diagram of a device for constructing counterfactual question and answer based on a self-generated graph provided by the present invention. DETAILED DESCRIPTION

[0042] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0043] A method for constructing counterfactual question-answering based on a graph generated in a bootstrapped manner, the method comprising:

[0044] S1. Randomly sample the initial chart question answering dataset containing question pairs related to chart images and extract visual descriptions and metadata from chart images.

[0045] Specifically, the chart image refers to a bar chart, a line chart and a pie chart, which are generally drawn from table data using drawing tools. The chart not only includes metadata information, but also visually displays visualized visual information.

[0046] The visual description included in the output refers to natural language text that describes the visual information of the chart, and expresses the visual type and visual characteristics of the chart.

[0047] The metadata included in the output refers to the specific data described in the chart, which is a table consisting of labels and specific values, and the table is output in the form of natural language text.

[0048] like Figure 2 As shown in the figure, the extraction process of visual description and metadata is completed by the visual language model, which consists of the ViT image encoder (Image Encoder) and the text decoder (Text Decoder). The Encoder-Decoder architecture is used to process the input chart image and output text:

[0049]

[0050]

[0051] The present invention defines that the input graph image is represented as follows after patching: ,in is the i-th patch of the image, m is the number of patches, the ViT image encoder Encoder converts these patches into feature representation E, based on the image feature representation E, the text decoder generates the output text T, in the generative task, the output text T is a series of tokens ,in is the i-th tag. The model predicts the probability of the t-th tag based on the image features E and the previously generated tags, where Indicates that given the image feature E and the previous The probability of generating the tth token under the condition of the tth token.

[0052] Specifically, the image encoder is a neural network with a bidirectional Transformer architecture, and the text decoder is a neural network with a unidirectional Transformer architecture. The bidirectional Transformer neural network is composed of L layers of Transformer blocks. Each block inputs m embeddings, which pass through the multi-head attention layer, residual connection and normalization layer, forward MLP layer, residual connection and normalization layer in sequence, and finally outputs m vectors. A casual mask is added to the multi-head attention layer in the unidirectional Transformer neural network to create a unidirectional attention mechanism.

[0053] Optionally, in some specific examples of the present invention, the visual language model can be used to extract visual descriptions of charts after fine-tuning with the following settings: fine-tuning is performed on two chart question and answer datasets, ChartQA and PlotQA, and a Chart-to-Text chart summary dataset, with a dataset ratio of ChartQA:PlotQA:Chart-to-Text=21.9:224:44.

[0054] Optionally, in some specific examples of the present invention, the visual language model can be used to extract chart metadata after fine-tuning with the following settings: using the chart in the above-mentioned dataset and the metadata included in the dataset as a chart-to-table fine-tuning dataset, and fine-tuning with relative mapping similarity (RMS) as the evaluation indicator.

[0055] S2. Based on the visual description text of the chart, generate the initial counterfactual hypothesis proposal corresponding to the visual description text of the chart through the few-shot prompt method of the large language model, and generate a proposal reserve that meets the quantity requirements in a bootstrap manner;

[0056] The generated hypothesis proposal refers to the natural language text that describes how to construct counterfactual hypothesis logic. For example, "select the value of an element in the graph and assume that its value becomes twice as much as before." The hypothesis in this example describes a counterfactual scenario in which the value of an element doubles. The output hypothesis proposal reserve contains a variety of hypothesis proposals.

[0057] The specific steps for bootstrapping the hypothesis proposal reserve are as follows:

[0058] S2.1. Several highly representative and diverse hypothesis proposal texts are manually designed, covering different logical modes (such as cause and effect, condition, comparison, etc.), and stored in the hypothesis proposal reserve. These initial proposals are reviewed by experts to ensure logical rigor and consistency with the diagram description.

[0059] S2.2. Randomly sample a set of hypothetical proposal samples from the proposal reserve. The sampling process is combined with diversity evaluation based on KL divergence or cosine similarity, and samples with large differences from existing proposals in the current reserve are given priority.

[0060] S2.3. Few-shot hint texts embed additional semantic guidance (such as “generate counterfactual hypotheses that are logically consistent with the input”) to optimize the generation quality of large language models.

[0061] S2.4. Iteratively expand the hypothesis reserve in a bootstrapped manner and add the screened high-quality hypothesis proposals to the proposal reserve. If the total number of proposals in the reserve does not reach the preset threshold (for example, 5,000), continue to execute the dynamic sampling and generation cycle; otherwise, terminate the process and output the final proposal reserve.

[0062] S2.5. The final hypothesis proposal reserve is classified and organized according to logical patterns and chart types to facilitate accurate calling by subsequent modules (such as counterfactual question and answer generation module); a visual proposal distribution statistical chart is provided to optimize the coverage and diversity of the reserve.

[0063] Optionally, in one embodiment of the present invention, the large language model adopts an open source text generation model based on Qwen2.5 and LLaMA3.1, cleans and screens the existing instruction data set, selects instructions related to the field of graphic documents, and then uses QLora fine-tuning method and AdamW to fine-tune the instruction data set.

[0064] Alternatively, if Figure 4 As shown in the figure, pyplot is used to draw a layered sunbrust chart to visualize the distribution of proposals in the hypothesis proposal reserve. For example, the inner layer "line", the middle layer "value" and the outer layer "increase" column in the figure show the hypothesis proposals that increase some values ​​in the line chart. Specifically, spaCy needs to be used to identify the parts of speech and dependencies of the text in the hypothesis proposal. At the same time, Python code needs to be written in combination with the chart metadata to classify the corresponding chart types into different chart types, such as line charts, pie charts, etc.

[0065] S3. Based on the graph metadata and the initial question-answer pair, counterfactual hypothesis proposal samples are sampled from the hypothesis proposal reserve, and counterfactual question-answer pairs are generated through the few-sample prompting method of the large language model;

[0066] The output counterfactual question-answer pair refers to a natural language text containing a counterfactual question and a corresponding answer. The counterfactual question is a new question obtained by adding the counterfactual hypothesis logic described in the hypothesis proposal sample to the original question, for example, "Assume that the value of the shortest column in the chart becomes twice as large as before, then what is the maximum value in the chart?" The first half of the question is the counterfactual hypothesis, and the second half is the original question. The output counterfactual question-answer pairs are combined to obtain a chart counterfactual question and answer dataset.

[0067] like Figure 5 As shown, in this step, the large language model is a generative model for counterfactual question-answer pairs. It is a closed-source large language model based on GPT-4, which has excellent instruction-following capabilities. The few-sample prompt gives multiple examples of counterfactual question-answer synthesis, allowing GPT-4 to follow text instructions and output counterfactual question-answer pairs based on the counterfactual synthesis examples.

[0068] S4. After generating and collecting the counterfactual question-answer pairs that meet the quantity requirement, verify the text and numerical value of the counterfactual question-answer pairs.

[0069] Corresponding to the aforementioned embodiment of a method for constructing counterfactual questions and answers based on bootstrapped graph generation, the present invention also provides an embodiment of a system for constructing counterfactual questions and answers based on bootstrapped graph generation.

[0070] like Figure 1 As shown in Figure 1, the system includes a graph understanding module, a hypothesis proposal generation module, and a counterfactual question and answer generation module:

[0071] The chart understanding module is used to receive an existing chart image and extract chart information. The input is a chart image in the original chart question answering dataset, and the output includes two parts: visual description and metadata.

[0072] The input chart images refer to bar charts, line charts and pie charts, which are generally drawn from table data using drawing tools. The charts not only include metadata information, but also visually display visualized visual information.

[0073] The visual description included in the output refers to natural language text that describes the visual information of the chart, and expresses the visual type and visual features of the chart.

[0074] The metadata included in the output refers to the specific data described in the chart, which is a table consisting of labels and specific values, and the table is output in the form of natural language text.

[0075] like Figure 2 As shown, in the diagram understanding module, the extraction process of visual description and metadata is completed by the visual language model, which consists of a ViT image encoder (Image Encoder) and a text decoder (Text Decoder). The Encoder-Decoder architecture is used to process the input diagram image and output text:

[0076]

[0077]

[0078] The present invention defines that the input graph image is represented as follows after patching: ,in is the i-th patch of the image, m is the number of patches, the ViT image encoder Encoder converts these patches into feature representation E, based on the image feature representation E, the text decoder generates the output text T, in the generative task, the output text T is a series of tokens ,in is the i-th tag. The model predicts the probability of the t-th tag based on the image features E and the previously generated tags, where Indicates that given the image feature E and the previous The probability of generating the tth token under the condition of the tth token.

[0079] Specifically, the image encoder is a neural network with a bidirectional Transformer architecture, and the text decoder is a neural network with a unidirectional Transformer architecture. The bidirectional Transformer neural network is composed of L layers of Transformer blocks. Each block inputs m embeddings, which pass through the multi-head attention layer, residual connection and normalization layer, forward MLP layer, residual connection and normalization layer in sequence, and finally outputs m vectors. A casual mask is added to the multi-head attention layer in the unidirectional Transformer neural network to create a unidirectional attention mechanism.

[0080] Optionally, in some specific examples of the present invention, the visual language model can be used to extract visual descriptions of charts after fine-tuning with the following settings: fine-tuning is performed on two chart question and answer datasets, ChartQA and PlotQA, and a Chart-to-Text chart summary dataset, with a dataset ratio of ChartQA:PlotQA:Chart-to-Text=21.9:224:44.

[0081] Optionally, in some specific examples of the present invention, the visual language model can be used to extract chart metadata after fine-tuning with the following settings: using the chart in the above-mentioned dataset and the metadata included in the dataset as a chart-to-table fine-tuning dataset, and fine-tuning with relative mapping similarity (RMS) as the evaluation indicator.

[0082] The input of the hypothesis proposal generation module is the output of the graph understanding module, which includes visual descriptions and metadata in natural language form, and the output is a hypothesis proposal reserve that stores a number of hypothesis proposals.

[0083] The hypothesis proposal in the output refers to natural language text that describes how to construct counterfactual hypothesis logic, for example, "select the value of an element in the graph and assume that its value becomes twice as much as before." The hypothesis in this example describes a counterfactual scenario in which the value of an element doubles. The output hypothesis proposal reserve includes a variety of hypothesis proposals.

[0084] like Figure 3 As shown, the hypothesis proposal generation module combines the few-sample prompts of the large language model to bootstrap a hypothesis proposal reserve based on the initial hypothesis proposal.

[0085] Specifically, in the hypothesis proposal generation module, the synthesis process of the hypothesis proposal reserve is bootstrapped, and the specific details are as follows:

[0086] 1. A number of highly representative and diverse hypothesis proposal texts are manually designed, covering different logical modes (such as cause and effect, condition, comparison, etc.), and stored in the hypothesis proposal reserve. These initial proposals are reviewed by experts to ensure logical rigor and consistency with the diagram description.

[0087] 2. Randomly sample a group of hypothetical proposal samples from the proposal reserve. The sampling process is combined with diversity evaluation based on KL divergence or cosine similarity, and samples with large differences from existing proposals in the current reserve are given priority.

[0088] 3. The few-shot prompt text embeds additional semantic guidance (such as “generate counterfactual hypotheses that are logically consistent with the input”) to optimize the generation quality of large language models.

[0089] 4. Iterate and expand the hypothesis reserve in a bootstrapped manner, and add the screened high-quality hypothesis proposals to the proposal reserve; if the total number of proposals in the reserve does not reach the preset threshold (for example, 5,000), continue to execute the dynamic sampling and generation cycle; otherwise, terminate the process and output the final proposal reserve.

[0090] 5. The final hypothesis proposal reserve is classified and organized according to logical patterns and chart types to facilitate accurate calling of subsequent modules (such as counterfactual question and answer generation module); a visual proposal distribution statistical chart is provided to optimize the coverage and diversity of the reserve.

[0091] Optionally, in one embodiment of the present invention, the large language model adopts an open source text generation model based on Qwen2.5 and LLaMA3.1, cleans and screens the existing instruction data set, selects instructions related to the field of graphic documents, and then uses QLora fine-tuning method and AdamW to fine-tune the instruction data set.

[0092] Alternatively, if Figure 4 As shown in the figure, pyplot is used to draw a layered sunbrust chart to visualize the distribution of proposals in the hypothesis proposal reserve. For example, the inner layer "line", the middle layer "value" and the outer layer "increase" column in the figure show the hypothesis proposals that increase some values ​​in the line chart. Specifically, spaCy needs to be used to identify the parts of speech and dependencies of the text in the hypothesis proposal. At the same time, Python code needs to be written in combination with the chart metadata to classify the corresponding chart types into different chart types, such as line charts, pie charts, etc.

[0093] The input of the counterfactual question and answer generation module is the output of the graph understanding module and the output of the hypothesis proposal generation module, including visual descriptions and metadata in the form of natural language text, and the output is a counterfactual question answer pair in the form of natural language text.

[0094] The output counterfactual question-answer pair refers to a natural language text containing a counterfactual question and a corresponding answer. The counterfactual question is a new question obtained by adding the counterfactual hypothesis logic described in the hypothesis proposal sample to the original question, for example, "Assume that the value of the shortest column in the chart becomes twice as large as before, then what is the maximum value in the chart?" The first half of the question is the counterfactual hypothesis, and the second half is the original question. The output counterfactual question-answer pairs are combined to obtain a chart counterfactual question and answer dataset.

[0095] like Figure 5 As shown, in the counterfactual question and answer generation module, the large language model is a generation model of counterfactual question and answer pairs. It is a closed-source large language model based on GPT-4, with excellent instruction-following capabilities. The few-sample prompt allows GPT-4 to follow text instructions by giving multiple examples of counterfactual question and answer synthesis, and output counterfactual question and answer pairs by imitating the counterfactual synthesis examples.

[0096] like Figure 6 As shown, the following takes the bar chart in the figure and the initial question answer pair "Question: What is the maximum value in the chart? Answer: 50" as an example to illustrate the method for generating a chart counterfactual question answer pair of the present invention:

[0097] Step 1: Input the chart image into a chart visual language model with data extraction capabilities to obtain chart metadata, including the numerical value and text label corresponding to each column in the bar chart. At the same time, use the chart visual language model with visual question-answering capabilities to obtain the chart visual description, which describes the type of chart is a horizontal bar chart and the information that only has a single data series.

[0098] Step 2: Combine metadata and visual description into prompt text, and then combine hypothesis proposal samples for few-sample prompts. According to the hypothesis proposal generation method specified in the prompt, the large language model outputs a hypothesis proposal corresponding to the chart information, "Select the value of an element in the chart and assume that its value becomes twice the original value."

[0099] Step 3: Combine the metadata and visual description into the prompt text, and use the few-shot prompt to instruct the large language model to output the answer pair of the counterfactual question in the specified format: "Question: Suppose the value of the shortest column in the chart doubles, what is the maximum value in the chart? Answer: 60".

[0100] Corresponding to the aforementioned embodiment of a method for constructing counterfactual questions and answers based on a graph generated in a bootstrap manner, the present invention also provides an embodiment of a device for constructing counterfactual questions and answers based on a graph generated in a bootstrap manner.

[0101] See also Figure 7 An embodiment of the present invention provides a device for constructing counterfactual questions and answers based on self-generated graphs, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a method for constructing counterfactual questions and answers based on self-generated graphs in the above-mentioned embodiment.

[0102] An embodiment of a device for constructing counterfactual questions and answers based on bootstrapped generated graphs provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 7 As shown, it is a hardware structure diagram of any device with data processing capability where a device for constructing a counterfactual question and answer based on a self-generated chart provided by the present invention is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0103] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0104] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0105] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for constructing counterfactual questions and answers based on a self-generated graph in the above embodiment is implemented.

[0106] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0107] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for constructing counterfactual questions and answers based on a self-generated graph.

[0108] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0109] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A method for constructing counterfactual question-answering based on self-generated graphs, characterized in that: The method includes: S1. Randomly sample the initial chart question answering dataset containing question pairs related to chart images and extract visual descriptions and metadata from chart images. S2. Based on the visual description text of the chart, generate an initial counterfactual hypothesis proposal corresponding to the visual description text of the chart through a few-sample prompting method of a large language model, and generate a proposal reserve that meets the quantity requirement in a bootstrap manner; the bootstrap method of generating a proposal reserve that meets the quantity requirement includes: By bootstrapping and iteratively expanding the hypothesis reserve, the high-quality hypothesis proposals that have been screened are added to the proposal reserve; if the total number of proposals in the reserve does not reach the preset threshold, the dynamic sampling and generation cycle continues; otherwise, the process is terminated and the final proposal reserve is output; the final hypothesis proposal reserve is classified and organized according to logical patterns and chart types; and a visual proposal distribution statistical chart is provided; S3. Based on the graph metadata and the initial question-answer pair, counterfactual hypothesis proposal samples are sampled from the hypothesis proposal reserve, and counterfactual question-answer pairs are generated through the few-sample prompting method of the large language model; S4. After generating and collecting the counterfactual question-answer pairs that meet the quantity requirement, verify the text and numerical value of the counterfactual question-answer pairs.

2. A method for constructing counterfactual questions and answers based on self-generated graphs according to claim 1, characterized in that: The extraction of visual description and metadata from the chart image is completed by a visual language model, which includes an image encoder and a text decoder. The processing process includes: E=Encoder(P1,P2,…,P m ); Where E is the image feature representation output by the image encoder, T is the output text of the text decoder, {P1,P2,…,P m } is the patched representation of the input graph image, P i is the i-th patch of the image, and m is the number of patches; P(w t |E,w1,…,w t-1 ) represents the probability of generating the tth tag given the image feature E and the first t-1 tags, {w1,w2,…,w n } is a series of tags for output text, w i is the i-th mark.

3. A method for constructing counterfactual questions and answers based on self-generated graphs according to claim 2, characterized in that: The image encoder is a bidirectional Transformer architecture neural network, and the text decoder is a unidirectional Transformer architecture neural network; the bidirectional Transformer neural network is composed of L layers of Transformer blocks stacked together, each block inputs m embedding vectors, which pass through the multi-head attention layer, residual connection and normalization layer, forward MLP layer, residual connection and normalization layer in sequence, and finally output m vectors. A casual mask is added to the multi-head attention layer in the unidirectional Transformer neural network to create a unidirectional attention mechanism.

4. A method for constructing counterfactual questions and answers based on self-generated graphs according to claim 1, characterized in that: The large language model in S2 adopts open source text generation models based on Qwen2.5 and LLaMA3.

1. It is cleaned and screened based on the existing instruction dataset. After selecting instructions related to the graphic document field, it is fine-tuned on the instruction dataset using the QLora fine-tuning method and AdamW.

5. A method for constructing counterfactual questions and answers based on self-generated graphs according to claim 1, characterized in that: The counterfactual content logic described by the initial counterfactual hypothesis proposal corresponds to the input visual description text of the chart, and the generation logic of the initial counterfactual hypothesis proposal corresponds to the examples in the few-shot prompt.

6. A method for constructing counterfactual questions and answers based on self-generated graphs according to claim 1, characterized in that: The sampling of counterfactual hypothesis proposal samples from the hypothesis proposal reserve includes: based on the diversity evaluation of KL divergence or cosine similarity, giving priority to samples with greater differences from the existing proposals in the current reserve.

7. A method for constructing counterfactual questions and answers based on self-generated graphs according to claim 1, characterized in that: The method of generating a counterfactual question-answer pair by using a few-sample prompting method of a large language model includes: The input contains visual descriptions and metadata in natural language text form, and the output is a counterfactual question-answer pair in natural language text form; The output counterfactual question-answer pair refers to a natural language text containing a counterfactual question and a corresponding answer. The counterfactual question is a new question obtained by adding the counterfactual hypothesis logic described in the hypothesis proposal sample to the original question, where the first half of the question is the counterfactual hypothesis and the second half is the original question. The output counterfactual question-answer pairs are combined to obtain a graphical counterfactual question and answer dataset.

8. A system for constructing a graph counterfactual question-answering dataset based on bootstrapped text generation for implementing the method of any one of claims 1 to 7, characterized in that: The system includes: a graph understanding module, a hypothesis proposal generation module, and a counterfactual question and answer generation module; The chart understanding module is used to extract visual descriptions and metadata from chart images; The hypothesis proposal generation module is used to generate an initial counterfactual hypothesis proposal corresponding to the visual description text of the chart by using a few-sample prompting method of a large language model, and to generate a proposal reserve that meets the quantity requirement in a bootstrap manner; The counterfactual question and answer generation module is used to sample counterfactual hypothesis proposal samples from the hypothesis proposal reserve, and generate counterfactual question and answer pairs through the few-sample prompting method of the large language model.

9. A device for constructing counterfactual questions and answers based on a self-generated graph, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a method for constructing counterfactual questions and answers based on a bootstrapped generated graph is implemented as described in any one of claims 1-7.

Images