Device, dataset, and computer-implemented method for testing automated reasoning, in particular automated tabular data reasoning

Abstract

To provide a device, a dataset and a computer-implemented method for testing automated reasoning, in particular, automated tabular data reasoning, for determining an answer to a question based on a table including contents of cells in a tabular structure.SOLUTION: The method includes: providing a table, a question, and an answer (300); determining a prediction for an answer by automated reasoning based on contents of cells of the table and the question (304-2, 306-2, 308-2); and determining, depending on a difference between the prediction and the answer, at least two of a robustness metric of automated reasoning, in particular, automated tabular data reasoning, with respect to changes in the tabular structure, an attention metric of automated reasoning, in particular, automated tabular data reasoning, with respect to at least one cell of the table, and a robustness metric with respect to changes in numerical contents of at least one cell of the table (304-3, 306-3, 308-3).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Background technology The present invention relates to an apparatus, a data set, and a computer-implemented method for testing automated reasoning, and in particular automated tabular data reasoning. [Background technology]

[0002] Tabular question answering (TQA) addresses natural language questions related to information organized in tables. TQA systems interact with relational databases through natural language and serve as a fundamental component for processing information across diverse domains, such as science and finance.

[0003] Processing tabular knowledge presents notable challenges. Although tables are structured, there is no uniform table layout for representing a particular type of data. Tables may contain different data types, such as text and numbers, requiring TQA systems to combine common sense based on text with numerical reasoning about structured data. Tables may have nested table structures.

[0004] Recent benchmarks for TQA systems have revealed that current TQA systems produce inconsistent responses with coarse-grained changes in tables and queries.

[0005] Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir R. Radev, 2023, "Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations," Annual Meeting of the Association for Computational Linguistics (Zhao et al., 2023), discloses an exemplary robustness evaluation dataset for benchmarking TQA, including header perturbation, content perturbation, and question perturbation. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir R. Radev, 2023, "Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations." (Annual Meeting of the Association for Computational Linguistics, Zhao et al., 2023) Summary of the Invention [Means for solving the problem]

[0007] Disclosure of the Invention A computer-implemented method for testing automated reasoning, particularly automated tabular data reasoning, for determining an answer to a question based on a table including the contents of cells in a tabular structure includes providing a table, a question, and an answer; determining a prediction for the automated reasoning answer based on the contents of the table cells and the question; and determining at least two of an indicator of robustness of the automated reasoning, particularly automated tabular data reasoning, to changes in the table structure, an indicator of attention of the automated reasoning, particularly automated tabular data reasoning, to at least one cell of the table, and an indicator of robustness to changes in the numerical content of at least one cell of the table, depending on the difference between the prediction and the answer. This means that automated tabular data reasoning includes making an automated inference based on the contents of the table cells and the question to determine a prediction. The indicators provide further insight into automated reasoning, particularly automated tabular data reasoning.

[0008] The method may include modifying the table to obtain a modified table, determining a prediction for an answer by automated inference based on the modified table and the question, and determining an indicator dependent on a difference between the prediction and the answer. The modification is a perturbation. Indicators based on the perturbation may be useful for better understanding automated inference, particularly automated tabular data inference.

[0009] Modifying the table may include shuffling at least two rows and / or at least two columns of the table. This perturbation method can examine whether automated inference, particularly automated tabular data inference, is robust to changes in row and / or column reordering.

[0010] Modifying a table can include shifting rows to the top, middle, or bottom of the table rows, or shifting columns to the left, middle, or right of the table columns. This perturbation method can examine whether changing the position of rows or columns introduces any biases into automated inference, especially automated tabular data inference.

[0011] Modifying the table may include identifying a table cell containing the answer and shifting the row containing the identified cell or shifting the column containing the identified cell, depending on the answer. This perturbation method can consider whether changing the position of the answer introduces any bias into automated inference, especially automated tabular data inference.

[0012] The method may include determining, dependent on the answer and the content of at least one cell of the table, that the answer is retrievable from the content of a single cell of the table, and determining an indicator of robustness of the automated reasoning, particularly the automated tabular data reasoning, to changes in the table structure. The question may be an Extraction Question (EQ) retrievable from a table. The indicator of robustness of the automated reasoning, particularly the automated tabular data reasoning, to changes in the table structure is suitable for assessing the ability of the automated reasoning, particularly the automated tabular data reasoning, to answer the EQ.

[0013] Modifying the table may include determining, depending on the question, the answer, and the content of the table cells, which table cells contain content relevant to the answer, and, in particular, deleting the content of the relevant table cells by replacing the content of the table cells with values ​​indicating empty cells. This perturbation method may explore whether automated reasoning, in particular automated tabular data reasoning, bypasses table cells and utilizes internal knowledge of the automated reasoning, in particular automated tabular data reasoning, in determining the answer.

[0014] Modifying the table may include, among other things, deleting the contents of the table cells by replacing them with values ​​that represent empty cells. This perturbation method may explore whether automated reasoning, particularly automated tabular data reasoning, bypasses the table entirely and utilizes the automated reasoning's, particularly automated tabular data reasoning's, internal knowledge in determining the answer.

[0015] Modifying the table may include determining, depending on the question, the answer, and the table cell content, the table cell containing content relevant to the answer, and shifting the table row or column containing the relevant cell. This perturbation method can consider the extent to which automated reasoning, and in particular automated tabular data reasoning, bypasses table cell values ​​and relies on the automated tabular data reasoning's internal knowledge to determine the answer.

[0016] The method may include determining, depending on the answer and the content of at least one cell of the table, that the answer requires aggregation of multiple cells of the table, and determining an automated tabular data reasoning attention indicator for the at least one cell of the table. The question may be a reasoning question (RQ) that requires aggregation across multiple cell values. The automated reasoning, particularly the automated tabular data reasoning attention indicator, for the at least one cell of the table is suitable for assessing the ability of the automated tabular data reasoning to answer the RQ.

[0017] Modifying the table may include determining at least one cell of a plurality of cells in the table that is associated with an answer, and modifying the numerical content of at least one cell depending on the question, the answer, and the content of at least one cell in the table, particularly to change or maintain the answer to the question. Perturbations that change the numerical content and the answer may consider the extent to which automated reasoning, particularly automated tabular data reasoning, performs correct aggregation and adapts the answer depending on the change in numerical value. Perturbations that change the numerical content and maintain the answer may consider whether automated reasoning, particularly automated tabular data reasoning, articulates the correct answer due to its own bias toward certain numerical values.

[0018] The method for modifying the content may include determining, depending on the question, the answer, and the content of at least one cell of the table, that the answer requires aggregation across multiple numerical values ​​in the table, and determining an indicator of robustness to changes in the numerical content of at least one cell of the table. The indicator of robustness of automated reasoning, particularly automated tabular data reasoning, to changes in numerical content is suitable for assessing the performance of automated reasoning, particularly automated tabular data reasoning, with respect to changes in numerical values.

[0019] An apparatus for testing automated reasoning, in particular automated tabular data reasoning, for determining answers to questions based on tables including the contents of cells in a tabular structure, comprises at least one processor and at least one memory, wherein the at least one memory stores instructions executable by the at least one processor that, when executed by the at least one processor, cause the apparatus to perform the method.

[0020] The computer program comprises computer readable instructions that, when executed by a computer, cause the computer to carry out the method.

[0021] Further embodiments can be derived from the following description and drawings. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram illustrating an apparatus for testing automated tabular data inference. [Figure 2] FIG. 1 is a schematic diagram showing a table. [Figure 3] 1 is a schematic diagram illustrating a flow chart including steps of a method for testing automated reasoning, in particular automated tabular data reasoning; DETAILED DESCRIPTION OF THE INVENTION

[0023] FIG. 1 shows a schematic diagram of an apparatus 100 for testing automated reasoning, and in particular automated tabular data reasoning.

[0024] Automated tabular data reasoning is one example of automated reasoning. Automated tabular data reasoning in this context refers to tabular question answering (TQA). TQA methods or systems interact with relational databases via natural language and serve as a fundamental component for processing information across diverse domains, such as science and finance.

[0025] Exemplary TQA systems are described in the following literature: ·Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. ArXiv, abs / 1709.00103 (Zhong et al., 2017). ·Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics. (Yu et al., 2018) ·Harsh Desai, Pratik Kayal, and Mayank Kumar Singh. 2021. Tablex: A benchmark dataset for structure and content information extraction from scientific tables. In IEEE International Conference on Document Analysis and Recognition. (Desai et al., 2021) ·Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. ArXiv, abs / 2105.07624 (Zhu et al., 2021). is disclosed in.

[0026] Processing tabular knowledge presents notable challenges. Although tables are structured, there is no uniform table layout for representing a particular type of data. Tables may contain different data types, such as text and numbers, requiring TQA systems to combine common sense based on text with numerical reasoning about structured data. Tables may have nested structures.

[0027] Automated tabular data inference can be performed using GPT-3.5 or: TAPEX: Table pretraining via learning a neural sql executor. ArXiv, abs / 2107.07653 (Liu et al., 2021). OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In North American Chapter of the Association for Computational Linbuistics (Jiang et al., 2022b). TaPas: Weakly supervised table parsing via pre-training. In Annual Meeting of the Association for Computational Linguistics (Herzig et al., 2020). LLaMA2: Open foundation and fine-tuned chat models. ArXiv, abs / 2307.09288 (Touvron et al., 2023). Binder: Binding language models in symbolic languages. ArXiv, abs / 2210.02875 (Cheng et al., 2022). It is one of them.

[0028] The apparatus 100 is configured to perform a method for testing automated reasoning, in particular automated tabular data reasoning, to determine answers to questions based on tables including the contents of cells in a tabular structure.

[0029] The apparatus 100 includes at least one processor 102 and at least one memory 104. The at least one processor 102 is configured to execute instructions that, when executed by the at least one processor 102, cause the apparatus 100 to perform the method. The at least one memory 104 stores these instructions.

[0030] A method for testing automated tabular data reasoning is based on tables, questions, and answers, for example, an original table, one or more perturbed tables, and a set of questions and answers to those questions.

[0031] The questions may be extraction questions (EQ) that can be retrieved from a table, or they may be inference questions (RQ) that require aggregation across multiple cell values.

[0032] The test dataset may include, for example, an original table that associates a set of questions with answers to those questions.

[0033] FIG. 2 illustrates a schematic of an exemplary table 200.

[0034] Table 200 includes a tabular structure with columns and rows. Table 200 includes cells 202 with content. Table 200 includes headers 204 for the columns.

[0035] First, the following example original table: Name Floor IDS Tower 55 Fos Tower 32 includes a column with a header "Name," which contains a cell with the content "IDS Tower" and a cell with the content "Fos Tower." This first example table, for example, includes a column with a header "Floor Number," which contains a cell with the content "55," and a cell with the content "32."

[0036] A cell with the content "IDS Tower" and the content "55" is in one row of the first exemplary table, and a cell with the content "Flos Tower" and the content "32" is in another row of the first exemplary table.

[0037] This means that the first example table associates the IDS tower with 55 and the Flos tower with 32.

[0038] An exemplary perturbation of the first exemplary original table would change the structure of the first exemplary table to be as follows: Name Floor Fos Tower 32 IDS Tower 55 Change to.

[0039] This means that the two lines containing the content have been swapped.

[0040] An example set of EQs and answers to the EQs for a test based on a first example original table is "How many stories does Fosh Tower have? [32 stories]", where the EQ is "How many stories does Fosh Tower have?" and the answer to this EQ is "32 stories".

[0041] Second example original table: Name Votes Leslie 15 Olson 4 includes a column with a header "Name" that contains one cell with the content "Leslie" and one cell with the content "Olson." This second example table, for example, includes a column with a header "Votes" that contains one cell with the content "15" and one cell with the content "4."

[0042] The cell with the content "Leslie" and the content "15" is in one row of the second exemplary table, and the cell with the content "Olson" and the content "4" is in another row of the second exemplary table.

[0043] This means that the second example table associates Leslie with 15 and Olson with 4.

[0044] An exemplary perturbation of the second exemplary original table would change the values ​​of the cells of the second exemplary table as follows: Name Number of votes Leslie 15 Olson 361 Change to.

[0045] This means that the cell containing the number 4 has been changed to the number 361.

[0046] An example set of RQs and answers to the RQs for a test based on a second example original table is "Who received the fewest votes? [Olson]", where the RQ is "Who received the fewest votes?" and the answer to this RQ is "Olson".

[0047] FIG. 3 shows a schematic flow chart including the steps of the method.

[0048] The method includes step 300 .

[0049] This step 300 includes providing a table, questions and answers.

[0050] The method may include providing tables, questions and answers from the dataset.

[0051] The method may include grouping questions from the dataset into elicitation questions (EQs) and inference questions (RQs).

[0052] The datasets are: WikiTableQuestions: Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations. In Annual Meeting of the Association for Computational Linguistics (Pasupat and Liang, 2025). WikiSQL: Seq2sql: Generating structured queries from natural language using reinforcement learning. ArXiv, abs / 1709.00103 (Zhong et al., 2017). Sequential Question Answering: Search-based neural structured learning for sequential question answering. In Annual Meeting of the Association for Computational Linguistics. (Iyyer et al., 2017). Tabular And Textual dataset for Question Answering: Tat-qa: A question answering benchmark on a hybrid of tabular and (Zhu et al., 2021). may include:

[0053] The method includes step 302 .

[0054] This step 302 includes determining, depending on the answer and the content of at least one cell in the table, whether the answer is retrievable from the content of a single cell in the table or whether the answer requires aggregation across multiple cells in the table. Step 302 includes determining, depending on the question, the answer, and the content of at least one cell in the table, whether the answer requires aggregation across values ​​in the table.

[0055] If it is determined that the answer is retrievable from the contents of a single cell of the table, step 304 is executed.

[0056] If it is determined that the answer requires the aggregation of multiple cells in the table, step 306 is performed.

[0057] If it is determined that the answer requires aggregation across the values ​​in the table, step 308 is performed.

[0058] The method may include performing steps 300 and 302 to determine a plurality of questions from the dataset.

[0059] This means that step 302 distinguishes between EQ and RQ, whether they require aggregation across numeric values. The answer to EQ can be looked up from a table. The answer to RQ additionally requires aggregation across multiple cell values.

[0060] Step 302 is optional if the dataset contains only one type of question. For example, if the dataset is known to contain only EQs, the method can continue after step 300 with step 304, and multiple questions can be obtained from the dataset. For example, if the dataset is known to contain only RQs, the method can continue after step 300 with step 306, and multiple questions can be obtained from the dataset. For example, if the dataset is known to contain only RQs that require aggregation over numbers in a table, the method can continue after step 300 with step 308, and multiple questions can be obtained from the dataset.

[0061] The data set may include, for example, a first exemplary table: The data set may include, for example, a second exemplary table:

[0062] The dataset may include, for example, the question "How many stories does the Fosh Tower have?" This question is an EQ because it only requires retrieving cell values ​​from the first example table.

[0063] A dataset may include, for example, the question "Who got the fewest votes?" This question is an RQ because it requires comparing the values ​​of multiple cells.

[0064] Step 304 includes step 304-1.

[0065] Step 304-1 involves modifying the table to obtain a modified table.

[0066] The step of modifying the table may include, for example, shuffling at least two rows and / or at least two columns of the table. Modifying the table may include, for example, randomly shuffling the rows and / or columns.

[0067] Modifying the table may include, for example, shifting the target row to the top, center, or bottom of the table rows, or shifting the target column to the left, center, or right of the table columns.

[0068] An exemplary target row in the table includes cells that provide answers. An exemplary target column in the table includes cells that provide answers.

[0069] Cells that provide answers are identified, for example, by applying unigram string matching between the answers and cell values.

[0070] To shift a target row, the method includes, for example, dividing the table into three equal-length sections called top, middle, and bottom. To shift a target column, the method includes, for example, dividing the table into two equal-length sections called front and back.

[0071] This means that, since on average, a table contains more rows than columns, the method uses more partitions for rows than for columns. The method can use an equal amount of partitions for rows and columns, or it can use more partitions for columns than for rows.

[0072] The method includes, for example, deleting target rows or columns from the table and reinserting them at random positions within each partition.

[0073] The step of modifying the table includes, for example, depending on the answer, identifying a cell in the table that contains the answer and shifting the row that contains the identified cell or shifting the column that contains the identified cell.

[0074] Modifying the table may include switching the contents of the table from columns to rows or from rows to columns. To remove the effect of header information in the header, modifying the table may include converting the header to an index.

[0075] Step 304 includes step 304-2.

[0076] Step 304-2 includes determining a prediction for the automated reasoning answer based on the table and the question, and determining a prediction for the automated reasoning answer based on the modified table and the question.

[0077] Step 304 includes step 304-3.

[0078] Step 304-3 involves determining a measure of robustness of the automated tabular data inference to changes in table structure depending on the difference between the prediction and the answer.

[0079] Step 306 includes step 306-1.

[0080] Step 306-1 involves modifying the table to obtain a modified table.

[0081] The step of modifying the table includes, for example, determining, depending on the question, depending on the answer, and depending on the content of the table cell, the table cell containing content related to the answer, and deleting the content of the related cell, which may include replacing the content of the table cell with a value indicating an empty cell.

[0082] Modifying the table may include, for example, deleting the contents of the table cells, which may include replacing the table with a dummy table, for example by replacing the contents of the table cells with values ​​that represent blank cells.

[0083] The step of modifying the table may include determining, depending on the question, depending on the answer, and depending on the content of the table cell, the table cell containing content related to the answer, and shifting the table row or column containing the related cell.

[0084] Shifting a row may involve, for example, shifting a row out of a table and reinserting the row into a random position in the table.

[0085] Step 306 includes step 306-2.

[0086] Step 306-2 includes determining a prediction for the automated reasoning answer based on the table and the question, and determining a prediction for the automated reasoning answer based on the modified table and the question.

[0087] Step 306 includes step 306-3.

[0088] Step 306-3 includes determining an automated tabular data reasoning attention indicator for at least one cell of the table depending on the difference between the prediction and the answer.

[0089] Step 308 includes step 308-1.

[0090] Step 308-1 involves modifying the table to obtain a modified table.

[0091] The step of modifying the table includes determining at least one cell among a plurality of cells of the table that is related to the answer, and modifying the numerical content of the at least one cell depending on the question, depending on the answer, and depending on the content of the at least one cell of the table.

[0092] The numerical content may be modified to change the answer to the question.

[0093] For example table 200, modifying the numerical content for the number of votes for Olson from 4 to 361 would result in the answer to the question "Who received the fewest votes?" being changed from Olson to Leslie.

[0094] The numerical content may be modified to maintain the answer to the question.

[0095] For example table 200, an automated tabular data inference may be the ability to compare 15 votes for Leslie with 4 votes for Olson. In a step of modifying the numerical content, the number of votes for Leslie may be changed from 15 to 1500 without changing the answer.

[0096] Automated tabular data inference may be made more difficult by this change, as 1500 may not be included in the cell value distribution during training of the automated tabular data inference.

[0097] Step 308 includes step 308-2.

[0098] Step 308-2 includes determining a prediction for the automated reasoning answer based on the table and the question, and determining a prediction for the automated reasoning answer based on the modified table and the question.

[0099] Step 308 includes step 308-3, which includes determining a measure of robustness to changes in the numerical content of at least one cell of the table depending on the difference between the prediction and the answer.

[0100] This robustness index is determined, for example, depending on one of the following criteria: Exact match accuracy (Em): Exact match accuracy checks whether the predicted answer of automated inference based on the table and questions from the dataset is identical to the true answer (ground truth answer) from the dataset to the question from the dataset.

[0101] This criterion has been implemented, for example, in "Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Annual Meeting of the Association for Computational Linguistics for evaluating TQA systems (Pasupat and Liang, 2015)," "Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Annual Meeting of the Association for Computational Linguistics (Yang et al., 2022)," and "Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. 2022b. Omnitab: Pretraining with natural and synthetic data for few-shot table-based question answering. In North American Chapter of the Association for Computational Linguistics (Jiang et al., 2022b)."

[0102] Exact match difference (Emd): The exact match difference measures the change in system performance before and after the perturbation (negative values ​​indicate a performance degradation). Emd description (Zhao et al., 2023).

[0103] Both Em and Emd focus on overall system performance.

[0104] Volatility (VP): Volatility measures how much the predictions change before and after a perturbation from an instance-level perspective, and is calculated using the following formula: VP=(C2W+W2C) / N where C2W counts the number of instances where the prediction changed from correct to incorrect due to the perturbation, and W2C is the number of instances where the prediction changed from incorrect to correct. N is the total number of instances. To reduce the effect of randomness in the experiments, for perturbations that include randomness (e.g., random shuffling), the mean and standard deviation of the scores across five runs with different random seeds are reported.

[0105] Yang et al. (2022) describe VP.

[0106] A dataset may be provided for testing automated reasoning, particularly automated tabular data reasoning, to determine answers to questions based on tables.

[0107] The data set includes at least one data field for a table, a question, and an answer.

[0108] The dataset includes at least one data field for prediction of an answer determined by automated reasoning based on the contents of the table cells and the question.

[0109] The dataset includes at least one data field for at least two of: a measure of robustness of automated tabular data reasoning to changes in table structure; a measure of attention of automated tabular data reasoning to at least one cell of the table; and a measure of robustness to changes in the numerical content of at least one cell of the table.

[0110] The contents of the table may be technical content to answer questions about the technical system described by the technical content.

[0111] For example, automated tabular data reasoners are tested for their ability to correctly answer questions about technical systems based on tables.

[0112] For example, automated tabular data reasoning is used to provide answers to questions of users or controllers of technical systems.

[0113] The automated tabular data inference may be related to diagnosing a technical system. The contents of the table may correlate errors in the technical system with their root causes. The questions may relate to questions and the answers may relate to their root causes.

Claims

1. 1. A computer-implemented method for testing automated reasoning, particularly automated tabular data reasoning, for determining answers to questions based on tables including the contents of cells in a tabular structure, comprising: The method comprises: Table (200), providing questions and answers (300); determining a prediction for an automated reasoning answer based on the table cell contents and the question (304-2, 306-2, 308-2); determining (304-3, 306-3, 308-3) at least two of an indicator of robustness of the automated reasoning, particularly automated tabular data reasoning, to changes in table structure, an indicator of attention of the automated reasoning, particularly automated tabular data reasoning, to at least one cell of the table, and an indicator of robustness to changes in the numerical content of at least one cell of the table, depending on the difference between the prediction and the answer; A method comprising:

2. The method comprises: Modifying the table (304-1, 306-1, 308-1) to obtain a modified table; determining a prediction for the answer by the automated reasoning based on the modified table and the question (304-2, 306-2, 308-2); determining the indicators depending on the difference between the predictions and the answers (304-3, 306-3, 308-3); The method of claim 1 , comprising:

3. Modifying the table (304-1) The method of claim 2 , comprising shuffling at least two rows and / or at least two columns of the table.

4. Modifying the table (304-1) 3. The method of claim 2, comprising shifting rows above, to the middle, or below the rows of the table, or shifting columns to the left, to the middle, or to the right of the columns of the table.

5. Modifying the table (304-1) depending on the answer, identifying a cell in the table containing the answer; shifting the row containing the identified cell or shifting the column containing the identified cell; The method of claim 4, comprising:

6. The method comprises: determining (302) that the answer is retrievable from the content of a single cell of the table, depending on the answer and the content of at least one cell of the table; determining (304-3) a measure of the robustness of the automated reasoning, particularly automated tabular data reasoning, to changes in the table structure; 6. The method of claim 3, comprising:

7. Modifying the table (306-1) determining, depending on the question, depending on the answer, and depending on the content of the cell of the table, a cell of the table containing content related to the answer; In particular, deleting the contents of the relevant cells of the table by replacing the contents of the cells with values ​​that indicate empty cells; The method of claim 2 , comprising:

8. Modifying the table (306-1) 3. The method of claim 2, comprising, inter alia, deleting the contents of the cells of the table by replacing the contents of the cells with values ​​that indicate empty cells.

9. Modifying the table (306-1) determining, depending on the question, depending on the answer, and depending on the content of the cell of the table, a cell of the table containing content related to the answer; shifting the row or column of the table containing the relevant cell; The method of claim 2 , comprising:

10. The method comprises: determining (302) that the answer requires an aggregation of multiple cells of the table depending on the answer and the content of at least one cell of the table; determining (306-3) an indicator of attention of the automated reasoning, in particular of automated tabular data reasoning, for at least one cell of the table; 10. The method of any one of claims 7 to 9, comprising:

11. Modifying the table (308-1) determining at least one cell of a plurality of cells in the table associated with the answer; - modifying the numerical content of at least one cell in dependence on the question, in dependence on the answer and in dependence on the content of at least one cell of the table, in particular to change or maintain the answer to the question; The method of claim 2 , comprising:

12. The method comprises: determining (302) that the answer requires aggregation across multiple values ​​in the table depending on the question, the answer, and the contents of at least one cell in the table; determining (308-3) a measure of robustness to changes in the numerical content of the at least one cell of the table; The method of claim 11 , comprising:

13. 1. An apparatus (100) for testing automated reasoning, in particular automated tabular data reasoning, for determining answers to questions based on tables containing the contents of cells in a tabular structure, comprising: The device (100) comprises: at least one processor (102); At least one memory (104); Equipped with The at least one memory (104) stores instructions executable by the at least one processor (102), the instructions, when executed by the at least one processor (102), causing the apparatus (100) to perform the method of any one of claims 1 to 12. An apparatus characterized in that

14. A computer program comprising computer readable instructions for causing a computer to carry out a method according to any one of claims 1 to 12 when the computer program is executed by the computer.

15. In a dataset for testing automated reasoning, particularly automated tabular data reasoning, to determine answers to questions based on tables including the contents of cells in a tabular structure. The data set is a table (200), including at least one data field for questions and answers; The data set is at least one data field for a prediction for an answer determined by automated reasoning based on the contents of the table cells and the question; The data set is at least one data field for at least two of: an indicator of robustness of the automated tabular data reasoning to changes in table structure; an indicator of attention of the automated tabular data reasoning to at least one cell of the table; and an indicator of robustness to changes in the numerical content of at least one cell of the table, determined depending on the difference between the prediction and the answer; A dataset characterized by:

Images