Data structure-based reasoning with language model neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a response to a query posed about a data structure. One of the methods may include receiving data that represents the data structure; receiving the query about the data structure; generating, using a language model neural network, a response to the query, the generating comprising, at each of one of more iterations: generating one or more network inputs for the iteration that each comprise (i) the query and (ii) data that represents a current data structure for the iteration; processing, by the language model neural network, the one or more network inputs to generate one or more network outputs for the iteration that define a data structure operation that modifies the current data structure; and performing the data structure operation on the current data structure to generate an updated data structure for the iteration.

Description

[0001]Attorney Docket No.: 56113-0543WO1 DATA STRUCTURE-BASED REASONING WITH LANGUAGE MODEL NEURAL NETWORKS CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to U.S. Provisional Application No.63 / 541,283, filed on September 28, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND This specification relates to processing data using neural networks. Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY This specification describes a system implemented as computer programs on one or more computers in one or more locations that obtains a system input that includes a query posed about a data structure and generates a system output that includes a response to the query. Rather than attempting to directly generate the response from structured data maintained in the data structure and the query about the structured data, however, the system executes a data structure-based reasoning process that is guided by the network outputs of a language model neural network over one or more iterations to iteratively update the data structure. In other words, at each iteration of the structure-based reasoning process, the language model neural network generates one or more intermediate network outputs that define a data structure operation, and the system performs the defined data structure operation to generate an updated data structure for the iteration. After the last iteration of the structure-based reasoning process, the system processes a network input that includes a final data structure and the query using the language model neural network to generate a network output that specifies the response to the query. Thus, Attorney Docket No.: 56113-0543WO1 the final response to the query is generated after the data structure initially included in the system input has been modified for one or more times over the structure-based reasoning process. Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Compared with generic reasoning, data structured-based, e.g., table-based, reasoning tasks requires the extraction of underlying semantics from both unstructured data, e.g., free- form questions, and structured or semi-structured data, e.g., tabular data. The complexity of the format of the data structure and the underlying relationship between data stored within individual rows and columns in the data structured increases the difficulty for neural networks to extract information. Using the techniques described in this specification can allow a language model neural network to perform machine learning tasks, and particularly tasks that involve processing structured or semi-structured data, more effectively than existing neural networks. Some techniques described in this specification can allow the neural network to incorporate structured information into a reasoning chain to generate more accurate and reliable outputs. In particular, by executing a reasoning process that is guided by the outputs of the neural network over one or more iterations to iteratively update a given data structure, the neural network is able to generate more accurate responses to queries about the given data structure. Some techniques described in this specification can allow a user to interact with structured information, e.g., a database, using natural language, providing an efficient means for the user to search and retrieve data from the database. The iterative reasoning process in effect can control the way the computer performs the search and retrieval operation on the structured information. Some techniques described in this specification can improve the overall effectiveness in terms of required computational resources (e.g., in terms of processing cycles, memory, or both) for enhancing the performance of a pre-trained neural network on such tasks. By virtue of the disclosed in-context learning technique where demonstrations of atomic data structure operations and an example operation chain for responding to a query are provided as a part of an input prompt to the neural network, the neural network can in-context learn how to execute a reasoning process on an arbitrary data structure, and thus reduces the need to specifically train the neural network on structured data reasoning tasks to learn new values for its parameters. Attorney Docket No.: 56113-0543WO1 The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1 is a diagram of an example system. FIG.2 is an example illustration of operations performed by a dynamic planning engine and operations performed by an argument generation engine. FIG.3 is an example illustration of operations performed by a system to generate a response to a query posed about a data structure. FIG.4 is a flow diagram of an example process for generating a response to a query posed about a data structure. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION FIG.1 is a diagram of an example system. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations that obtains a system input that includes a query 102 posed about a data structure 104 and generates a system output that includes a response 132 to the query 102. The system 100 can obtain the system input in any of a variety of ways. In some implementations, the system 100 can receive the query 102 as input, e.g., as text or audio input, from a user of the system. When the query 102 is received as an audio input, the system 100 can transcribe the audio input into a transcription, i.e., a text representation of the query 102, and use the transcription as the query 102. In some implementations, the system 100 can receive the data structure 104 as an upload from a user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system. The system 100 can receive the data structure 104 as an upload from either the same or different user that provides the query 102. In some implementations, the system 100 can receive an input from the user specifying which data that is already maintained by the system or another system that is accessible by the system should be used as the data structure 104. Attorney Docket No.: 56113-0543WO1 For example, the system 100 can receive an identifier that identifies the data structure 104 as part of or in association with the query 102, and obtain the data structure 104 by using the identifier. Thus, the system 100 need not store the data structure 104. An identifier can be in any format that can be used to locate an actual data structure 104. For example, the identifier may be a uniform resource locator (URL) or another form of network address referencing a data structure that can be accessed by the system 100, Unlike the query 102, which may include unstructured data, e.g., free-form natural language text, the data structure 104 includes structured data. “Structured data” refers to data that has been organized in accordance with a set of rules that define how data is to be organized within the data structure 104. Examples of structured data include data stored in RDF stores, databases, tables, spreadsheets, forms, relational databases, including structured query language (SQL) databases, to name just a few. As a particular example, the data structure 104 can be a table, and the structured data can include tabular data. Tabular data refers to digital data or information that is arranged in rows and columns or in the form of a matrix of cells, where a cell may be defined as an intersection between a horizontal row with a vertical column in a table having rows and columns. Tabular data refers to the arrangement of the data and not to the specific type of data found at a given location in the column, row, or cell. Nor does tabular data refer to the actual data that may be represented by the tabular data. In some implementations, the system 100 can be configured to perform question- answering, where the query 102 represents a question posed about the data structure 104, and the response 132 represents an answer to the question. In particular, the system 100 assesses the information stored in the data structure 104 to identify the answer to the question. For example, the data structure 104 can be a table that includes a “capital” column that represents information about capital cities of different countries, where each row entry in the “capital” column identifies the capital of a corresponding country, and the query 102 can represent the following question: “What is the capital of Liechtenstein?” In this example, the system 100 can leverage the information about capital cities of different countries that is provided in the table to generate the following response 132: “The capital of Liechtenstein is Vaduz.” that represents an answer to the question represented in the query 102. In some implementations, the system 100 can be configured to perform fact-checking, where the query 102 represents a statement posed about the data structure 104 and the response 132 represents a prediction about whether the statement is factually true. In Attorney Docket No.: 56113-0543WO1 particular, the system 100 assesses the information stored in the data structure 104 to predict whether the statement is factually true or not. For example, the data structure 104 can be a table that includes a “capital” column that represents information about capital cities of different countries, where each row entry in the “capital” column identifies the capital of a corresponding country, and the query 102 can represent the following statement: “The capital of Liechtenstein is Vaduz.” In this example, the system 100 can leverage the information about capital cities of different countries that is provided in the table to generate the following response 132: “This is correct.” that represents a prediction about that the statement represented in the query 102 is factually true. To generate the response 132, the system 100 uses a language model neural network 110. The language model neural network 110 can have any appropriate language model neural network architecture that allows the neural network to map an input sequence of text tokens from a vocabulary to an output sequence of text tokens from the vocabulary. The text tokens in the vocabulary can be any appropriate text tokens, e.g., characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. For example, the language model neural network 110 can have any of a variety of Transformer-based neural network architectures, e.g., encoder-only Transformer architectures, encoder-decoder Transformer architectures, decoder-only Transformer architectures, diffusion Transformer architectures, other attention-based architectures, and so on. Examples of such Transformer-based neural network architectures include those described in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few- shot learners. arXiv preprint arXiv:2005.14165, 2020; Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311; Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023; and Gemini Team, et al. Attorney Docket No.: 56113-0543WO1 Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). As a particular example, the language model neural network 110 can be an auto- regressive neural network that auto-regressively generates the output sequence of text tokens by generating each particular text token in the output sequence conditioned on a current input sequence that includes (i) the input sequence followed by (ii) any text tokens that precede the particular text token in the output sequence. More specifically, to generate a particular text token, the language model neural network 110 can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of text tokens. The language model neural network 110 can then select, as the particular text token, a text token from the vocabulary using the score distribution. For example, the language model neural network 110 can greedily select the highest-scoring token or can sample, e.g., using top-k sampling, nucleus sampling or another sampling technique, a token from the distribution. For example, the system 100, or a separate training system, can have trained the language model neural network 110 on training tasks that include a language modeling task, e.g., a task that requires predicting, given a current sequence of text tokens, the next token that follows the current sequence in the training data. As a particular example, the language model neural network 110 can be trained on a maximum-likelihood objective on a large dataset of text, e.g., text that is publicly available from the Internet or another text corpus. Notably, the training of the language model neural network 110 may, but need not, include training language model neural network 110 to perform data structure-based reasoning tasks, training language model neural network 110 on data structure-based training data, or both. The system 100 uses the language model neural network 110 together with a dynamic planning engine 120 and an argument generation engine 130 to perform a data structure- based reasoning process that includes a sequence of one or more data structure-based reasoning iterations (or “iterations” for short) to iteratively update the data structure 104, thereby yielding the final response 132 to the query 102. At a high level, at each iteration of the data structure-based reasoning process, the dynamic planning engine 120 generates a dynamic planning output – and the system 100 provides the dynamic planning output as a first network input that is in the form of an input sequence of text tokens to the language model neural network 110 for processing to generate a first network output that is in the form of an output sequence of text tokens and that Attorney Docket No.: 56113-0543WO1 includes data that defines a data structure operation to be performed on a current data structure. Next, the argument generation engine 130 generates an argument generation output – and the system 100 provides the argument generation output as a second network input that is in the form of an input sequence of text tokens to the language model neural network 110 for processing to generate a second network output that is in the form of an output sequence of text tokens and that includes data that defines one or more arguments of the data structure operation defined by the first network output. The system 100 then performs the data structure operation with the one or more arguments on the current data structure to generate an updated data structure for the iteration. FIG.2 is an example illustration of operations performed by the dynamic planning engine 120 and operations performed by the argument generation engine 130. The dynamic planning engine 120 can perform the operations illustrated in the top half of FIG.2 at a particular iteration in the sequence of one or more iterations. Analogously, the argument generation engine 130 can perform the operations illustrated in the bottom half of FIG.2 at a particular iteration in the sequence of one or more iterations. At the particular iteration, the dynamic planning engine 120 generates a dynamic planning output. The dynamic planning output can include (i) the query 102 included in the system input, (ii) data that represents a current data structure for the particular iteration, and (iii) a current operation chain. The current data structure is a current (modified) version of the data structure 104 as of the particular iteration that has been generated as a result of one or more preceding iterations that have been performed so far during the data structure-based reasoning process. That is, the current data structure is the updated data structure generated in the most recent preceding iteration. The current operation chain includes data that represents a history of the one or more preceding iterations that have been performed so far. The current operation chain can include data that represents (a) data structure operations defined by first network outputs that have been generated by the language model neural network 110 in the one or more preceding iterations; and (b) arguments defined by second network outputs that have been generated by the language model neural network in the one or more preceding iterations. For example, as illustrated in the top half of FIG.2, the dynamic planning engine 120 generates a dynamic planning output that includes a triplet (T, Q, chain), where “T” represents the current data structure for the particular iteration (“intermediate table”), “Q” represents the query (“Which country had the most cyclists finish with in the top3?”), and Attorney Docket No.: 56113-0543WO1 “chain” represents the current operation chain which defines the data structure operation and its arguments in each of the two preceding iterations that have been performed so far (where the data structure operation in the earliest iteration is a “f_add_col” operation having an argument of “Country” while the data structure operation in the more recent iteration is a “f_select_row” operation having arguments of “1, 2, 3”). If the particular iteration is the first iteration, because there is no preceding iteration, the dynamic planning engine 120 generates a dynamic planning output that can include (i) the query 102 included in the system input, (ii) data that represents the data structure 104 included in the system input, and (iii) a beginning tag and, optionally, an initial operation chain, where the beginning tag can be placed at the beginning of the dynamic planning output. The beginning tag can include one or more predetermined text tokens that represent the beginning of the data structure-based reasoning process. For example, the beginning tag can be “[B]”, “[Begin]”, or “”. The initial operation chain can include data that represents no data structure operation has been generated yet. For example, the initial operation chain can include “ ∅” or “[ ]”. In some implementations, the data structure operation is an atomic operation selected from a predefined set of atomic operations that can be performed on a data structure. Each atomic operation can modify any of a variety of aspects of the data structure. For example, as illustrated on right hand side of the top half of FIG.2, the predefined set of atomic operations can include one or more of: an add column operation (“f_add_column()”), a select rows operation (“f_select_column()”), a select columns operation (“f_select_row()”), a group rows operation (“f_group_by()”), or a sort rows operation (“f_sort_by()”). In other examples, the predefined set of atomic operations can additionally or alternatively include other atomic operations including, for example, an add row operation, an edit column operation, an edit row operation, a copy row operation, a copy column operation, and so on. In some of these implementations, the dynamic planning output generated by the dynamic planning engine 120 for the particular iteration can include a listing of the predefined set of atomic operations. Optionally, the dynamic planning output generated by the dynamic planning engine 120 for the particular iteration can also include, for each atomic operation in the predefined set, a demonstration of how the atomic operation can modify one or more aspects of a demonstration data structure. In some implementations, the dynamic planning output generated by the dynamic planning engine 120 for the particular iteration can include a demonstration of how each of Attorney Docket No.: 56113-0543WO1 one or more operation chains are used to generate a demonstration response to a demonstration query. The demonstration can be a few-shot demonstration, where the demonstration can include a relatively small number of, e.g., between 1 and 10, operation chains. As a particular example, the dynamic planning output can be of the form: # (a) Part I - Atomic Operation Demonstration {OPERATION_INSTRUCTION} {DATA_STRUCTURE} Query:{QUESTION} Operation:{OPERATION_AND_ARGUMENTS} Explanation:{INTRODUCION_TO_THE_OPERATION} … # (b) Part II: Operation Chain Demonstration {OPERATION_CHAIN_INSTRUCTION} {DATA_STRUCTURE} Query:{QUESTION} Operation Chain:{COMPLETE_OPERATION_CHAIN} … # (c) Part III: Input Sample {CURRENT_DATA_STRUCTURE} Query:{QUESTION} Candidates:{POSSIBLE_NEXT_OPERATIONS} Operation Chain:{CURRENT_OPERATION_CHAIN} In this example, statements that follow a # are not included in the dynamic planning output (or are optional) and are only included in the example for explanation. Statements below “# (c) Part III: Input Sample” represent a part of the dynamic planning output that includes (i) the query included in the system input, (ii) data that represents the current data structure for the iteration, (iii) a current operation chain, and, optionally, (iv) the predefined set of atomic operations. For example, {QUESTION} can be “Which country had the most cyclists with in the top 10?”, {POSSIBLE_NEXT_OPERATIONS} can be “f_select_row, f_select_column, …”, and {CURRENT_OPERATION_CHAIN} can be “f_add_column(Country) →” where the Attorney Docket No.: 56113-0543WO1 “→” tokens indicate that the current operation chain is an incomplete operation chain for yielding a response to a query. Statements below “# (a) Part I - Atomic Operation Demonstration” but above “…” represent an optional part of the dynamic planning output that includes, for each atomic operation in the predefined set of atomic operations, a demonstration of how the atomic operation can modify one or more aspects of a demonstration data structure. For example, {OPERATION_AND_ARGUMENTS} can be “f_select_column(County, Population)” and {INTRODUCION_TO_THE_OPERATION} can be “The question asks … We only need column Country and Population to answer the question so we use f_select_column to select them”. Statements below “# (b) Part II: Operation Chain Demonstration” but above “…” represent an optional part of the dynamic planning output that includes a demonstration of how each of one or more operation chains are used to generate a demonstration response to a demonstration query. For example, {QUESTION} can be “What was the last year where this team was a part of the USL a-league?” and {COMPLETE_OPERATION_CHAIN} can be “f_add_column(Year) → f_select_row(row 1, row 2) → f_select_column(Year, League) → f_sort_column(Year) → [E]” where the “[E]” tokens indicate that the operation chain is a complete operation chain for yielding a response to a query. At the particular iteration, after having generated the dynamic planning output for the particular iteration by using the dynamic planning engine 120, the system 100 provides the dynamic planning output as a first network input to the language model neural network 110 for processing to generate a first network output which includes data that defines a data structure operation to be performed on the current data structure. For example, as illustrated in the top half of FIG.2, the data structure operation defined by the first network output is a “f_group_by” operation. At the particular iteration, the argument generation engine 130 then generates an argument generation output. The argument generation output can include (i) the query 102 included in the system input, (ii) the data that represents either the data structure included in the system input (when the particular iteration is the first iteration) or the current data structure for the particular iteration (when the particular iteration is a subsequent iteration), and (iii) the data structure operation defined by the first network output generated by the language model neural network 110 for the particular iteration. Attorney Docket No.: 56113-0543WO1 In some cases, the argument generation output can also include one or more possible arguments for the data structure operation. The argument generation engine 130 can generate the one or more possible arguments based on applying keyword extraction or other text analysis techniques on data stored in the current data structure. For example, as illustrated in the bottom half of FIG.2, the argument generation engine 130 generates an argument generation output that includes a triplet (T, Q, f), where “T” represents the current data structure for the iteration (“intermediate table”), “Q” represents the query (“Which country had the most cyclists finish with in the top3?”), and “f” represents the data structure operation (the “f_group_by” operation) defined by the first network output generated by the language model neural network 110 for the particular iteration. As illustrated on right hand side of the bottom half of FIG.2, the argument generation output also includes three possible arguments for the “f_group_by” operation: “Rank”, “Cyclist”, and “Country”, which correspond to the types of information stored along a dimension of the data structure. For example, the argument generation engine 130 can extract the headers of the columns of the table in the example of FIG.2, and use the extracted headers as the possible arguments. As a particular example, the argument generation output can be of the form: {OPERATION_INSTRUCTION} {CURRENT_DATA_STRUCTURE} Query:{QUESTION} For example, the {OPERATION_INSTRUCTION(f)} can be “f_select_row ()”, which is optionally followed by possible arguments including “row1”, “row 2”, “row 3”, and so on. As another example, the {OPERATION_INSTRUCTION(f)} can be “f_sort_by”, which is optionally followed by possible arguments including “from-large-to-small”, “from- small-to-large”, “ascending”, or “descending”, and so on. At the particular iteration, after having generated the argument generation output for the particular iteration by using the argument generation engine 130, the system 100 then provides the argument generation output as a second network input to the language model neural network 110 for processing to generate a second network output which includes data that defines one or more arguments of the data structure operation defined by the first network output. For example, as illustrated in the bottom half of FIG.2, for the “f_group_by” Attorney Docket No.: 56113-0543WO1 operation defined by the first network output, the second network output defines that the argument for the “f_group_by” operation includes “Country”. Thus, in the example of FIG.2, the data structure operation to be performed at the particular iteration “f_group_by(Country)” which defines a group rows operation to group the rows by row entries in the “Country” column, such that the same row entries in the “Country” column are grouped together as a combined row entry, where the combined row entry includes a count of the individual row entries before they are grouped together. FIG.3 is an example illustration of operations performed by the system 100 to generate a response 132 to a query 102 posed about a data structure 104. In the example of FIG.3, the query 102 is a question “Which actor has the most NAACP image awards?”, the data structure 104 is a table which stores data that includes a listing of years in association with actors, motion pictures, and nominees, and the data that represents the data structure 104 can be of the form: / * col : Year | Actor | Motion Picture | Nominees row 1 : 1995 | Al Freeman, Jr | Malcolm X | Delroy Lindo … row 2 : 1996 | Laurence Fishburne Higher | Learning Charles Dutton… row 3 : 1997 | Samuel L. Jackson | A Time to Kill | Blair Underwood… row 4 : 1998 | Morgan Freeman | Amistad | Clarence Williams * / The system 100 then performs a data structure-based reasoning process that includes a sequence of three iterations to iteratively update the data structure 104, thereby yielding the final response 132 to the query 102. FIG.3 illustrates the operations performed in the first iteration of the data structure- based reasoning process. The system 100 uses the dynamic planning engine 120 to generate a dynamic planning output, and provides the dynamic planning output as a first network input to the language model neural network 110 for processing to generate a first network output. The first network output includes data that defines a “f_sel_col()” operation to be performed on the data structure 104. The system 100 uses the argument generation engine 130 to generate an argument generation output, and provides the argument generation output as a second network input to the language model neural network 110 for processing to generate a second network output. The second network output includes data that defines “Year” and “Actor”, as arguments for the “f_sel_col()” operation that is defined by the first network output. Attorney Docket No.: 56113-0543WO1 The system 100 then performs the “f_sel_col(Year, Actor)” operation on the data structure 104 to generate an updated data structure for the first iteration. The updated data structure for the first iteration (“[Intermediate Table 1]”) thus includes two columns: a “Year” column and an “Actor” column. The updated data structure will be used as the current data structure for the next, second iteration. By repeatedly performing iterations in the data structure-based reasoning process in this way, the system 100 can generate an updated data structure for the last iteration. In the example of FIG.3, the data structure operation to be performed in the first iteration is a “f_sel_col(Year, Actor)” operation; the data structure operation to be performed in the second iteration is a “f_group_by(Actor)” operation; and the data structure operation to be performed in the last, third iteration is a “f_sort_by(Country, Descend)” operation. The updated data structure for the last iteration (“[Intermediate Table 3]”) thus includes three columns: a “Group” column, an “Actor” column, and a “Count” column. After the last iteration in the data structure-based reasoning process, the system 100 uses the dynamic planning engine 120 to generate a final dynamic planning output, and provides the final dynamic planning output as a final network input that is in the form of an input sequence of text tokens to the language model neural network 110 for processing to generate a final network output that is in the form of an output sequence of text tokens. The final dynamic planning output can include (i) the query 102 included in the system input, and (ii) data that represents the updated data structure for the last iteration. The final network output generated by the language model neural network 110 defines the response 132 to the query 102. In example of FIG.3, the final network output is “Morgan Freeman” which is the response to the question “Which actor has the most NAACP image awards?” FIG.4 is a flow diagram of an example process 400 for generating a response to a query posed about a data structure. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system, e.g., the system 100 of FIG.1, appropriately programmed in accordance with this specification can perform the process 400. In general, the system receives a system input that includes (i) data that represents a data structure and (ii) a query posed about the data structure, and performs the process 400 to executing a data structure-based reasoning process that includes one or more iterations to generate a system output that includes a response to the query. The data structure includes structured data. Attorney Docket No.: 56113-0543WO1 The system generates, using a dynamic planning engine, a dynamic planning output for a particular iteration (step 402). If the particular iteration is the first iteration in the data structure-based reasoning process, the dynamic planning output can include (i) the query included in the system input, (ii) data that represents the data structure included in the system input, and (iii) a beginning tag. The beginning tag can include one or more predetermined text tokens that represent the beginning of the data structure-based reasoning process. For example, the beginning tag can be “[B]”, “[Begin]”, “”, or the like. If the particular iteration is a subsequent iteration in the data structure-based reasoning process, the dynamic planning output can include (i) the query included in the system input and (ii) data that represents a current data structure for the particular iteration, and (iii) a current operation chain. The current data structure is a current version of the data structure as of the particular iteration that has been generated as a result of one or more preceding iterations that have been performed so far during the data structure-based reasoning process. That is, the current data structure is the updated data structure generated in the most recent preceding iteration. The current operation chain includes data that represents a history of the one or more preceding iterations that have been performed so far in the data structure-based reasoning process. The current operation chain can include data that represents (a) data structure operations defined by first network outputs that have been generated by the language model neural network in the one or more preceding iterations; and (b) arguments defined by second network outputs that have been generated by the language model neural network in the one or more preceding iterations. The system provides the dynamic planning output as a first network input to the language model neural network and processes, by the language model neural network, the first network input to generate a first network output (step 404). The first network output includes data that defines a data structure operation to be performed at the particular iteration on the data structure included in the system input (when the particular iteration is the first iteration) or on the current data structure (when the particular iteration is a subsequent iteration). For example, to define a data structure operation, the first network output can include text that describes the data structure operation, or can alternatively include an identifier that identifies the data structure operation. In some cases, the first network output defines only one data structure operation whereas, in other cases, the first network output defines multiple data structure operations, i.e., it defines the data structure operation, a subsequent data structure operation to be Attorney Docket No.: 56113-0543WO1 performed after the data structure operation, a further subsequent data structure operation to be performed after the subsequent data structure operation, and so on, in that order. In those other cases, only the first data structure operation defined by the first network output will be used, and any remaining subsequent data structure operations may be disregarded. The system generates, using an argument generation engine, an argument generation output for the particular iteration (step 406). The argument generation output can include (i) the query included in the system input, (ii) the data that represents either the data structure included in the system input (when the particular iteration is the first iteration) or the current data structure for the particular iteration (when the particular iteration is a subsequent iteration, (iii) the data structure operation defined by the first network output generated by the language model neural network for the particular iteration, and, optionally, (iv) one or more possible arguments for the data structure operation. The system provides the argument generation output as a second network input to the language model neural network and processes, by the language model neural network, the second network input to generate a second network output (step 408). The second network output includes data that defines one or more arguments of the data structure operation defined by the first network output. For example, to define an argument, the second network output can include text that describes the argument, or can alternatively include an identifier that identifies the argument. The system performs the data structure operation with the one or more arguments on either the data structure included in the system input (when the particular iteration is the first iteration) or the current data structure for the particular iteration (when the particular iteration is a subsequent iteration) to generate an updated data structure for the particular iteration (step 410). The data structure operation is defined by the first network output. The one or more arguments of the data structure operation are defined by second first network output. In some cases, the second network input includes an ending tag that follows the one or more arguments in the output sequence of text tokens whereas, in other cases, the second network input does not include any ending tag. The ending tag can include one or more predetermined text tokens that represent the ending of the data structure-based reasoning process. For example, the ending tag can be “[E]”, “[End]”, “<E>”, or the like. In other words, when the second network input generated by the language model neural network includes the ending tag, the system determines that the particular iteration is the last iteration in the data structure-based reasoning process, and the process 400 proceeds from step 410 to step 412. Attorney Docket No.: 56113-0543WO1 Alternatively, when the second network input generated by the language model neural network does not include any ending tag, the system determines that the particular iteration is not the last iteration in the data structure-based reasoning process, and the process 400 returns from step 410 to step 402, such that a subsequent iteration of the data structure-based reasoning process can be executed. If the subsequent iteration is to be performed, the subsequent iteration will use the updated data structure for the particular iteration that is generated in step 410 as the current data structure. After the last iteration of the one or more iterations in the data structure-based reasoning process, the system generates a final dynamic planning output, and provides the final dynamic planning output as a final network input to the language model neural network for processing to generate a final network output (step 412). The final dynamic planning output can include (i) the query included in the system input, and (ii) data that represents the updated data structure for the last iteration. The final network output generated by the language model neural network defines the response to the query. An example algorithm for generating a response to a query about a data structure that stores structured data based on executing a data structure-based reasoning process that includes one or more iterations is shown below. In this algorithm, the data structure is a table, the structured data includes tabular data, the query is a question, and the response is an answer to the question. However, this example algorithm is broadly applicable to a wider range of data structure (and structured data stored therein), including, for example, RDF stores, databases, spreadsheets, forms, relational databases, including structured query language (SQL) databases, to name just a Attorney Docket No.: 56113-0543WO1 few. Moreover, this example algorithm is broadly applicable to other data structured-based reasoning tasks, e.g., fact-checking tasks where the query is a statement, and the response is prediction about whether the statement is factually true. This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Attorney Docket No.: 56113-0543WO1 A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be Attorney Docket No.: 56113-0543WO1 supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework. Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that Attorney Docket No.: 56113-0543WO1 includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program Attorney Docket No.: 56113-0543WO1 components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. What is claimed is:

Claims

Attorney Docket No.: 56113-0543WO1 CLAIMS 1. A computer-implemented method comprising: receiving data that represents a given data structure; receiving a query about the given data structure; generating, using a language model neural network, a response to the query, the generating comprising, at each of one of more iterations: generating one or more network inputs for the iteration that each comprise (i) the query and (ii) data that represents a current data structure for the iteration; processing, by the language model neural network, the one or more network inputs to generate one or more network outputs for the iteration that define a data structure operation that modifies the current data structure; and performing the data structure operation on the current data structure to generate an updated data structure for the iteration.

2. The method of claim 1, wherein the one or more network outputs for the iteration comprise a first network output that defines an atomic operation to be performed on the given data structure.

3. The method of claim 2, wherein the atomic operation is selected from a given set of atomic operations.

4. The method of claim 3, wherein the given set of atomic operations comprise one or more of: an add column operation, a select rows operation, a select columns operation, a group rows operation, or a sort rows operation.

5. The method of any one of claim 2-4, wherein the first network output also defines atomic operations to be performed in one or more subsequent iterations.

6. The method of any one of claims 2-5, wherein the one or more network outputs for the iteration comprise a second network output that defines one or more arguments of the atomic operation.Attorney Docket No.: 56113-0543WO1 7. The method of any one of claims 2-6, wherein at a first iteration, the one or more network inputs for the iteration comprise a first network input that comprises: (i) the query, (ii) the given data structure for the iteration, and (iii) a beginning tag.

8. The method of any one of claims 2-7, wherein at each subsequent iteration after the first iteration, the one or more network inputs for the iteration comprise a first network input that comprises: (i) the query, (ii) the data that represents the current data structure for the iteration, and (iii) a current operation chain that represents (a) atomic operations defined by first network outputs that have been generated by the language model neural network in one or more preceding iterations, and (b) arguments defined by second network outputs that have been generated by the language model neural network in the one or more preceding iterations.

9. The method of any one of claims 2-8, wherein the one or more network inputs for the iteration comprise a second network input that comprises: (i) the question, (ii) the data that represents the current data structure for the iteration, and (iii) the atomic operation defined by the first network output generated by the language model neural network for the iteration.

10. The method of any one of claims 2-9, wherein generating the one or more network outputs for the iteration comprises: processing, by the language model neural network, the first network input to generate the first network output; and after generating the first network output, processing, by the language model neural network, the second network input to generate the second network output.

11. The method of any one of claims 3-9, wherein the first network input for the iteration further comprises a demonstration of how each of the given set of atomic operations can modify one or more aspects of a demonstration data structure.

12. The method of any one of claims 2-10, wherein the first network input for the iteration further comprises a demonstration of how an operation chain is used to generate a demonstration response to a demonstration query.Attorney Docket No.: 56113-0543WO1 13. The method of any one of claims 1-12, wherein generating the response to the query comprises, after a last iteration of the one or more iterations: generating a final network input that comprises (i) the question and (ii) data that represents an updated data structure for the last iteration; processing, by the language model neural network, the final network input to generate a final network output that defines the response to the query.

14. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any one of claims 1- 13.

15. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-13.

Images