A conversational interface for spreadsheet data management

Abstract

Disclosed are implementations, including a method for interacting with a data storage that includes first data comprising data values for rendering in a renderable view and second data comprising specifications for determining the values in the first data. The method includes processing the data values of the first data to yield processed first data including semantic interpretation of the data values, processing the specifications of the second data to yield processed second data including functional definitions of the data values of the first data, accepting a natural language input, and processing the natural language input using the first processed data and the second processed data to respond to the natural language input. The data storage may include a spreadsheet, and the renderable view may include an array of cells of the spreadsheet in which at least some of the cells include renderings of respective data values.

Description

A CONVERSATIONAL INTERFACE FOR SPREADSHEET DATAMANAGEMENTCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 529,000, filed on July 26, 2023, which is incorporated herein by reference.BACKGROUND

[0002] The present invention relates to the use of a natural-language interface to search for and extract information from data structures that include both data that can be seen and further data that defines the data that is hidden.

[0003] Entities, such as individuals and corporations, often maintain important information in electronic spreadsheets. Examples of such spreadsheets include those that arc created using Microsoft Excel. Examples of such information include financial information.

[0004] A user who looks at a spreadsheet sees one or more tabs, each of which includes a table. The table includes information rendered therein. However, this is only the surface information. There exists, within the spreadsheet, information that is generally hidden.

[0005] For example, a user can easily read a particular value in a particular cell of the table. In some cases, the user can also listen to a description of that value. But despite all these ways of learning that value, it is not so easy to learn the underlying formulas or links led to that value. The user interface of a spreadsheet makes it easy to see what a thing is. But it is not quite as good at making it easy to learn why that thing is what it is.

[0006] A typical spreadsheet includes considerable hidden information. For example, there exist relationships between various cells within the same table or across different tables. There also exist various formulas that underlie that which can be seen visually on the spreadsheet. While this hidden information may have been obvious to whoeverdesigned the spreadsheet, those who consume information in that spreadsheet are unlikely to be aware of it. This, in turn, tends to impede their understanding of the information.

[0007] This hidden information nevertheless continues to exist. A diligent user could, with considerable effort, discover it. However, the casual user would be unlikely to search for this hidden information. Regular users may make some effort to search for this hidden information. However, these regular users often find their efforts stymied by the lack of any straightforward way to search for and extract such information from that spreadsheet.SUMMARY

[0008] In a broad aspect, a proposed framework is directed to a spoken or text-based conversational searching interface for use with spreadsheets, or with other data structures that include viewable data (i.c., surface data) and definitional hidden data. Generally, data in a spreadsheet can show tabular data, and that tabular data may be generated by formulas embedded in the spreadsheet. Tabular data may be accessed via natural language interfaces, such as via a question-answering interface or via a natural-language to query language (e.g., NL2SQL) to access values in the table, with column and row headings and / or captions, etc. providing semantic information about what the data in the sheet represents. A natural language interface may then be used to search, access, and retrieve information in a spreadsheet (and optionally manipulate, c.g., edit, amend, insert) data in the sheet, by natural commands. Additionally, the use of a natural language interface allows for a natural navigation of the spreadsheet by performing NL2SQL translation to resolve all relational intents in the query, to find the answer, and highlight the answer on the spreadsheet user interface. Since there is typically a unique one-to-one mapping between the relational database (RDB) table and the spreadsheet table, when the answer in the query is identified in the RDB, a simple API can bring the specific cell in the spreadsheet into focus and highlight it along with the rows and columns that were involved in the SQL processing.

[0009] In addition to the surface data in the spreadsheet, which is the data that is generally visible or that might be printed or otherwise rendered from a spreadsheet, the spreadsheet may have hidden (i.e., recorded and accessible but not generally visible) information that specifies how the data is computed. For example, in an Excel spreadsheet, formulas may be used to relate one column with another. In more complex examples, the table itself may be defined by a database query that accesses a larger data store (e.g., using Microsoft Query in Excel, or queries and other data manipulation specification in Microsoft Power Query also accessible via Microsoft Excel). This “hidden” information forms second data that may be queried and / or manipulated. In a natural language access interface, this second data can be used to resolve relationships such as “determined from” to represent that there is a functional relationship between values in two cells or columns of a table, for example, “How is the profit determined from the total sales in the Boston branch?” which may use the surface structure of the sheet to identify the cells representing “total sales in the Boston branch” and use the hidden structure for the “determined from” and the surface structure to determine which cells correspond to a “profit” value.

[0010] In some implementations, each tab in a spreadsheet is mapped into two tables. One table contains the surface form of the data (i.e., the actual alpha-numerical content visible to a user when passively viewing the spreadsheet without moving the cursor or hovering over any entry) and the other containing the underlying formulas or relationship that generated the specific content viewable by the user. Tables can be converted and loaded into a relational DB. Natural-language-to-SQL (NL2SQL) processing can then be applied to get information into or out of each of the tables. For example, the natural language query “sort companies by revenue” returns the result of “SELECT * FROM current-Table-Surface ORDER BY revenue DESC.” The formulas can be used for explaining the surface forms, e.g., “how was ROI for X calculated” gives back the formula.[Oil] In operation, an information retrieval system includes a data storage including first data (e.g., viewable entries in cells of a spreadsheet) comprising data values for rendering in a renderable view, and second data comprising specifications for determining the values in the first data (e.g., functional formulas based on which some ofthe cells of the spreadsheet are derived). The system processes the data values of the first data to yield processed first data including semantic interpretation of at least some of said data values and processes the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data (e.g., relating the data values by defining some of the data values in terms of other of the data values). Subsequently, the system accepts a natural language input (e.g., a query), and processes the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0012] By way of an example, when the stored data being processed is a spreadsheet, the renderable view may include an array of cells of the spreadsheet in which at least some of the cells include renderings of respective data values. In this example, the array of cells of the spreadsheet includes multiple cells with rendered viewable data, with a first set of cells from the multiple cells including rendered viewable data inputted into the array of cells (e.g., raw input data), and a second set of cells from the multiple cells in which each cell of the second set of cells includes rendered viewable data derived according to a respective formula or function, defined in the specification of the second data, relating each cell of the second set of cells to respective one or more other cells of the multiple cells in the array of cells. Processing the data values, in this example, includes transforming the array of cells into a first resultant arrangement comprising the rendered viewable data corresponding to the data included in the first set of cells and the second set of cells, and processing the specification includes transforming the specification into a second resultant arrangement comprising viewable relationship data (e.g., formula, function, or expression) representing the respective relationship relating the each cell of the second set of cells to the respective one or more other cells.

[0013] The proposed framework converts hard-to-search spreadsheets containing raw and derived data, as well as hidden formulas, into a more standardized multi-table data that can be arranged in relational databases, and can more easily be searched, manipulated, and explained. Searching and explainability are performed, in some embodiments, through natural language (NL) input and output, as described herein in the present disclosure. Data manipulation (of spreadsheet data or relational database data) may beused for process automation in RPA (Robotic Process Automation) and IPA (Intelligent Process Automation).

[0014] The proposed framework can leverage its natural language (question-answer) functionality to search the transformed content of spreadsheets in several modes / forms. For example, transformed spreadsheet data can be searched using SQL (or some other query language).

[0015] The solutions achieved by the proposed framework include one or more of:• Using RDB derived from spreadsheets and NL2SQL techniques;• Discovering the functional relationship of a cell value to other cells, contributing for explaining causes and effects;• Using one-to-one mapping between data elements of the RDB and source data structures (such as spreadsheets) to identify (e.g., highlight) the cell that has an answer responsive to a query (e.g., natural language query);• Mapping the SQL to operations in spreadsheet to arrive at a cell containing an answer (2nd style of navigation);• Integrating two RDB tables into RPA / IPA elements for improved process automation; and• Using a Question-and- Answer system (QA system, such as Pryon Inc.’s QA system) to process spreadsheets (or other data structures with surface data and hidden functional data defining relationships between various data elements) and determine answers to questions therefrom.

[0016] Thus, in some variations, a method for interacting with a data storage that includes first data comprising data values for rendering in a renderable view and second data comprising specifications for determining the values in the first data is provided. The method includes processing the data values of the first data to yield processed first data including semantic interpretation of at least some of said data values, processing the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data, accepting a naturallanguage input, and processing the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0017] In some variations, a system is provided that includes a communication unit to communicate with one or more remote computing devices, one or more storage devices to store data and computer-executable instructions, with the data including first data comprising data values for rendering in a renderable view and second data comprising specifications for determining the values in the first data, and a processor-based controller electrically coupled to the communication unit and the one or more storage devices. The processor-based controller is configured to process the data values of the first data to yield processed first data including semantic interpretation of at least some of said data values, process the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data, accept a natural language input, and process the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0018] In some variations, a non-transitory computer readable media is provided for interacting with a data storage including first data comprising data values for rendering in a rcndcrablc view and second data comprising specifications for determining the values in the first data. The computer readable media is programmed with instructions, executable on one or more processors of a computing system to process data values of a first data to yield processed first data including semantic interpretation of at least some of said data values, process the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data, accept a natural language input, and process the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0019] Practices of the method include those in which the data storage comprises a spreadsheet, the renderable view comprises an array of cells of the spreadsheet, and the cells comprise cells that include renderings of respective data values. Among these practices are those in which the functional definitions comprise computational specifications for determining at least some of the data values for rendering in cells of the spreadsheet and those in which the functional definitions comprise data-accessspecifications for determining at least some of the data values for rendering in cells of the spreadsheet.

[0020] Still other practices include those in which the natural-language input comprises a natural-language query for a data value. In such practices, processing the naturallanguage input comprises using the functional definitions to identify the data value that is to be used to respond to the query.

[0021] Among the other practices of the invention are those in which the data storage comprises a spreadsheet having an array of cells that forms the renderable view. This array includes first and second sets of cells, both of which include those cells that include rendered and viewable data. The first set includes those cells that include rendered viewable data inputted into the array of cells. The second set includes those cells that include rendered viewable data, the rendered viewable data having been derived according to a respective formulation defined in the specification of the second data that relates each cell of the second set to respective one or more cells in the array of cells. In such practices, processing the data values comprises transforming the array of cells into a first resultant arrangement and processing the specification comprises transforming the specification into a second resultant arrangement. As a result, the first resultant arrangement comprises the rendered and viewable data that corresponds to the data included in the first set and the second set and the second resultant arrangement comprises viewable relationship data that represents the respective formulation relating each cell of the second set of cells to the respective one or more other cells.

[0022] Among the foregoing practices are those in which the relationship data comprises formulas that relate cells of the second set as functions of data in the respective one or more other cells and those that further include the step of converting the rendered and viewable data of the first resultant arrangement and the viewable relationship-data of the second resultant arrangement into relational-database data. Among the practices in the latter case are those that further include receiving a natural-language query that relates to the relational-database data and converting the natural-language query into an SQL query that is compatible with the relational-database data. In those practices that include receiving the natural-language query that relates to the relational-database data, there exist alternative practices in which wherein converting the rendered and viewable dataand the viewable relationship data defines one-to-one mapping relationships between each of the multiple cells of the spreadsheet and a respective data element of the relational database data. Such practices thus include the further steps of applying the SQL query to the relational database data to identify one or more data elements responsive to the SQL query and identifying result cells from the multiple cells of the spreadsheet based on the identified one or more data elements responsive to the SQL query and the one-to-one mapping relationships between each of the multiple cells of the spreadsheet and the respective data element of the relational database data. Among those practices that include applying the SQL query, there exist practices with further optional steps, such as determining answer data responsive to the SQL query and presenting the answer data as output through an output interface of a chatbot, determining answer data responsive to the SQL query and presenting the answer data as output through an output interface of a question-and-answer portal, and determining answer data responsive to the SQL query and presenting the answer data as output through rendered output presented in at least one of the results cells identified based on the identified one or more data elements responsive to the SQL query and the one-to-one mapping relationships between each of the multiple cells of the spreadsheet and the respective data element of the relational database data.

[0023] Also among the practices in which the data storage comprises a spreadsheet are those that include the steps of obtaining one or more natural language queries, applying at least part of the one or more natural language queries to one or more unstructured cells in the array of cells of the spreadsheet through a question-and-answer system to obtain respective responses, and determining structured information for the one or more unstructured cells of the spreadsheet based on the respective responses to the one or more natural language queries.

[0024] Embodiments of any of the above method, system, and / or computer-readable media may include at least some of the features described in the present disclosure and may be combined with any other embodiment or variation of the methods, systems, media, and other implementations described herein.

[0025] Other features and advantages of the invention are apparent from the following description, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] These and other aspects will now be described in detail with reference to the following drawings.

[0027] FIG. 1 shows a question-answering system.

[0028] FIG. 2 shows details of a spreadsheet stored in the library shown in FIG. 1.

[0029] FIG. 3 is a schematic diagram of the architecture of an example information retrieval system on which the techniques described herein can be implemented.

[0030] FIG. 4 is a flowchart of an example procedure for interacting with data storage, including to use spreadsheet data to answer natural language queries.

[0031] Like reference symbols in the various drawings indicate like elements.DESCRIPTION

[0032] A number of embodiments to implement a conversational interface for managing and searching spreadsheet data are described. These implementations can be used in conjunction with implementations of the question-and-answer-based document retrieval system described in WO 2021 / 263138, titled “DOCUMENT PROCESSING AND RESPONSE GENERATION SYSTEM,” published on December 30, 2021, which is incorporated herein by reference.

[0033] FIG. 1 shows a question-answering system 100 that serves several entities, each of which has its own local network 150a- 150n. Examples of an “entity” include a corporation, a government agency or division, an academic or religious institution, a research laboratory, and a military organization. Members of such an entity would submit a natural-language query 172 via a link 170, 176 and receive a natural-language response.

[0034] As shown in FIG. 1, the question-answering system 100 includes a documentprocessing agent 110. In some embodiments, the document-processing agent 110 is an Al-based agent.

[0035] A variety of communication interfaces enable communication between the document-processing agent 1 10 and an entity’s local network 150a-150n. These include a server interface 120, an administration interface 125, a user-query interface 130, and an expert interface 132.

[0036] Embodiments include those that implement the document-processing agent 110 on an independent remote server that serves multiple entities, each with its own corresponding local network 150a-150n. Other embodiments implement the documentprocessing agent 110 at a dedicated node within an entity’s local network 150a rather than at a remote location.

[0037] A typical entity maintains source documents 162a-162d in a document library 160. Examples of such source documents 162a-162d include various user guides, spec sheets, memoranda, and e-mails, technical documentation relevant to the customer’s operations, administrative documents such as human-resource documents, and all other types of written documents in electronic form. Among these source documents 162a- 162d are such documents as a spreadsheet 162a.

[0038] Authorized users access the entity’s local network 150a using user stations 154a-c within the network 150a. An administrator uses an administrator station 154 to control access to the source documents.

[0039] The source documents 162a-162d themselves are rarely conducive to efficient searching. To promote more efficient searching, a document-ingestion engine 126 carries out an “ingestion process” on the source documents 162a-162d. This ingestion process maps the source documents 162a-162d into document objects 142a-142n, which it then stores in a repository 140. The document objects 142a- 142n, which are sometimes referred to as “document-object models” or “DOM,” are more readily searchable than the source documents 162a-162d.

[0040] The mapping of a source document 162a is typically preceded by segmenting the source document 162a into segments, with two-hundred- word segments being particularly common. Segmentation is performed according to various rules for adjoining content from various parts of the source document 162a into discrete segments. An example of a segmenting rule is one that constructs segments using a sliding window of afixed or variable length that combines one or more headings preceding the content captured by the sliding window. Rules of this type thereby create a contextual association between one or more headings and the content captured by the window. Such a rule ensures that the transformation performed on a segment combines important contextual information with content located remotely from the segment being processed.

[0041] In response to a query 172, the question-answering system 100 searches through a cache of answers to frequently-asked questions and provides an answer, if available.

[0042] If no answer is readily available, the question-answering system 100 transforms the query 172 into a format compatible with the format of document objects 142a- 142n to identify portions in one or more of the document objects 142a- 142n that may contain the answer to the user’s query 172. Embodiments include those in which the transformations of the query 172 takes place locally at the entity’s network 150a and those in which it takes place at a central server.

[0043] As noted above, among the source documents is a spreadsheet 162a, details of which are shown in FIG. 2.

[0044] As shown in FIG. 2, a spreadsheet 162a comprises tables 164, as shown in FIG. 2. The spreadsheet 162a stores two kinds of information: surface information 166 and subsurface information 168.

[0045] Upon opening a spreadsheet 162a with a suitable application, one sees a table rendered on a display. The table has cells with data values therein. These data values are rendered in the cell. The entire view that one sees is therefore a “renderable view.” The surface information 166 comprises that data that has been rendered on the table.

[0046] However, a spreadsheet 162a contains far more information than what one can see visually on a display. This invisible body of information, referred to herein as the subsurface information 168, comprises the remaining information on the spreadsheet 162a.

[0047] Although the subsurface information 168 is not visible, it nevertheless has a significant effect on the surface information 166 for much the same reason that dark matter has a significant influence on galactic structure. This subsurface information 168includes such features as formulas or algorithms used to arrive at the surface information 166 and relationships between different parts of the surface information 166. This subsurface information 168 therefore comprises specifications for determining what values are to be in the surface information 166. This subsurface information 168 thus plays a significant role in knitting the surface information 166 together.

[0048] The ingestion engine 126 is further configured to carry out certain steps as part of ingesting a spreadsheet 162a from the document library 160 to form a corresponding spreadsheet document-object 142a in the repository 140. These additional steps capture both the surface information 166 that is visible on the spreadsheet 162a and subsurface information 168 that knits the surface information 166 together. As a result, the spreadsheet document-object 142a derived from the spreadsheet 162a begins to acquire some of the properties of a relational database. This makes it possible to transform a natural-language query into a structured query language (e.g., ‘‘SQL) that can then be used to interrogate the spreadsheet document-object 142a.

[0049] The ingestion engine 126 is configured to process both the surface information 166 and the subsurface information 168.

[0050] When processing the surface information 166, the ingestion engine 126 generates processed surface-information 180 that comprises semantic interpretations of data values in the surface information 166. It then incorporates those semantic interpretations into the spreadsheet document-object 142a. In some cases, the processed surface-information 180 would include headers for rows and columns. In other cases, if doing so would promote efficiency in searching, the ingestion engine 126 moves some metadata visible in the spreadsheet 162a into another table. In such cases, doing so promotes the ability to convert the tables thus created into a relational database.

[0051] For those cases in which the spreadsheet 142a also has subsurface information 168, the ingestion engine 126 processes the specifications found in the subsurface information 168 to generate processed subsurface- information 182. This processed subsurface information 182 comprises functional definitions of the data values that were part of the surface information 166. It then incorporates this sub-surface information 182, including those functional definitions into the spreadsheet document-object 142a.

[0052] The surface information 166 and the subsurface information 168 are organized in the spreadsheet document-object 142a in a way that mimics a relational database. Accordingly, natural-language queries seeking information that was originally included in the spreadsheet 162a can instead be carried out by simply converting the naturallanguage query into a corresponding query in a structured query language and interrogating the spreadsheet document-object 142a.

[0053] A schematic diagram of the architecture of an example information retrieval system on which the techniques described herein (including spreadsheet-to relational database conversion, and searching and retrieving of data maintained in such spreadsheets and / or relational database platforms) can be implemented is provided in FIG. 3.

[0054] Because enterprises throughout the world extensively use spreadsheets with customized designs, interacting with such spreadsheets by users not familiar with the intricacies of the customized spreadsheet structures can become complicated. The proposed framework mitigates this problem by transforming complex spreadsheets into tables. The resulting tabular representation can be converted into relational form representations that arc compatible with natural language queries.

[0055] The user interface of the information retrieval platform described herein allows a user to present questions (in a conversation mode) about the data, the data model, or relationships between the data. The natural language input that is handled by the conversation interface of the platform lets users access and modify data, to insert information and otherwise manipulate tabular data generated from a spreadsheet. If there are interrelationships within a table, like formulas, or interrelationships between sheets of the spreadsheet, the framework allows the user to search a searchable data model that can be accessed through natural language query (questions).

[0056] As discussed herein, the present framework implements a mechanism to transform spreadsheet data (which typically is usable on dedicated interfaces of the associated spreadsheet application) into a more widely usable form that captures both the surface data that would be viewable using the normal spreadsheet interface (i.e., the current values visible in the cells of the spreadsheet) and the formulation / functional datathat represents the relationship between different cells (e.g., the formulas defining the values of the different cells as a function of values in other cells). The present framework is generally used in conjunction with structured data (e.g., spreadsheet data arranged in some pre-determined way into cells in a table comprising columns and rows). However, and as will be discussed below, in various examples, techniques implemented for the proposed framework may also be applied to structure unstructured data.

[0057] The framework allows a user to interact with table data derived (transformed, e.g., by the document ingestion engine of the information retrieval platform) from spreadsheet data by asking relational questions. For example, the user query interface can perform natural language (NL) to SQL (NL2SQL) mapping to search and extract responses to natural language questions.

[0058] As noted, the proposed implementations are configured to transform a tab in a spreadsheet into multiple (e.g., two or more) tables in which at least one of the resultant tables (the “first table”) contains the surface data that was rendered in the passive view of the spreadsheet, and at least one separate table (the “formula” or “functional ” table) that includes the underlying information about the formulas and functions that control the output of different cells of the table or spreadsheet. The functions / formulas defined for certain cells of the spreadsheet calculate the sales amount of each sales-person for a given month (adding daily values). The separate table (corresponding to hidden information of a spreadsheet) could also include relational data that would be utilized when, for example, it is desirable to identify the sales person with the best result in a given month. This is a relational function that in RDBs are available as one of the operations, and in spreadsheet can be performed by sorting a specific column segment.

[0059] In situations where a spreadsheet has only surface data (e.g., all data is raw data received as input), only a single table may be generated (that includes numerical and descriptive values, and optionally metadata, e.g., column and row headings) or alternatively a black / empty formula and function table (that would typically include the hidden formulas and functions data of the spreadsheet). In the latter scenario, some metadata from the first table (e.g., column and row names / headings) may be moved to the second table, to thus allow the two (or more) tables derived from a spreadsheet to be converted into relational database data.

[0060] The resultant table(s) are converted and arranged as relational database (RDB) data stored into one or more relational databases. Subsequently, queries to determine information that was originally included in the spreadsheet can be performed by submitting a natural language query that is converted to SQL (according to an NL2SQL process) and performed on the one or more relational database instead of on the spreadsheet. Thus, NL2SQL are performed to determine unambiguous operations that need to be performed on the RDB to execute the query. As noted above, during conversion of the spreadsheet data to RDB data, one-to-one mappings between different cells of the spreadsheet and data elements of the RDB data can be determined. Consequently, after submitting an SQL search or a NL2SQL search to be performed on the RDB data, and determining search results (e.g., RDB data elements responsive to the SQL or NL2SQL search), the one-to-one mapping relationship between the spreadsheet data and the RDB data can be used to identify one or more spreadsheet cells corresponding to the search results from the SQL or NL2SQL queries). As a result, the position on the spreadsheet of the answers to the query that was applied to the RDB data can be determined. The framework described herein therefore allows a user to navigate to the relevant cell(s) in a spreadsheet matching the SQL or NL2SQL query results determined from the RDB data. Additionally, in many cases, it is possible to work out a deterministic mapping of the RDB SQL operation into a spreadsheet operation such as sorting when the max or min values are sought. Since the sequence of going from NL to SQL, the SQL operation to RDM, and the unique mapping of RDB cells to spreadsheet cells are available, the proposed framework provides at least one complete solution to navigate the spreadsheet in addition to just bringing back an answer. Subsequently, a query language (e.g., SQL) can be used to retrieve information from the tables (or alternatively from relational databases that were created or updated using the resultant tables generated from the spreadsheet). Thus, for example, operations such as sorting based on revenue can be performed using the table form (e.g., the surface table) or from relational database data generated from the spreadsheet. Additionally, the proposed framework also allows users to submit queries that explain how something (e.g., revenue) was calculated. In such situations, the platform may use the formula table to extract and return information responsive to a query that relates to the actual way the surface valueswere computed. The platform also allows for information in the tables to be stored in, and retrieved from, relational databases.

[0061] The mapping of a natural language query to an SQL query involves mapping from the form of the query to a well- structured database query. For many natural language queries, such as “What is the X of Y,” the mapping into a database query may involve finding a table named T, with a field name F corresponding to X, and finding a field G in which Y is a value - the query then can have the form “SELECT F FROM T WHERE G=Y.”

[0062] In an example of performing a natural language query of data generated from spreadsheet data, a user can select a cell to determine its value and can then submit a query to inquire about how that value of the selected cell was generated (how did this thing come about?). The query about how the cell value was derived may reveal the formula (which may have been captured in the relationship data table generated from the spreadsheet) defining a relationship between the cell and other cells and may further reveal that the formula depends on other formulas. The functional / formulation data (relationship data) can thus be expanded to reveal the complete functional form that controls the value outputted by the selected cell.

[0063] The information captured through the transformation of the spreadsheet into surface data and relationship data (in separate tables) facilitates the creation of a better user interface to unwrap all of the intertwined formulas defining the relationships between cells, as well as their semantic and contextual meaning. To illustrate, suppose cell number 34 in row 8 is defined as a function of three other cells (based on various mathematical operations). But it is possible that one or more of the cells on which cell 34 in row 8 depends are a function of some other cells or variable inputs, resulting in a recursive chain that, through the regular spreadsheet interface, would be difficult to untangle. However, the proposed approach allows those functional relationship to be unwrapped to identify how a particular cell or value was calculated from the input values through the chain of cells that rely on the input value. Because at least some cells may further be associated with a heading description (e.g., the cell’s column may be titled “rent”) and may also be associated with row descriptions and cell-specific contextual information, the relationships between cells can be much better visualized andrepresented than the way it is represented in the source spreadsheet. Consequently, formulating queries to extract needed information from the resultant representation of the spreadsheet data becomes simpler. The representations of spreadsheet data and relationship data thus allow a user to determine a natural language explanation of the data, and potentially that explanation data could be folded into supporting question-like queries, as well as using the transformed tables for process automation.

[0064] As noted, the proposed framework facilitates the implementation of a Question- Answering (QA) system to handle spreadsheets and provide responses to relational queries regarding the spreadsheet data (e.g., “which vendors located in Canada got the highest amount of funding from the US government?”). The example query is a natural language query, which in turn can be converted in an SQL query (e.g., using a NL2SQL conversion / mapping process). The idea is that a spreadsheet is turned into an SQL- supported database, which provides enormous flexibility in managing and extracting needed data.

[0065] Additionally, in some embodiments, the framework is configured to transform unstructured content into a data arrangement that has structure, and for which relational queries (in the form of natural language questions, or directly as SQL queries) can be asked.

[0066] Some spreadsheets have cells that hold unstructured content. It is plausible that one cell’s unstructured data might have relational content to other cells.

[0067] For example, cells of a spreadsheet might include data representing the facts that three different US presidents attended Yale University and two of them graduated. Such information may be arranged in some cells in the spreadsheet for each president that has a paragraph about their educational history and achievement from primary to college. Assume that a question is submitted to find all presidents that attended or graduated Yale. In the embodiments described herein, this task may be performed through a process in which the submitted question is first converted into an SQL query via an NL2SQL operation. The query would cause selection of a column about presidents’ education history, and then QA query can be applied to those cells to identify those that attended vs. those who graduated from a particular college. Based on the QA on the unstructuredcells, the presidents matching the query processed would be selected from the “presidents” columns.

[0068] Thus, in some embodiments, the proposed framework can be used to structure unstructured data (in data structures lacking column / row heading, and / or lacking other types of metadata). For example, domain- specific document ingestion techniques, as well as NL2SQL rules, may be used to facilitate the mapping of natural language queries to SQL. For example, in a financial document domain, there may be a number of possible relations that are expected to be found in documents, for instance, revenue by date, etc. Therefore, manual rule-based mapping to table and field names, or domain- specific training of such mappings, may be particularly effective. As an example of domainspecific processing, in the case of a set of resumes, the name of the applicant may be automatically extracted, along with the highest degree, the set of skills that the candidate mentions, and the languages that the candidate knows. Each of these aspects becomes a column in the ensuing database. The features may be extracted using rules, statistical models, or a combination of both. The features may be extracted using a question answering system in which specific questions are hand-generated and asked against the document. For example, for a DB constructed around a set of laptop models, a good question to ask might be “How much RAM does model XYZ have?” The amount of RAM for each model can be extracted using this method, and that information can be used as a column in the DB, where rows would be laptop models. The set of features to be collected is subjective, and dependent on what traits the system owner finds useful.

[0069] In some examples, operations involving the structuring of an unstructured file may be configured to extract a table and make up an RDB, and to apply SQL query derived from a natural language question to get the answer. Such an answer could be presented in Pryon Portal, a Chatbot, or integrated inside the view of an Application. Alternatively, an extracted table can be visualized as a spreadsheet, which would enable getting the user experience (UX) benefits of a spreadsheet user interface, which would allow at least some of the non-NLP search operations that can be performed on the spreadsheet interface to be available to handle the table data.

[0070] Accordingly, implementations of the proposed framework include a system including a communication unit to communicate with one or more remote computingdevices, one or more storage devices to store data and computer-executable instructions, with the data including first data comprising data values for rendering in a renderable view and including second data comprising specifications for determining the values in the first data, and a processor-based controller electrically coupled to the communication unit and the one or more storage devices. The processor-based controller is configured to process the data values of the first data to yield processed first data including semantic interpretation of at least some of said data values, process the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data, accept a natural language input, and process the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0071] In some variations, a non-transitory computer readable media is provided for interacting with a data storage including first data comprising data values for rendering in a renderable view and second data comprising specifications for determining the values in the first data. The computer readable media is programmed with instructions, executable on one or more processors of a computing system to process data values of a first data to yield processed first data including semantic interpretation of at least some of said data values, process the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data, accept a natural language input, and process the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0072] With reference to FIG. 4, a flowchart of an example procedure 200 for interacting with a data storage is provided. The data storage including first data comprising data values for rendering in a rcndcrablc view and including second data comprising specifications for determining the values in the first data is provided. The procedure 200 includes processing 210 the data values of the first data to yield processed first data including semantic interpretation of at least some of said data values, processing 220 the specifications of the second data to yield processed second data including functional definitions of at least some of the data values of the first data, accepting 230 a natural language input, and processing 240 the natural language input using the first processed data and the second processed data to respond to the natural language input.

[0073] In various examples, the data storage includes a spreadsheet and the renderable view may include an array of cells of the spreadsheet in which at least some of the cells include renderings of respective data values. In such examples, the functional definitions of the second data may include formulation / computational or data access specifications for determining at least some of the data values for rendering in cells of the spreadsheet.

[0074] The natural language input may include a natural language query for a data value, and processing the natural language input may include using the functional definitions of the second data to identify the data value to respond to the query. The array of cells of the spreadsheet may include multiple cells with rendered viewable data, with a first set of cells from the multiple cells including rendered viewable data inputted into the array of cells, and a second set of cells from the multiple cells in which each cell of the second set of cells may include rendered viewable data derived according to a respective formulation, defined in the specification of the second data, relating each cell of the second set of cells to respective one or more other cells of the multiple cells in the array of cells. Processing the data values may include transforming the array of cells into a first resultant arrangement comprising the rendered viewable data corresponding to the data included in the first set of cells and the second set of cells, and processing the specification may include transforming the specification into a second resultant arrangement comprising viewable relationship data representing the respective formulation relating the each cell of the second set of cells to the respective one or more other cells.

[0075] The relationship data may include formulas relating cells of the second set of the multiple cells as functions of data in the respective one or more other cells. The procedure 200 may further include converting the rendered viewable data of the first resultant arrangement and the viewable relationship data of the second resultant arrangement into relational database data. The procedure 200 may further include receiving a natural language query relating to the relational database data and converting the natural language query into an SQL query compatible with the relational database data. Converting the rendered viewable data and the viewable relationship data may define one-to-one mapping relationships between each of the multiple cells of the spreadsheet and a respective data element of the relational database data, hi suchembodiments, the procedure 200 may further include applying the SQL query to the relational database data to identify one or more data elements responsive to the SQL query, and identifying result cells from the multiple cells of the spreadsheet based on the identified one or more data elements responsive to the SQL query and on based on the one-to-one mapping relationships between each of the multiple cells of the spreadsheet and the respective data element of the relational database data.

[0076] In some examples, the approaches described herein are not restricted querying information in spreadsheets with hidden formulas. There may be other forms of data with hidden definitions defining and / or relating visible data. For example, the visible data may be a table, and the definitional data may be a database query (e.g., an SQL query) that was used to retrieve and / or compute the data in the table. The database query then provides the functional definition and relationships of cells of the table. For instance, a Microsoft Excel file may have embedded within it a SQL query (e.g., using Microsoft Power Query) that is used to access external data, and may have other definitional information embedded within it in the form of cell formulas or use other definitional information (e.g., Microsoft Power Business Intelligence (BI) definitions in languages including M and DAX). Such functional definitions, potentially in conjunction with semantic information about the external data, may be used in processing user queries.

[0077] In some examples, the procedure may further include determining answer data responsive to the SQL query, and presenting the answer data as output through one or more of, for example, output interface of a chatbot, output interface of a question-and- answer portal, rendered output presented in at least one of the results cells identified based on the identified one or more data elements responsive to the SQL query and the one-to-one mapping relationships between each of the multiple cells of the spreadsheet and the respective data element of the relational database data. The procedure 200 may further include obtaining one or more natural language queries, applying at least part of the one or more natural language queries to one or more unstructured cells in the array of cells of the spreadsheet through a question-and-answer system to obtain respective responses, and determining structured information for the one or more unstructured cells of the spreadsheet based on the respective responses to the one or more natural language queries.

[0078] In implementations described herein that are based on learning machines, different types of learning architectures, configurations, and / or implementation approaches may be used. Examples of learning machines include neural networks, including convolutional neural network (CNN), feed-forward neural networks, recurrent neural networks (RNN), etc. Feed-forward networks include one or more layers of nodes (“neurons” or “learning elements”) with connections to one or more portions of the input data. In a feedforward network, the connectivity of the inputs and layers of nodes is such that input data and intermediate data propagate in a forward direction towards the network’s output. There are typically no feedback loops or cycles in the configuration / structure of the feed-forward network. Convolutional layers allow a network to efficiently learn features by applying the same learned transformation(s) to subsections of the data. Other examples of learning engine approaches / architectures that may be used include generating an auto-encoder and using a dense layer of the network to correlate with probability for a future event through a support vector machine, constructing a regression or classification neural network model that indicates a specific output from data (based on training reflective of correlation between similar records and the output that is to be identified), etc.

[0079] The neural networks (and other network configurations and implementations for realizing the various procedures and operations described herein) can be implemented on any computing platform, including computing platforms that include one or more microprocessors, microcontrollers, and / or digital signal processors that provide processing functionality, as well as other computation and control functionality. The computing platform can include one or more CPU’s, one or more graphics processing units (GPU’s, such as NVIDIA GPU’s, which can be programmed according to, for example, a CUDA C platform), and may also include special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application- specific integrated circuit), a DSP processor, an accelerated processing unit (APU), an application processor, customized dedicated circuity, etc., to implement, at least in part, the processes and functionality for the neural network, processes, and methods described herein. The computing platforms used to implement the neural networks typically also include memory for storing data and software instructions for executing programmedfunctionality within the device. Generally speaking, a computer accessible storage medium may include any non-transitory storage media accessible by a computer during use to provide instructions and / or data to the computer. For example, a computer accessible storage medium may include storage media such as magnetic or optical disks and semiconductor (solid-state) memories, DRAM, SRAM, etc.

[0080] The various learning processes implemented through use of the neural networks described herein may be configured or programmed using TensorFlow (an open- source software library used for machine learning applications such as neural networks). Other programming platforms that can be employed include KERAS (an open-source neural network library) building blocks, NumPy (an open-source programming library useful for realizing modules to process arrays) building blocks, etc.

[0081] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly or conventionally understood. As used herein, the articles “a” and “an” refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, “an element” means one element or more than one element. “About” and / or “approximately” as used herein when referring to a measurable value such as an amount, a temporal duration, and the like, encompasses variations of ±20% or ±10%, ±5%, or ±0.1% from the specified value, as such variations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein. “Substantially” as used herein when referring to a measurable value such as an amount, a temporal duration, a physical attribute (such as frequency), and the like, also encompasses variations of ±20% or ±10%, ±5%, or ±0.1% from the specified value, as such variations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein.

[0082] As used herein, including in the claims, “or” as used in a list of items prefaced by “at least one of’ or “one or more of” indicates a disjunctive list such that, for example, a list of “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and B and C), or combinations with more than one feature (e.g., AA, AAB, ABBC, etc.). Also, as used herein, unless otherwise stated, a statement that a function or operation is “based on” an item or condition means that the function or operation is based on thestated item or condition and may be based on one or more items and / or conditions in addition to the stated item or condition.

[0083] Although particular embodiments have been disclosed herein in detail, this has been done by way of example for purposes of illustration only and is not intended to be limit the scope of the invention, which is defined by the scope of the appended claims. Any of the features of the disclosed embodiments described herein can be combined with each other, rearranged, etc., within the scope of the invention to produce more embodiments. Some other aspects, advantages, and modifications are considered to be within the scope of the claims provided below. The claims presented are representative of at least some of the embodiments and features disclosed herein. Other unclaimed embodiments and features are also contemplated.

Claims

WHAT IS CLAIMED IS:

1. A method for interacting with data storage that stores first data and second data, wherein the first data comprises data values that are to be rendered in a renderable view and wherein the second data comprises specifications for determining the data values that are in the first data, the method comprising: processing the data values to yield processed first-data, wherein the processed first-data includes semantic interpretation of the data values; processing the specifications to yield processed second-data, wherein the processed second-data includes functional definitions relating the data values of the first data; accepting a natural language input; and processing the natural language input using the first processed data and the second processed data to respond to the natural language input.

2. The method of claim 1, wherein the data storage comprises a spreadsheet, wherein the renderable view comprises an array of cells of the spreadsheet, and wherein the cells comprise cells that include renderings of respective data values.

3. The method of claim 2, wherein the functional definitions comprise computational specifications for determining at least some of the data values from other of the data values for rendering in cells of the spreadsheet.

4. The method of claim 2, wherein the functional definitions comprise data-access specifications for determining at least some of the data values for rendering in cells of the spreadsheet.

5. The method of any of the preceding claims wherein the natural-language input comprises a natural-language query for a data value and wherein processing the natural-language input comprises using the functional definitions to identify the data value that is to be used to respond to the query.

6. The method of claim 2, wherein the array of cells comprises a first set of cells and a second set of cells, both of which include those cells that include rendered and viewable data; wherein the first set includes those cells that include rendered viewable data inputted into the array of cells; wherein the second set includes those cells that include rendered viewable data, the rendered viewable data having been derived according to a respective formulation defined in the specification of the second data that relates each cell of the second set to respective one or more cells in the array of cells; wherein processing the data values comprises transforming the array of cells into a first resultant arrangement; wherein processing the specification comprises transforming the specification into a second resultant arrangement; wherein the first resultant arrangement comprises the rendered and viewable data that corresponds to the data included in the first set and the second set; and wherein the second resultant arrangement comprises viewable relationship data that represents the respective formulation relating each cell of the second set of cells to the respective one or more other cells.

7. The method of claim 6, wherein the relationship data comprises formulas that relate cells of the second set as functions of data in the respective one or more other cells.

8. The method of claim 6, further comprising converting the rendered and viewable data of the first resultant arrangement and the viewable relationship-data of the second resultant arrangement into relational-database data.

9. The method of claim 8, further comprising receiving a natural -language query that relates to the relational-database data and converting the natural-language query into an SQL query that is compatible with the relational-database data.

10. The method of claim 9, wherein converting the rendered and viewable data and the viewable relationship data defines one-to-one mapping relationships between each of the multiple cells of the spreadsheet and a respective data element of the relational database data and wherein the method further comprises: applying the SQL query to the relational database data to identify one or more data elements responsive to the SQL query and identifying result cells from the multiple cells of the spreadsheet based on the identified one or more data elements responsive to the SQL query and the one-to-one mapping relationships between each of the multiple cells of the spreadsheet and the respective data element of the relational database data.

11. The method of claims 10, further comprising determining answer data responsive to the SQL query and presenting the answer data as output through an output interface of a chatbot.

12. The method of claims 10, further comprising determining answer data responsive to the SQL query and presenting the answer data as output through an output interface of a question-and-answer portal.

13. The method of claims 10, further comprising determining answer data responsive to the SQL query and presenting the answer data as output through rendered output presented in at least one of the results cells identified based on the identified one or more data elements responsive to the SQL query and the one-to-one mappingrelationships between each of the multiple cells of the spreadsheet and the respective data element of the relational database data.

14. The method of claim 2, further comprising: obtaining one or more natural language queries; applying at least part of the one or more natural language queries to one or more unstructured cells in the array of cells of the spreadsheet through a question-and- answer system to obtain respective responses; and determining structured information for the one or more unstructured cells of the spreadsheet based on the respective responses to the one or more natural language queries.

15. A system comprising: a communication unit to communicate with one or more remote computing devices; a storage device to store data and computer-executable instructions, the data including first data and second data, the first data comprising data values for rendering in a renderable view, and the second data comprising specifications for determining the values in the first data; and a processor-based controller electrically coupled to the communication unit and the storage device, the processor-based controller configured: to process the data values of the first data to yield processed first-data; to process the specifications of the second data to yield processed second- data; to accept a natural language input; andto process the natural language input using the first processed-data and the second processed-data to respond to the natural language input;wherein the processed first-data includes semantic interpretation of said data values and wherein the processed second-data includes functional definitions of at least some of the data values of the first data;16. A non-transitory computer readable media for interacting with data storage that includes first data and second data, the first data comprising data values for rendering in a renderable view and the second data comprising specifications for determining the values in the first data, wherein the computer readable media stores instructions that executable on a processor of a computing system to: process data values of the first data to yield processed first-data; process the specifications of the second data to yield processed second-data; accept a natural language input; and process the natural language input using the first processed data and the second processed data to respond to the natural language input; wherein the processed first-data includes semantic interpretation of at least some of said data values and wherein the processed second-data includes functional definitions of at least some of the data values of the first data;

Images