Method, system, and program for generating summaries for artificial intelligence-based question answering systems

The method generates fact sheets for AI-based question answering systems using a Universal Test Engine to evaluate and summarize system performance, addressing the lack of standardized information templates in traditional techniques and improving user understanding and system performance.

JP2024542416AActive Publication Date: 2024-11-15INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024527362
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-30
Filing Date
2022-10-12
Publication Date
2024-11-15
Estimated Expiration
2042-10-12

AI Technical Summary

Technical Problem

Traditional data processing techniques do not support the generation of AI fact sheets for artificial intelligence-based question answering systems, particularly for TableQA systems, which are unable to provide standardized information templates for capturing relevant data and ensuring trust, transparency, and comparative fairness.

Method used

A method and system for automatically generating fact sheets for AI-based question answering systems, utilizing a Universal Test Engine (UTE) to process and evaluate these systems against tabular data, generating accuracy values, query sets, and human-readable summaries to improve system performance and user understanding.

Benefits of technology

Enables the creation of standardized fact sheets that inform users about the capabilities and limitations of TableQA systems, enhancing trust, transparency, and fairness by providing detailed insights into system performance and potential improvements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542416000001_ABST
    Figure 2024542416000001_ABST
Patent Text Reader

Abstract

A method, system, and program for generating a summary of an artificial intelligence based question answering system is provided herein. The computer-implemented method includes processing at least one given artificial intelligence based question answering system against tabular data using at least one test engine, generating an accuracy value attributed to the at least one given artificial intelligence based question answering system in association with the particular tabular data based on the processing, generating a set of queries determined to be addressable by the at least one given artificial intelligence based question answering system against the particular tabular data based on the processing, generating at least one human-readable summary of the at least one given artificial intelligence based question answering system based on the accuracy value and the queries determined to be addressable, and performing one or more automated actions based on the at least one human-readable summary.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application relates generally to information technology, and more particularly to data processing technology. More specifically, artificial intelligence (AI) fact sheets are increasingly being used to standardize comparisons between different candidate models. As used herein, an AI fact sheet represents an attempt to standardize information templates to capture relevant information about a given model, with the goal of promoting trust, transparency, and comparative fairness for informed reusability among end users. However, traditional data processing approaches are unable to provide an equivalent of an AI fact sheet for other artificial intelligence-based tools, such as, for example, a TableQA system that receives a table and a natural language question that can be answered on the table, with the goal of finding the correct answer from the table. Summary of the Invention

[0002] In one embodiment of the present invention, a technique for automatically generating fact sheets for an artificial intelligence based question answering system is provided. An exemplary computer-implemented method may include processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine, and generating one or more accuracy values ​​attributed to the at least one given artificial intelligence based question answering system in association with the particular tabular data based at least in part on the processing of the at least one given artificial intelligence based question answering system. The method also includes generating a set of one or more queries determined to be addressable by the at least one given artificial intelligence based question answering system against the particular tabular data based at least in part on the processing of the at least one given artificial intelligence based question answering system. In addition, the method includes generating at least one human-readable summary of the at least one given artificial intelligence based question answering system based at least in part on the one or more accuracy values ​​and the one or more queries determined to be addressable, and performing one or more automated actions based at least in part on the at least one human-readable summary.

[0003] Another embodiment of the invention or elements thereof may be implemented in the form of a computer program product tangibly embodying computer-readable instructions that, when executed, cause a computer to perform a number of method steps as described herein. Still further, another embodiment of the invention or elements thereof may be implemented in the form of a system including a memory and at least one processor coupled to the memory and configured to perform the method steps. Still further, another embodiment of the invention or elements thereof may be implemented in the form of a means or elements thereof for performing the method steps described herein. The means may include hardware modules or a combination of hardware and software modules, the software modules being stored in a tangible computer-readable storage medium (or multiple such media).

[0004] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a diagram of a system architecture according to an embodiment of the present invention. [Diagram 2] FIG. 1 illustrates a system architecture including a Universal Test Engine (UTE) architecture, in accordance with an embodiment of the present invention. [Diagram 3] FIG. 2 illustrates a query generation control module according to an exemplary embodiment of the present invention. [Figure 4] 1 is a flow diagram illustrating a technique in accordance with an exemplary embodiment of the present invention. [Diagram 5] FIG. 1 is a system diagram of an exemplary computer system capable of implementing at least one embodiment of the present invention. [Figure 6] FIG. 1 is a diagram of a cloud computing environment in accordance with an exemplary embodiment of the present invention. [Figure 7] FIG. 1 is a diagram of an abstract model layer according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As described herein, an embodiment of the present invention includes automatically generating fact sheets for an artificial intelligence based question answering system. By way of example, at least one embodiment can include automatically generating at least one fact sheet for one or more question answering systems (e.g., TableQA systems) for a given table. Such an embodiment can include generating and / or implementing a UTE to test any black-box TableQA system S for various question complexities. Thus, in such an embodiment, given a TableQA system S and a table T, the UTE generates a number of outputs. For example, such outputs can include a determination of the overall accuracy, including accuracy across different complexity dimensions, of the system S given table T for queries of various complexities automatically generated by the UTE. Such accuracy determinations can be measured, for example, based on at least one standardized test bed of natural language questions. In addition, the UTE output can include example queries that the system S is expected to address for table T. Furthermore, such an embodiment can also include generating human understandable summaries to improve the TableQA system S.

[0007] Solely by way of example and explanation, such summaries may include statements such as the following: The system handles simple search queries with high accuracy, and also handles some simple aggregation queries that explicitly refer to aggregations. The system understands natural language paraphrases, but cannot handle queries involving paraphrases that involve abbreviations or non-exact matches. The system does not recognize unanswerable queries and fails for tables with data volumes greater than approximately 512 cells.

[0008] Thus, one or more embodiments may include informing a user of the current capabilities of a given TableQA system. In addition, such embodiments may include generating and / or utilizing an application programming interface (API) around a given query sheet. As used herein, a query sheet includes a diverse set of queries that a given system has been tested to answer correctly. For example, when inquiring whether a system S can handle a particular query Q, an API may be implemented to explore query Q within a given query sheet.

[0009] There are various implicit considerations for an AI-based QA system, such as the types of queries the system can answer, the kinds of natural language phrases the system can recognize, the difference between what can be implicitly inferred by the system and what must be explicitly stated (to the system), and whether the system can detect queries that it cannot answer. For example, query types can include: simple project queries on a single table, select(filter)-project queries on a single table (where filtering can be, for example, on entity names, numeric comparisons, time-based filters, etc.), aggregation-select-project queries on a single table (e.g., sum, average, max, min, etc. operations), aggregation-select-project-join queries on multiple tables, aggregation-select-project-join-group-by / order-by / having queries on multiple tables, nested queries, etc.

[0010] Additionally, natural language phrasing support may be provided for alias phrasing, e.g., "more than 30" vs. "above 30," "total amount" vs. "sum amount," and for argument position, e.g., "total amount" vs. "amount in total." Also, the distinction between the implicit capabilities of a system and the need for its explicit statements may be based, at least in part, on, for example, time filter arguments (e.g., "loans in 2019" vs. "loan with start date in 2019"), aggregation arguments (e.g., "average loans" vs. "average amount of loans"), and entity linking and / or abbreviations.

[0011] FIG. 1 illustrates a system architecture according to an embodiment of the present invention. As an example, FIG. 1 depicts a universal test engine 108 that takes a table T 104 as input in a fact sheet generator and extension API 105 and generates different categories of questions covering different test cases to create a uniform test bed QS. A user 102 can provide the universal test engine 108 with budget information on how many such queries can be used to test the QA system S106, thereby helping to determine the test bed size. All such queries are sent to a query evaluator 110 to evaluate the performance of the QA system S106 with respect to the QS. The output of the query evaluator 110 is used to create two fact sheet components: a query sheet 114 that captures the queries that were correctly answered by the QA system S106, and an accuracy sheet 116 that describes the performance of the QA system S106 against different categories of queries in the test bed. Based on the reports generated by the query sheet 114 and accuracy sheet 116, the user 102 can execute the cloud-based API 118 to improve system performance.

[0012] In at least one example embodiment, there may be at least two such high-level APIs, including a generate training example API (API1) and an improve QA performance API (API2). With respect to the generate training example API (API1), the user 102 may provide (to the query evaluator 110) his or her own set of training examples for further scrutiny and / or to improve system performance. With respect to the improve QA performance API (API2), the user 102 may initiate (via the self-supervised training component 112) a self-supervised based training example generation mechanism, which will generate more training examples for query categories for which the system S106 has failed, and thus likely improve the overall performance of the system S106.

[0013] FIG. 2 illustrates a system architecture including a UTE architecture, according to an embodiment of the present invention. As an example, FIG. 2 depicts components of a universal test engine 208 within a fact sheet generator and extended API 205, which aims to generate queries covering various aspects of testing the QA system S206. For example, a budget-based query generation (QG) plan 220 intelligently selects the ratio of different query types depending on the budget information given by the user 202. A focused QG 222 performs focused query generation of different categories according to the budget plan determined by the component 220. An adversarial paraphrase 224 introduces paraphrases of questions generated by the component 222, aiming to introduce adversarial noise into the natural language queries and / or test the performance of the QA system S206 in terms of understanding and / or handling the natural language queries when phrased differently. The QG with table perturbation 226 further attempts to test the performance of the QA system S206 against different types of tables by perturbing the input table T204 to introduce different complexities associated with the table structure.

[0014] Based on the performance of the QA system S206, an intelligent query summarization 230 summarizes the performance by selecting a subset of the operational queries entered in the query sheet 214, an accuracy report in the accuracy sheet 216, and a human readable summary 217 for the types of operational versus non-operational queries in conjunction with the generate summary component 228. The summary 217 is intended for a human user to understand the performance of the QA system S206 through a textual description. As also depicted in FIG. 2, an extended API 232 for feedback can be given by the user 202, and the cloud API 218 functions similarly to that described in FIG. 1 in conjunction with the cloud API 118.

[0015] As detailed herein, at least one embodiment includes refined query generation. Such an embodiment includes generating question and answer pairs associated with a given table. Additionally or alternatively, such an embodiment may include generating questions with one or more adjustable controls, which may include using at least one sample structured query language (SQL) for a given table and translating such SQL content into at least one natural language question. With regard to adjustable controls, in one or more embodiments, such controls may include: aggregations (e.g., SUM, AVG, COUNT, MIN, MAX, etc.), number of WHERE conditions, nested queries (e.g., group by, having, etc.), inequality conditions (e.g., greater than, less than, not equal, etc.), row order dependencies (e.g., first, last, next, etc.), multi-cell vs. single-cell answers, type of columns and rows to pick (e.g., text vs. value, column categories identified by named entity recognition (NER), etc.), use of abbreviations and / or synonyms, etc.

[0016] Also, as described further herein (e.g., in connection with FIG. 3 below), one or more embodiments include budget-based smart query distribution. For example, a QG system may generate many (e.g., thousands) of questions per table, but it may be cost-prohibitive to test the TableQA system with all such generated questions. Thus, at least one embodiment may include determining and using the TableQA system for a fixed number of cases.

[0017] FIG. 3 illustrates a query generation control module according to an exemplary embodiment of the present invention. As an example, FIG. 3 depicts a QG control module 352 that, given a list of tables 350 and one or more predictions (from TableQA system 356) for already generated questions, selects one of the tables and a set of one or more control parameters that define the type of question to next generate. The selected table is provided to the TableQA system 356, while the set of control parameters is provided to a question generator 354, which generates and provides at least one question-answer pair to the TableQA system 356. In addition, the TableQA system 356 can generate and return to the QG control module 352 one or more predictions related to the at least one question-answer pair provided by the question generator 354.

[0018] The generation of question-answer pairs can, in one or more embodiments, include using rule-based techniques, where such embodiments start with one or more simple questions and move on to more complex questions if the model performs well. Alternatively, for example, if the model fails on a two-clause question, such embodiments do not generate three- or four-clause questions. Also, one or more embodiments can include utilizing a minimum set of question types that are covered in relation to a given budget, and selecting tables according to the type of questions to be generated (e.g., size, numeric vs. text, etc.). Furthermore, at least one embodiment can include implementing a probabilistic system that estimates how the model will perform for a given type of question by analyzing past performance. Such embodiments can include emphasizing the generation of questions for which the system is uncertain (in terms of answering in the past) and / or has been uncertain.

[0019] As also described in more detail herein (e.g., in connection with component 230 of FIG. 2), at least one embodiment includes intelligent query summarization. As a mere example, consider an exemplary use case in which a user provides a number (e.g., 100) to summarize what types of queries work with the model. In an exemplary embodiment, the summarization should select a subset of the working queries (e.g., from the queries of the UTE generation) whose examples cover all test segments and are diverse in type. Thus, such an embodiment may include selecting a proportionate number of test cases per test segment, as in the UTE, but scaled using the user-provided budget (i.e., 100). For example, such selections may include 25% simple search test cases, 20% simple aggregation test cases, 15% advanced query test cases, 15% paraphrase test cases, 15% non-exact match test cases, and 10% abbreviation test cases. If a particular segment fails completely (e.g., an abbreviated test case), this distribution of test cases will be distributed (e.g., evenly distributed) across all other segments.

[0020] Additionally or alternatively, for each test segment, one or more embodiments may include selecting a subset of queries with minimal pairwise overlap of required properties and operations to cover the complete test space of the input table T. For example, a numeric property used in an aggregation query may not be reused for testing search queries due to budget constraints.

[0021] At least one embodiment also includes paraphrase generation. Such embodiments include entity matching by entity type and masking natural language phrases by specific action operators. For example, such masks can relate to entities (e.g., people, companies, etc.), named entities, numeric entities, numeric comparison operators, aggregation operators, numbers, etc. Additionally or alternatively, such embodiments can include masking the natural language query using one or more applicable masks. As an example, consider the following: "Show me companies in California" can be masked to read as "Show me [entity] in [named entity]" and "What is the average salary of persons with an age of more than 30" can be masked to read as "What is the [aggregation operator] [numeric entity 1] of [entity] with [numeric entity 2] [numeric comparison operator] [numeric]."

[0022] One or more embodiments may also include using at least one high-level grammar and / or learning a language model for masked queries such that for each type of query, a set of masked templates can be implemented to generate queries of that type. Such queries may include, for example: simple project queries on a single table, e.g., "Show me [entity]", select (filter)-project queries on a single table, e.g., "Show me [entity] of [named entity]|Show me [entity] with [numeric entity][numeric comparison operator][numeric]|", aggregation-select-project queries on a single table, e.g., "Show me [aggregation][numeric entity] of [named entity]", aggregation-select-project-join-group-by / order-by / having queries on multiple tables, e.g., "Show me [aggregation][numeric entity] of [named entity] by [entity]|Show me top [numeric][entity] by [numeric entity]".

[0023] Also, for each masked entity type, at least one embodiment may include varying the likelihood of generating natural language query variants from the same template. By way of example, such embodiments may include introducing non-exact matches by using unique subwords, introducing broader synonyms by using at least one vocabulary database and / or thesaurus, or introducing filter and / or select clause candidates with omitted references, or a combination thereof.

[0024] Additionally, at least one embodiment includes a performance analysis for table perturbation. Given a table T, such an embodiment includes adding a synthetic row r, where the values ​​of each column c in r are generated by looking at the data distribution of c in the original table T. Such an embodiment also includes adding one or more new columns to an existing table by applying one or more arithmetic operations and / or one or more other aggregation operations to two columns (or more columns, if the column values ​​are numeric). The new columns may also be added by combining values ​​from the two columns with text data.

[0025] Further, given a question Q and a table T, one or more embodiments include analyzing trigger words to identify possible aggregation operations from the question and, depending on the trigger words, adding a new table column with a value to be returned after the aggregation operation on the column. By way of example only, words such as "sum" and "total" may be trigger words for a "SUM" type aggregation, and words such as "mean", "avg" may be trigger words for an "AVERAGE" type aggregation. For each column c in table T, at least one embodiment includes using existing sources (e.g., WordNet, word ontologies, etc.) to determine possible acronyms and / or abbreviations for the column name. A table T may be created in which the column names are replaced with their corresponding abbreviations and / or acronyms. Such embodiments may include converting table T to table T prime (T') by incorporating row headers in all rows and swapping row and column headers.

[0026] At least one embodiment may also include generating unanswerable queries. Given an answerable question Q and a corresponding table T that directly or indirectly contains an answer A, such an embodiment includes generating an unanswerable question Q' such that the question cannot be answered using table T directly or indirectly, and question Q' must be related to question Q and table T. More specifically, generating unanswerable questions may include, for example, clustering tables with similar columns, generating answerable questions using at least one question generation module for each table in a given cluster, and shuffling questions in the cluster such that questions appended to tables in the cluster are unanswerable.

[0027] As also detailed herein (e.g., via element 228 of FIG. 2), one or more embodiments include generating summary responses (e.g., human-readable summaries of a given set of tests conducted). By way of example, such summaries may include tables and / or charts of model performance based at least in part on a subset of the test instances. In at least one embodiment, generating the summary response may be used to identify interesting or consistent patterns of model performance or both, and may provide visual representations of key or notable figures or both (e.g., model performance vs. aggregate operations). Additionally, in some embodiments, the summary response may include one or more anecdotal examples of questions and predictions (e.g., examples representing types of questions for which the model performs well, examples of questions for which the model behaves unexpectedly (e.g., the model fails on easy questions and / or correctly predicts answers to difficult questions)). Additionally or alternatively, one or more embodiments may include using a template-based method to generate the summary response text and / or may include using one or more natural language generation (NLG) techniques.

[0028] At least one embodiment may additionally include generating and / or implementing one or more extension APIs (e.g., via element 232 of FIG. 2). Such an embodiment may include creating an ecosystem around at least one TableQA fact sheet for various use cases. As an example, consider a first API related to search operations on a model M, a table T, and / or a natural language query Q. Consider a use case of this first API, for example, related to determining whether M can handle Q given T. If not, an example embodiment may include using the first API to provide a query Q' that is closest to Q, or to rephrase Q to Q', or both, such that Q' can be handled by M given T. If no such Q' exists, such an embodiment may include searching for at least one similar query Q'' from example queries in a given fact sheet given M, T.

[0029] Additionally or alternatively, consider a second API related to obtaining training data for model M and / or table T. For example, consider a use case for this second API where a user has an external model M and wants to improve model M with additional training data to increase its accuracy with respect to T. At least one embodiment may include, in association with the second API, using a UTE to generate one or more questions with respect to table T and returning questions that model M fails with respect to T.

[0030] Further, consider a third API related to improving the model with respect to table T. For example, consider a use case for this third API where the model is internal to a given enterprise and a user wants to fine-tune the model with respect to table T. At least one embodiment can include using the second API described above (e.g., get_training_data() API) in conjunction with this third API to obtain non-operational queries and using at least a portion of the non-operational queries to improve the model using self-supervised techniques.

[0031] One or more embodiments may also include deriving answerable queries for a black-box natural language interface to database (NLIDB) system. Such an embodiment may include obtaining a sample natural language query q from a list of queries generated from a QA language model, and D q on S D refers to an instantiation of a system S in a domain D. If q fails, such an embodiment may further include generating a paraphrased version q′ of query q (e.g., using one or more rule-based generation techniques, which may include changing the wording and / or position of certain argument words in the query), and resolving S DIf q' is successful, such an embodiment may include encoding at least one failure rule regarding why q failed, encoding rewrite rules that transformed q to q', encoding at least one set of training data,<q,q’> and adding q' to a list of answerable queries. Otherwise, if q is successful on the first execution attempt, such embodiments may include adding q to a list of answerable queries and calculating at least one accuracy measure, e.g., based on the number of answered queries divided by the number of asked queries. Additionally, a final output of such embodiments may include the answerable query list and one or more marked rewrite rules.

[0032] 4 is a flow diagram illustrating a technique according to an embodiment of the present invention. Step 402 includes processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine. In at least one embodiment, processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine includes testing at least one given artificial intelligence based question answering system against the particular tabular data using a plurality of questions of varying complexity.

[0033] Step 404 includes generating one or more accuracy values ​​attributable to the at least one given artificial intelligence based question answering system in association with the particular tabular data based at least in part on processing of the at least one given artificial intelligence based question answering system. In one or more embodiments, generating the one or more accuracy values ​​includes generating at least one accuracy value measured against at least one standardized test set of natural language questions.

[0034] Step 406 includes generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, a set of one or more queries determined to be addressable by the at least one given artificial intelligence based question answering system for the particular tabular data.

[0035] Step 408 includes generating at least one human-readable summary of the at least one given artificial intelligence based question answering system based at least in part on the one or more accuracy values ​​and the one or more queries determined to be actionable. In one or more embodiments, automatically generating the at least one human-readable summary includes determining and outputting one or more suggestions for improving the at least one given artificial intelligence based question answering system.

[0036] In at least one embodiment, processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine includes processing a plurality of artificial intelligence based question answering systems against the tabular data using the at least one test engine, and generating a set of one or more queries based at least in part on the processing of the plurality of artificial intelligence based question answering systems includes generating a universal test bed of queries determined to be addressable by the plurality of artificial intelligence based question answering systems. Such an embodiment may also include comparing performance of the plurality of artificial intelligence based question answering systems relative to the universal test bed of queries.

[0037] Step 410 includes performing one or more automated actions based at least in part on the at least one human-readable summary. In at least one embodiment, performing the one or more automated actions includes automatically generating one or more application programming interfaces associated with the at least one given artificial intelligence based question answering system based at least in part on the at least one human-readable summary. In such an embodiment, generating one or more application programming interfaces associated with the at least one given artificial intelligence based question answering system may include generating at least one application programming interface associated with a search operation relating to the at least one given artificial intelligence based question answering system, the particular tabular data, and one or more natural language queries. Additionally or alternatively, generating one or more application programming interfaces associated with the at least one given artificial intelligence based question answering system may include generating at least one application programming interface associated with obtaining training data for at least one of the at least one given artificial intelligence based question answering system and the particular tabular data. Additionally, generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system may include generating at least one application programming interface related to modifying and / or improving the at least one given artificial intelligence based question answering system with respect to the particular tabular data.

[0038] In one or more embodiments, performing the one or more automated actions may also include training the at least one given artificial intelligence based question answering system based at least in part on at least a portion of the at least one human readable summary. Additionally or alternatively, performing the one or more automated actions may include automatically updating one or more existing application programming interfaces associated with the at least one given artificial intelligence based question answering system based at least in part on the at least one human readable summary.

[0039] Additionally, in at least one embodiment, software implementing the techniques depicted in FIG. 4 can be provided as a service in a cloud environment.

[0040] It will be appreciated that, as used herein, a "model" refers to a set of electronic, digitally stored executable instructions and data values ​​associated with one another, capable of receiving and responding to programmatic or other digital calls, invocations, or resolution requests based on specified input values, and capable of providing one or more output values ​​that are the basis for computer-implemented recommendations, output data displays, machine controls, and the like. While those skilled in the art find it convenient to express models using mathematical expressions, such forms of expression are not intended to constrain the models disclosed herein to be abstract concepts; instead, each model herein has actual application in a computer in the form of stored executable instructions and data that implement the model using a computer.

[0041] The techniques depicted in FIG. 4 may also include providing a system as described herein, the system including separate software modules, each of the separate software modules embodied in a tangible computer-readable recordable storage medium. All of the modules (or any subset thereof) may be on the same medium, for example, or each may be on different media. The modules may include any or all of the components shown in the figures or described herein, or combinations thereof. In an embodiment of the invention, the modules may be executed, for example, on a hardware processor. Then, the method steps may be performed using the separate software modules of the system as described above, executed on the hardware processor. Furthermore, a computer program product may include a tangible computer-readable recordable storage medium having code adapted to be executed to perform at least one method step as described herein, including providing a system having separate software modules.

[0042] 4 may be implemented using a computer program product that may include computer usable program code stored on a computer readable storage medium in a data processing system, the computer usable program code being downloaded over a network from a remote data processing system. Also, in embodiments of the invention, the computer program product may include computer usable program code stored on a computer readable storage medium in a server data processing system, the computer usable program code being downloaded over a network to the remote data processing system for use in the computer readable storage medium of the remote system.

[0043] Embodiments of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and configured to perform the exemplary method steps.

[0044] In addition, embodiments of the present invention may utilize software running on a computer or workstation. With reference to FIG. 5, such implementations may utilize an input / output interface formed, for example, of a processor 502, a memory 504, and, for example, a display 506 and a keyboard 508. As used herein, the term "processor" is intended to include any processing device, such as a device that includes a CPU (Central Processing Unit) or other forms of processing circuitry or both. Furthermore, the term "processor" may refer to two or more individual processors. The term "memory" is intended to include memory associated with a processor or CPU, such as, for example, a RAM (random access memory), a ROM (read only memory), a fixed memory device (e.g., a hard drive), a removable memory device (e.g., a diskette), a flash memory, and the like. Additionally, as used herein, the phrase "input / output interface" is intended to include, for example, mechanisms for inputting data into a processing unit (e.g., a mouse) and mechanisms for providing results associated with a processing unit (e.g., a printer). The processor 502, memory 504, and input / output interfaces such as a display 506 and keyboard 508 may be interconnected, for example, via a bus 510 as part of a data processing unit 512. Suitable interconnections may also be provided, for example via the bus 510, to a network interface 514, such as a network card provided to interface with a computer network, and to a media interface 516, such as a diskette or CD-ROM drive provided to interface with media 518.

[0045] Thus, computer software containing instructions or code for carrying out the methods of the present invention, as described herein, can be stored in an associated memory drive (e.g., ROM, fixed or removable memory) and, when ready for use, can be loaded partially or entirely (e.g., into RAM) and implemented by a CPU. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0046] A data processing system suitable for storing and / or executing program code includes at least one processor 502 coupled directly or indirectly to memory elements 504 via a system bus 510. The memory elements may include local memory utilized during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from bulk storage during execution.

[0047] Input / output devices, or I / O devices (including but not limited to keyboard 508, display 506, pointing device, etc.) can be coupled to the system either directly (such as via a bus 510) or through an intervening I / O controller (omitted for clarity).

[0048] Network adapters, such as network interface 514, may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Currently available types of network adapters include modems, cable modems, and Ethernet cards.

[0049] As used herein, including in the claims, a "server" includes a physical data processing system (e.g., system 512 as shown in FIG. 5) running a server program. It should be understood that such a physical server may or may not include a display and keyboard.

[0050] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0051] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or grooved structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0052] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to an individual computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the individual computing / processing device.

[0053] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or all on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0054] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, are implemented by computer readable program instructions.

[0055] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flow charts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flow charts and / or block diagrams.

[0056] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flow charts and / or block diagrams.

[0057] The flow charts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flow charts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, or may be performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flow chart illustrations, and combinations of blocks in the block diagrams and / or flow chart illustrations, are implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0058] It should be noted that any of the methods described herein may include the additional step of providing a system including separate software modules embodied in a computer readable storage medium, where the modules may include, for example, any or all of the components described in detail herein, where the method steps may be performed using separate software modules and / or sub-modules of such a system executed on the hardware processor 502. Furthermore, a computer program product may include a computer readable storage medium having code adapted to be implemented to perform at least one method step described herein, including providing a system having separate software modules.

[0059] In any event, it will be understood that the components illustrated herein may be implemented in various forms of hardware, software, or combinations thereof, such as application specific integrated circuits (ASICs), functional circuitry, a suitably programmed digital computer with associated memory, etc. Given the teachings of the invention provided herein, one of ordinary skill in the art will be able to contemplate other implementations of the components of the invention.

[0060] In addition, it should be appreciated at the outset that implementation of the teachings described herein is not limited to any particular computing environment: rather, embodiments of the present invention may be implemented in conjunction with any type of computing environment now known or later developed.

[0061] For example, cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0062] Its features are as follows:

[0063] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed without requiring human interaction with the service provider.

[0064] Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client and thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0065] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated according to demand. There is a sense of location independence in that consumers typically have no control or information about the exact location of the resources provided, although the location may be specified at a high level of abstraction (e.g., country, state, or data center).

[0066] Rapid Scalability: Capabilities can be provisioned quickly and scalably, in some cases automatically, quickly scaled out, and quickly released and quickly scaled in. To the consumer, the capabilities available for provisioning often appear unlimited, and any amount can be purchased at any time.

[0067] Metering of Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0068] The service model is as follows:

[0069] Software as a Service (SaaS): The functionality offered to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the exception of limited user-specific application configuration settings.

[0070] Platform as a Service (PaaS): The capability offered to the consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0071] Infrastructure as a Service (IaaS): The functionality offered to the consumer is the provisioning of processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0072] The deployment model is as follows:

[0073] Private Cloud: Cloud infrastructure is operated exclusively for an organization, is managed by that organization or a third party, and can exist on-premise or off-premise.

[0074] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with a shared set of objectives (e.g., mission, security requirements, policies, and compliance concerns). It can be managed by the organizations or a third party and can exist on-premise or off-premise.

[0075] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0076] Hybrid Cloud: Cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a unique entity, but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0077] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0078] 6, an exemplary cloud computing environment 50 is depicted. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof. The nodes 10 can communicate with each other. They can be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as described herein above (not shown). This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on a local computing device. It will be understood that the types of computing devices 54A-N shown in FIG. 6 are intended to be merely exemplary, and that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network or over a network-addressable connection (e.g., using a web browser), or both.

[0079] Referring now to Figure 7, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 6) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 7 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0080] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67, and database software 68.

[0081] The virtualization layer 70 provides an abstraction layer at which the following examples of virtual entities are provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75. In one example, the management layer 80 may provide the functions described below: Resource provisioning 81 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and billing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources.

[0082] In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides access to the cloud computing environment to consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance agreement on cloud computing resources in anticipation of future demand, and procurement of cloud computing resources according to SLAs.

[0083] The Workload Layer 90 provides examples of functionality for which cloud computing environments are utilized. Examples of workloads and functions that may come from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and, in accordance with one or more embodiments of the present invention, fact sheet generation 96.

[0084] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprise" and / or "comprising", as used herein, specify the presence of stated features, steps, operations, elements, or components or combinations thereof, but do not exclude the presence or addition of other features, steps, operations, elements, components or groups or combinations thereof.

[0085] At least one embodiment of the present invention may provide beneficial effects such as, for example, automatically generating fact sheets for an artificial intelligence based question answering system.

[0086] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used in this specification is chosen to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to allow others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method, comprising: processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine; generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, one or more accuracy values ​​attributed to the at least one given artificial intelligence based question answering system in association with particular tabular data; generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, a set of one or more queries determined to be addressable by the at least one given artificial intelligence based question answering system for the particular tabular data; generating at least one human readable summary of the at least one given artificial intelligence based question answering system based at least in part on the one or more accuracy values ​​and the one or more queries determined to be actionable; performing one or more automated actions based at least in part on the at least one human readable summary; and wherein the method is performed by at least one computing device.

2. processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine comprises processing a plurality of artificial intelligence based question answering systems against the tabular data using said at least one test engine; 2. The computer-implemented method of claim 1 , wherein generating a set of one or more queries based at least in part on the processing of the plurality of artificial intelligence based question answering systems comprises generating a universal test bed of queries determined to be addressable by the plurality of artificial intelligence based question answering systems.

3. The computer-implemented method of claim 2 , further comprising: comparing performance of the plurality of artificial intelligence based question answering systems relative to the universal test bed of queries.

4. 2. The computer-implemented method of claim 1, wherein performing the one or more automated actions comprises automatically generating one or more application programming interfaces associated with the at least one given artificial intelligence based question answering system based at least in part on the at least one human-readable summary.

5. 5. The computer-implemented method of claim 4, wherein generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system comprises generating at least one application programming interface related to search operations relating to the at least one given artificial intelligence based question answering system, the particular tabular data, and one or more natural language queries.

6. 5. The computer-implemented method of claim 4, wherein generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system comprises generating at least one application programming interface related to obtaining training data for the at least one given artificial intelligence based question answering system and at least one of the particular tabular data.

7. 5. The computer-implemented method of claim 4, wherein generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system comprises generating at least one application programming interface related to modifying the at least one given artificial intelligence based question answering system with respect to the particular tabular data.

8. 2. The computer-implemented method of claim 1 , wherein performing the one or more automated actions comprises training the at least one given artificial intelligence based question answering system based at least in part on at least a portion of the at least one human readable summary.

9. 2. The computer-implemented method of claim 1, wherein processing the at least one given artificial intelligence based question answering system against tabular data using at least one test engine comprises testing the at least one given artificial intelligence based question answering system against the particular tabular data using a plurality of questions of varying complexity.

10. 2. The computer-implemented method of claim 1 , wherein automatically generating the at least one human-readable summary comprises determining and outputting one or more suggestions for improving the at least one given artificial intelligence based question answering system.

11. 2. The computer-implemented method of claim 1 , wherein generating the one or more accuracy values ​​comprises generating at least one accuracy value measured against at least one standardized test set of natural language questions.

12. 2. The computer-implemented method of claim 1 , wherein performing the one or more automated actions comprises automatically updating one or more existing application programming interfaces associated with the at least one given artificial intelligence based question answering system based at least in part on the at least one human readable summary.

13. The computer-implemented method of claim 1 , wherein software implementing the method is provided as a service in a cloud environment.

14. 1. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a computing device to cause the computing device to: processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine; generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, one or more accuracy values ​​attributed to the at least one given artificial intelligence based question answering system in association with particular tabular data; generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, a set of one or more queries determined to be addressable by the at least one given artificial intelligence based question answering system for the particular tabular data; generating at least one human readable summary of the at least one given artificial intelligence based question answering system based at least in part on the one or more accuracy values ​​and the one or more queries determined to be actionable; performing one or more automated actions based at least in part on the at least one human readable summary; and A computer program product that causes

15. 15. The computer program product of claim 14, wherein performing the one or more automated actions comprises automatically generating one or more application programming interfaces associated with the at least one given artificial intelligence based question answering system based at least in part on the at least one human readable summary.

16. 16. The computer program product of claim 15, wherein generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system comprises generating at least one application programming interface related to a search operation relating to the at least one given artificial intelligence based question answering system, the particular tabular data, and one or more natural language queries.

17. 16. The computer program product of claim 15, wherein generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system comprises generating at least one application programming interface related to obtaining training data for the at least one given artificial intelligence based question answering system and at least one of the particular tabular data.

18. 16. The computer program product of claim 15, wherein generating one or more application programming interfaces related to the at least one given artificial intelligence based question answering system comprises generating at least one application programming interface related to modifying the at least one given artificial intelligence based question answering system with respect to the particular tabular data.

19. 15. The computer program product of claim 14, wherein performing the one or more automated actions comprises training the at least one given artificial intelligence based question answering system based at least in part on at least a portion of the at least one human readable summary.

20. 1. A system comprising: a memory configured to store program instructions; a processor operatively coupled to the memory, processing at least one given artificial intelligence based question answering system against the tabular data using at least one test engine; generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, one or more accuracy values ​​attributed to the at least one given artificial intelligence based question answering system in association with particular tabular data; generating, based at least in part on the processing of the at least one given artificial intelligence based question answering system, a set of one or more queries determined to be addressable by the at least one given artificial intelligence based question answering system for the particular tabular data; generating at least one human readable summary of the at least one given artificial intelligence based question answering system based at least in part on the one or more accuracy values ​​and the one or more queries determined to be actionable; performing one or more automated actions based at least in part on the at least one human readable summary; and said processor executing said program instructions to perform A system comprising:

Citation Information

Patent Citations

  • Information retrieval device and information retrieval program

    JP2016024765A

  • Interactive device, interacting method, and computer program for the same

    JP2016212541A

  • System and methods for automatically identifying answerable questions

    US20070067293A1

  • Automatically Assessing Question Answering System Performance Across Possible Confidence Values

    US20170140289A1

  • Computing Performance Scores Of Conversational Artificial Intelligence Agents

    US20200137002A1