Trusted questions for question-answering system

Abstract

Disclosed are implementations that include a method for question answering including receiving query data representative of a question, and performing a search of a document repository to identify one or more candidate answers responsive to the question, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers. The method further includes performing, in response to a determination that none of the one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository storing trusted question-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria, and providing output data based on results of the second search.

Description

TRUSTED QUESTIONS FOR QUESTION- ANS WERING SYSTEMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 533,417, filed on August 18, 2023, titled “Trusted Questions for Improving Consistency and Automation of a Question-Answering System,” which is incorporated herein by reference.BACKGROUND

[0002] The present invention relates to responding to a question using known questionanswer pairs.

[0003] Approaches to natural-language-based question answering may receive a natural language question from a user, and use natural language processing techniques to determine an answer to the question from a collection of documents. Some such techniques are described in PCT Application Publication WO2021 / 263138, titled “Document Processing and Response Generation”, published on December 30, 2021, which is incorporated herein by reference. In general, segments of documents are processed to find possible answers to the question and the set or set of best answers are presented to the user. One technique described in the publication makes use of a cache of questions and corresponding answers to avoid the need to search the segments of the documents. In that technique, upon receipt of a question from a user, the system determines if there is an appropriate answer in the cache because the question matches one of the questions in the cache, for example, according to a meaning similarly between questions, and that answer is presented to the user without having to search the documents.

[0004] Other natural language techniques may be used to determine an answer to a question, for example, based on Large Language Models (LLMs). An LLM may be trained on a collection of documents, and a user’s question may be provided as an input to the LLM and an answer is produced as the output of the LLM. Variants of suchtechniques make use of Retrieval Augmented Generation (RAG) techniques in which an input prompt to an LLM may be augmented with content retrieved from the document collection.

[0005] In some situations involving questions submitted by users, the best answer identified by a natural language processing system using a search of the documents may not be very good. The system may consequently look to the submitting user as unreliable (i.e., “stupid”) even though the user thought that they made a reasonable request.Answers produced by the natural language processing system may have scores associated with them, and one way to avoid having the system appear to be unreliable is to set a high threshold such that only high-scoring answers are provided to users, and for low scoring answers, the system simply returns an “I don’t know answer.” However, if the system starts returning too many “I don’t know” answers it may again make the system look unreliable or ineffective (i.e., “ignorant”). For example, to avoid providing the user with most unsatisfactory answers (“stupid” answers), the system may end up answering “I don’t know” 60% of the time. That is, the “automation rate” at which the system provides an automated answer may be 40% under these circumstances.

[0006] There is a need to provide responses to user questions that arc reasonable and accurate, and to avoid having the system appear “stupid” or “ignorant.”SUMMARY

[0007] In a broad aspect, a natural language question answering system makes use of a set of questions and corresponding answers (which may be referred to as “questionanswer pairs” or “Q / A pairs”). These Q / A pairs may be “trusted” in the sense that if a user asks one of those questions, the corresponding answer may be provided as a response with relatively high certainty that it will be responsive to the user’s question. More generally, a user may be unlikely to exactly repeat one of the questions for these trusted Q / A pairs. However, the user’s question may be similar enough in meaning to one (or more) of the questions of the Q / A pairs that the system may provide the corresponding answer back to the user.

[0008] There are a number of ways for operating a question answering system to use of the set of trusted Q / A pairs. One way is to use the trusted Q / A pairs only after a first attempt to answer a user’s question by searching the documents fails to return a suitable answer. For instance, a question-answering procedure is used to process the question to determine one or more answers from the collection of documents. Each determined answer is associated with a numerical score or at least some indication of reliability of the answer. If no determined answer has a suitable numerical score (e.g., a score above a fixed threshold) or indication of reliability, then the trusted Q / A pairs arc then used to attempt to produce an answer to the user’s question. This approach is different than use of a cache of Q / A pairs that is consulted first to avoid a search in that here the trusted questions are consulted, for example, only after there is no sufficiently high scoring answer under the normal content search.

[0009] There are a variety of ways of determining whether a user’s question matches a question of a trusted Q / A pair. For example, the user’ s question and each of the questions of the Q / A pairs may be transformed into a vector representation of the meaning of the question (e.g., a numerical array, a “sentence embedding”, a “question embedding”, see, e.g., Cer et al., “Universal Sentence Encoder”, arXiv 1803.11175, 2018), and then these vectors are compared according a measure of closeness (e.g., a vector distance, and inner product, etc.). In other examples, the word sequences of a user’s question and each of the questions of the Q / A pairs are compared to one another to determine whether they are paraphrases of one another (e.g., using a Recurrent Neural Network or a Transformer Neural Network model). In yet other examples, a large language model (LLM) may be prompted with the user’s question and each of the questions of the Q / A pairs with a prompt directive to determine the closest match (if any). Optionally, the prompt to the LLM may also incorporate the answers of the Q / A pairs and the prompt directive may be to provide the answer to the closest question.

[0010] There are a variety of ways that a system can form the trusted Q / A pairs. For example, the trusted Q / A pairs may result from prior user questions received in operation where the scores of the answers are found to be high or where there is explicit user feedback (“thumbs up”), implicit user feedback (e.g., clicking through an answer link, not searching further with a semantically similar question, etc.), or explicit third-partylabeling (after the fact) based on manual review of the questions and answers. Such trusted questions and their answers are kept in the trusted Q / A set for future use with questions for that or other users. As another example, some trusted Q / A questions may be curated, for example, based on noticing that no high-scoring answers have been found for a set of similar questions, and one or more Q / A pairs are added by a subject matter expert (SME). As yet another example, a set of predetermined questions are used to elicit answers from the document collection, and these questions and answers are used to populate the set of trusted Q / A pairs (see, e.g., using a process as described in PCT Publication No. W02024 / 015252, published on January 18, 2024, which is incorporated herein by reference). As yet another example, the Q / A pairs may be determined during ingestion of the documents of the document collection, for example, using an LLM to determine Q / A pairs that may be answered by segments (e.g., paragraphs) of documents in the collection. These various examples of how the trusted Q / A pairs are generated and may be combined such that more than one technique is used to form a full set of trusted Q / A pairs.[Oil] In some examples of the approach, a core set of trusted Q / A pairs may be formed from a larger set. For example, the core set may be formed by a clustering process in which each Q / A pair of the core set may be representative of a number of Q / A pairs of the larger set. In one such clustering approach, the trusted question-answer pairs may be used as individual datapoints in a nearest-neighbor-like approach, or clustered in a k- means-like approach. The advantage of the k-means approach is to help address the issue that small variations in the input may lead to different answers. For example, if one trusted question-answer pair is “Q: How many US states border the Atlantic Ocean? A: 14.” And another trusted question-answer pair is “Q: How many US states do not border the Atlantic Ocean? A: 36”. Then, if a question is submitted such as “what is the number of states along the east coast in no particular order?”, this submitted question might map to either of these trusted questions leading to different answers. If instead of a nearest- neighbor type approach, the questions are first clustered using k-means on the question embedding vector, and a representative question-answer pair is then selected for each cluster, the variability in the answer would be reduced.

[0012] Advantageously, one can consider the framework as implementing a process of automatically rephrasing a user’s question when a low answer score is initially returned, but the user does not have to do the rephrasing, and the system already knows that there is a good answer to the rephrased question. Therefore, the user’s experience may be improved.

[0013] Thus, in some variations, a method for question answering is provided that includes receiving, at a local device from a remote device, query data representative of a question, and performing a search of a document repository to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers. The method further includes performing, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository storing trusted questionanswer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria, and providing output data based on results of the second search.

[0014] In some variations, a system is provided that includes a communication unit configured to receive, from a remote device, query data representative of a question, one or more storage devices to store data and computer-executable instructions, and a processor-based controller electrically coupled to the communication unit and to the one or more storage devices. The processor-based controller is configured to perform a search of a document repository, implemented on the one or more storage devices, to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers. The processor-based controller is further configured to perform, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository, implemented on the one or more storage devices, storing trustedquestion-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria, and provide output data based on results of the second search.

[0015] In some variations, a non-transitory computer readable media is provided that is programmed with instructions, executable on one or more processors of a computing system to receive, at a local device from a remote device, query data representative of a question, and perform a search of a document repository to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers. The instructions further cause the one or more processors of the computing system to perform, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository storing trusted question-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria, and provide output data based on results of the second search.

[0016] Embodiments of any of the above method, system, and / or computer-readable media may include at least some of the features described in the present disclosure, and may be combined with any other embodiment or variation of the methods, systems, media, and other implementations described herein.

[0017] Other features and advantages of the invention are apparent from the following description, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] These and other aspects will now be described in detail with reference to the following drawings.

[0019] FIG. 1 is a data flow diagram that illustrates the use of a set of trusted questionanswer pairs;

[0020] FIG. 2 is a schematic diagram of the architecture of an example of the information retrieval system on which the techniques described herein can be implemented; and

[0021] FIG. 3 is a flowchart of an example of a procedure for question answering.

[0022] Like reference symbols in the various drawings indicate like elements.DESCRIPTION

[0023] Referring to FIG. 1, an operational flow in a question-answering system 100 begins with a user 102 providing a question 104. The question is provided in a natural language form, for example, being entered by a keyboard or being automatically transcribed from a spoken query. In some alternatives, the natural language question 104 may be generated in whole or in part from non-linguistic input from the user 102, such as selections by the user in a graphical interface, or formulas in a spreadsheet, referred to by the user. In any case, the question is provided to the document search component 110, which implements a natural language procedure to determine an answer set 112 from a document collection 108 that was previously ingested, for example, by searching through segments of documents in the collection for an answer to the question.

[0024] The answer set 112 determined by the document search component 110 may be empty if the question could not be answered by the component, and more generally has one or more answers 113, and each answer has a corresponding score 114 (or some other indicator of reliability or pertinence to the question). Without loss of generality, the answers 113 may be sorted such that the first is “best” according to the scores.

[0025] In this operational flow, after the answer set 112 is returned, a test 130 is performed to determine if there is a suitable answer in the answer set 112 for presentation to the user in response to their question. Here, the score 114 for the best answer 113 is compared to a predetermined threshold, and if the score does not exceed the threshold, then a process using a set of trusted question-answer (Q / A) pairs 120 is followed. The setof Q / A pairs 120, which may be referred to as “trusted Q / A pairs”, includes corresponding questions 122 and answers 124 and optionally scores 126. Optionally, a question embedding 128, which may be considered to represent the meaning of the question, is also stored for each of the trusted questions, for example, computed using a technique such as described in Cer et al., “Universal Sentence Encoder”, arXiv 1803.11175, 2018.

[0026] In one implementation, a Q / A pair may be selected by a trusted Q / A match component 140 of the system from the set of Q / A pairs 120 based on similarity of question embeddings. A question embedding is computed for the user’s question 104, and compared to question embeddings 128 for the question of each of the trusted Q / A pairs to determine a similarity (i.e., a “semantic match”) for each of the Q / A pairs (e.g., as a Euclidean distance between the vectors, and inner product or angle between the vectors, etc.). The most similar of the trusted Q / A pairs is then determined.

[0027] The similarity may be used to determine if the trusted Q / A pair should be used, and if so, optionally how it should be presented. For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold), then no answer from the trusted Q / A pairs may be presented to the user and a response to the user may be something of the sort “I am sorry, I don’t have an answer to your question.”

[0028] If the similarity is high enough (e.g., the distance between vectors is low), the answer of the trusted Q / A pair may be provided as the answer to the user’s question without any indication that another question (i.e., the question of the trusted Q / A pair) was used to determine that answer. The response may be of the sort “The answer to your question is <trusted answer>”.

[0029] If the similarity is in an intermediate range (e.g., between a high threshold and a low threshold), the answer of the trusted Q / A pair may be presented in a form such as “I didn’t find a good answer to your question, but I think you might be asking ‘<trusted question>’ in which case the answer would be ‘<trusted answer>’”. In some embodiments, multiple trusted question-answers may be provided in such a manner, for example, to balance the semantic match of the original question and the trusted question,and the Q-A score or quality feedback (e.g., how many stars the answer got) of the trusted question and the trusted answer.

[0030] The use of question embeddings is only one approach to comparing a user’s question to questions of a set of trusted Q / A pairs. More specific similarity between such pairs of questions may make use of neural networks that receive both the questions as input. For example, a transformer model may be trained to determine whether two questions are related, for example, using techniques described in Reimers et al., “Sentence-BERT: Sentence embeddings using Siamese BERT-networks.” arXiv preprint arXiv: 1908.10084 (2019). Alternatively, a single BERT-based model may be trained to determine whether a separated concatenation of two questions represent paraphrases of each other. There are various ways to train such approaches, in particular, to determine which questions are related to one another. One approach is to use manually curated sets of related questions. Another approach is to collect questions presented by users that have elicited equivalent answers (e.g., as determined by similarity of sentence embeddings of the answers), and therefore can be assumed to represent at least somewhat interchangeable questions for the purpose of the procedure described above. Yet another approach is to use a generative system (e.g., an LLM -based system) to generate paraphrases of the questions, which are then used as a “synthetic” training set.

[0031] Yet another approach to determining if a trusted question matches a user’s question is to first generate a set of paraphrase of the user’ s question, for example, using a generative LLM-based system), and then compare those paraphrases to the questions of the trusted Q / A pairs, for example, using comparison approaches described above.

[0032] In the approaches described above, the document search component 110 may be implemented using techniques described in WO 2021 / 263138, titled “Document Processing and Response Generation System,” published on December 30, 2021, which is incorporated herein by reference. Furthermore, the approaches described above may be used to augment the processing of a user’s question after that prior system has been used to attempt to find an answer in the document collection. A schematic diagram of another example architecture of an example information retrieval system 200 on which the techniques described herein (including obtaining answers from a pool of trusted Q-A pairs) can be implemented is provided in FIG. 2.

[0033] Referring again to FIG. 1, the approach described above relies on the existence of the set of trusted Q / A pairs 120 prior to accepting the user’s question 104. There are various ways of creating, maintaining, and / or curating such a set. In general, the document search component may process a large number of questions (e.g., 10,000 questions are hitting the system) but only a portion of those questions (e.g., 1000) get high scores. Those high scores can be in the form of “thumbs up,” high confidence scores, other types of user-feedback scores, or computed (e.g., algorithmic) scores. In the rawest form it may be desirable to implement an unsupervised scoring scheme (i.e., algorithmic), with enhancement to the framework including supervised (manual) scoring added at a later point. A trusted Q / A collector 118 is shown in FIG. 1 as serving the function of collecting high scoring question answer pairs that have been encountered by the document search component 110.

[0034] The trusted Q-A approach is premised on the idea that instead of matching new questions only to answers (e.g., performing a search between the query data and document repository data), the proposed framework matches new questions against previously vetted (automatically or based on user feedback) questions that are paired with good answers. Sometimes a submitted (new) question is matched (through a search of the document repository) to an answer that is not a good answer. While the submitted question may not have yielded a very good answer based on a regular search of the repository, it might yield a good match against a question from the trusted Q-A repository. Since the matched trusted question is paired with an answer that has already been determined to be a good answer, there is a high likelihood that the trusted answer paired with the matched trusted question would be a good answer to the submitted (new) question.

[0035] Thus, under the approaches and solutions of the present disclosure, a question-to- question search is performed, in response to a determination that a question-to-content search of a document repository yielded low scoring answers (e.g., under some quality threshold), and whatever the answer for the matched question is, that answer is used as the answer to the new question. Under this approach the framework augments content without extending the labels. The framework adds other contents for a view (i.e., for a new question). That is, for a particular view, not only is there an answer, but there arealso other ensembles, which are the question ensembles that people have asked which are surrounding this answer. So if a new question can be matched to any of these trusted questions, the answer for that new question can be (e.g., if the answers resulting from a regular search of the document repository does not yield high matching scores) the answer that one of those trusted questions has.

[0036] An important advantage of the framework proposed herein is that it adds paraphrase matching to the problem. The trusted Q-A approach is not fine-tuning the question to the answer. Rather, the framework implements a good paraphrase matching approach that allows you to bring new knowledge (that was not used before) into play. The approach thus relies on similarity of questions and a library of good questions (namely, the trusted questions) to help get the good answer, instead of trying to map the poorly phrased question, because mapping into a well posed question may get you a better answer. This allows the framework to use the paraphrasing as a separate tool to help in question answering, which becomes new knowledge (new knowledge can help the system).

[0037] As introduced above, similarity matching between the different submitted questions and questions of the trusted Q-A set can be implemented using different approaches. An example uses Google’s universal sentence encoder, which is an embedding encoder (the embedding can be based on a different system than that used for the retriever system, thus allowing new information to be brought into the model), with similarity being based on similarity of the computed embedding vectors. Phrase matching can be implemented using vector transformers, which eliminates the need for lexical matching because it only focuses on the semantic relationship between sentences. Machine learning models for semantic meaning determination may also be used.

[0038] As noted, the present framework is an unsupervised implementation. While some of the verified answers that are compiled may come from having people (subject matter experts) build answers when no answer previously existed (as noted, good answers may be compiled without using feedback, based on automatic matching I scoring processes), the use of verified answers in the present system strengthen the retrieval system in that such use is done from the angle of improving the system in use.

[0039] The trusted Q-A set can be built over time, incorporating user feedback. If a trusted Q-A set is built (e.g., based on human feedback), the system may perform better than simply relying on high scoring questions. However, it takes time to build a large set of verified questions, and thus initially a question and answer scoring mechanism (to compile an initial trusted Q-A set) may be used.

[0040] One thing to keep in mind is that the present implementation is solving two problems. One problem is the that of having low scoring questions. Instead of returning a low-scoring question-answer, or returning a “no answer could be found” message, the proposed framework mitigates the problem of low scoring questions by checking if there is a high scoring question in the neighborhood of the submitted question (that resulted in low scoring response). If there is, the system will return the corresponding trusted answer, and as a result will improve the automation rate. The other problem the proposed framework can solve is variability in the answer. It has been observed that a small change in the way a question is phrased (e.g., “what is a good way to yz”, versus “how can I xyz” can result in very different answers even though the two versions of the question are semantically very similar. So, by mapping all of those variations to a single question (as is done in the proposed framework), the framework can reduce variability in the output.

[0041] Thus, in various examples, an information retrieval system is provided that includes a communication unit configured to receive, from a remote device, query data representative of a question, one or more storage devices to store data and computerexecutable instructions, and a processor-based controller electrically coupled to the communication unit and to the one or more storage devices. The processor-based controller is configured to perform a search of a document repository, implemented on the one or more storage devices, to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers. The processor-based controller is further configured to perform, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository, implemented on the oneor more storage devices, storing trusted question-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria, and provide output data based on results of the second search. In some embodiments, the processor-based controller configured to perform the second search is configured to determine semantic similarity between the question represented by the query data and the question portions for trusted Q-A pairs.

[0042] With reference to FIG. 3, a flowchart of an example procedure 200 for question answering is provided that includes receiving 210, at a local device from a remote device, query data representative of a question, and performing 220 a search of a document repository to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers. The procedure 200 further includes performing 230, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository storing trusted question-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria, and providing 240 output data based on results of the second search.

[0043] In various examples, performing the second search may include determining semantic similarity between the question represented by the query data and the question portions for trusted Q-A pairs. In such examples, providing the output data based on the results of the second search may include returning trusted Q-A pairs with associated semantic matching levels between the question represented by the query data and the respective questions portions of the returned one or more trusted Q-A pairs exceeding a semantic matching threshold, or returning a message indicating no match was found in response to a determination that none of the respective question portions of the trusted Q- A pairs in the trusted repository is associated with a semantic matching level between the question portions and the question represented by the query data exceeding the semantic matching threshold. Determining the semantic similarity between the questionrepresented by the query data and the question portions for trusted Q-A pairs may include transforming the question represented by the query data into a vector representation question in an embedding space defined to generate vector representations for semantic meanings of questions, and comparing the vector representation question to respective vector representations of the question portions of the trusted Q-A pairs.

[0044] In some embodiments, the trusted Q-A pairs may correspond to pairs of questions and answers determined to represent, based on user feedback, satisfactory responses to the respective questions of the Q-A pairs. The trusted Q-A pairs may correspond to pairs of questions and answers that each comprises a question portion with a high question score according to a question quality criterion and an answer portion with a high answer score according to an answer quality criterion. The document repository may include a plurality of document objects corresponding to transformed content of one or more source documents.

[0045] The approach described herein may be combined with a Q-A cache framework by first consulting the trusted (or curated) questions-answers based on a high degree of similarity of the user question and the cached question - a high enough match can avoid a content search. If a content search is performed, and no sufficiently high scoring answer is found, then the trusted questions can be consulted again with a lower threshold of semantic similarity of the user question and the trusted question than in the initial presearch consulting of the stored question-answers.

[0046] In implementations described herein that are based on learning machines, different types of learning architectures, configurations, and / or implementation approaches may be used. Examples of learning machines include neural networks, including convolutional neural networks (CNNs), feed-forward neural networks, recurrent neural networks (RNNs), etc. Feed-forward networks include one or more layers of nodes (“neurons” or “learning elements”) with connections to one or more portions of the input data. In a feedforward network, the connectivity of the inputs and layers of nodes is such that input data and intermediate data propagate in a forward direction towards the network’s output. There are typically no feedback loops or cycles in the configuration / structure of the feed-forward network. Convolutional layers allow a network to efficiently leam features by applying the same learned transformation(s) tosubsections of the data. Other examples of learning engine approaches or architectures that may be used include generating an auto-encoder and using a dense layer of the network to correlate with probability for a future event through a support vector machine, constructing a regression or classification neural network model that indicates a specific output from data (based on training reflective of correlation between similar records and the output that is to be identified), etc.

[0047] The neural networks (and other network configurations and implementations for realizing the various procedures and operations described herein) can be implemented on any computing platform, including computing platforms that include one or more microprocessors, microcontrollers, and / or digital signal processors that provide processing functionality, as well as other computation and control functionality. The computing platform can include one or more CPUs, one or more graphics processing units (GPUs, such as NVIDIA GPUs, which can be programmed according to, for example, a CUDA C platform), and may also include special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application- specific integrated circuit), a DSP processor, an accelerated processing unit (APU), an application processor, customized dedicated circuity, etc., to implement, at least in part, the processes and functionality for the neural network, processes, and methods described herein. The computing platforms used to implement the neural networks typically also include memory for storing data and software instructions for executing programmed functionality within the device. Generally speaking, a computer accessible storage medium may include any non-transitory storage media accessible by a computer during use to provide instructions and / or data to the computer. For example, a computer accessible storage medium may include storage media such as magnetic or optical disks and semiconductor (solid-state) memories, DRAM, SRAM, etc.

[0048] The various learning processes implemented through use of the neural networks described herein may be configured or programmed using TensorFlow (an open- source software library used for machine learning applications such as neural networks). Other programming platforms that can be employed include keras (an open-source neural network library) building blocks, NumPy (an open-source programming library useful for realizing modules to process arrays) building blocks, etc.

[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly or conventionally understood. As used herein, the articles “a” and “an” refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, “an element” means one element or more than one element. “About” and / or “approximately” as used herein when referring to a measurable value such as an amount, a temporal duration, and the like, encompasses variations of ±20% or ±10%, ±5%, or +0.1% from the specified value, as such variations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein. “Substantially” as used herein when referring to a measurable value such as an amount, a temporal duration, a physical attribute (such as frequency), and the like, also encompasses variations of ±20% or ±10%, ±5%, or +0.1% from the specified value, as such variations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein.

[0050] As used herein, including in the claims, “or” as used in a list of items prefaced by “at least one of” or “one or more of’ indicates a disjunctive list such that, for example, a list of “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and B and C), or combinations with more than one feature (e.g., AA, AAB, ABBC, etc.). Also, as used herein, unless otherwise stated, a statement that a function or operation is “based on” an item or condition means that the function or operation is based on the stated item or condition and may be based on one or more items and / or conditions in addition to the stated item or condition.

[0051] Although particular embodiments have been disclosed herein in detail, this has been done by way of example for purposes of illustration only, and is not intended to limit the scope of the invention, which is defined by the scope of the appended claims. Any of the features of the disclosed embodiments described herein can be combined with each other, rearranged, etc., within the scope of the invention to produce more embodiments. Some other aspects, advantages, and modifications are considered to be within the scope of the claims provided below. The claims presented are representative of at least some of the embodiments and features disclosed herein. Other unclaimed embodiments and features are also contemplated.

Claims

WHAT IS CLAIMED IS:

1. A method for question answering comprising: receiving, at a local device from a remote device, query data representative of a question; performing a search of a document repository to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers; in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, performing a second search of a trusted repository storing trusted questionanswer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria; and providing output data based on results of the second search.

2. The method of claim 1, wherein performing the second search comprises: determining semantic similarity between the question represented by the query data and the question portions for trusted Q-A pairs.

3. The method of claim 2, wherein providing the output data based on the results of the second search comprises: returning one or more trusted Q-A pairs with associated semantic matching levels between the question represented by the query data and the respective questions portions of the returned one or more trusted Q-A pairs exceeding a semantic matching threshold; or returning a message indicating no match was found in response to a determination that none of the respective question portions of the trusted Q-A pairs in the trusted repository is associated with a semantic matching level between the question portions and the question represented by the query data exceeding the semantic matching threshold.

4. The method of claim 2, wherein determining the semantic similarity between the question represented by the query data and the question portions for trusted Q-A pairs comprises: transforming the question represented by the query data into a vector representation question in an embedding space defined to generate vector representations for semantic meanings of questions; and comparing the vector representation question to respective vector representations of the question portions of the trusted Q-A pairs.

5. The method of claim 1, wherein the trusted Q-A pairs correspond to pairs of questions and answers determined to represent, based on user feedback, satisfactory responses to the respective questions of the Q-A pairs.

6. The method of claim 1, wherein the trusted Q-A pairs correspond to pairs of questions and answers that each comprises a question portion with a high question score according to a question quality criterion and an answer portion with a high answer score according to an answer quality criterion.

7. The method of claim 1, wherein the document repository comprises a plurality of document objects corresponding to transformed content of one or more source documents.

8. A system comprising: a communication unit configured to receive, from a remote device, query data representative of a question; one or more storage devices to store data and computer-executable instructions; and a processor-based controller electrically coupled to the communication unit and to the one or more storage devices, the processor-based controller configured to:perform a search of a document repository, implemented on the one or more storage devices, to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers; perform, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository, implemented on the one or more storage devices, storing trusted question-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the question represented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria; and provide output data based on results of the second search.

9. The system of claim 8, wherein the processor-based controller configured to perform the second search is configured to determine semantic similarity between the question represented by the query data and the question portions for trusted Q-A pairs.

10. A non-transitory computer readable media programmed with instructions, executable on one or more processors of a computing system to: receive, at a local device from a remote device, query data representative of a question; perform a search of a document repository to identify one or more candidate answers responsive to the question represented by the query data, with each of the one or more candidate answers associated with a matching score representative of an answer quality for the respective one or more candidate answers; perform, in response to a determination that none of the identified one or more candidate answers is associated with a respective matching score exceeding an answer quality threshold, a second search of a trusted repository storing trusted question-answer (Q-A) pairs to identify trusted Q-A pairs candidates based on matching the questionrepresented by the query data to respective question portions of the identified trusted Q-A pairs according to one or more matching criteria; and provide output data based on results of the second search.

Images