Context clarification and disambiguation for question answering processes

Abstract

Disclosed are methods, systems, devices, apparatus, media, and other implementations, including a method including receiving query data representing a question related to source content of one or more source documents, and causing a search of a data repository maintaining data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions. The method additionally includes identifying one or more concepts associated with the set of a plurality of matches, at least one of the identified concepts being associated with at least some of the plurality of matches and including distinct respective values ​​associated with some of the plurality of matches; obtaining disambiguation information associated with at least one of the identified concepts; and selecting at least one of the plurality of matches based on the obtained disambiguation information.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is an international application claiming priority to U.S. Provisional Patent Application No. 63 / 293,343, filed December 23, 2021, which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates to question-answering systems for generating responses to queries submitted by users, and in particular to techniques for facilitating the identification of relevant answers to queries through the determination of disambiguation information.

[0003] Computer users often have access to large amounts of data, whether accessible through a public network (such as the Internet) or a private network, that the user can search to find answers and information to specific or general inquiries regarding some topic or subject. For example, some organizations often collect a large number of documents to serve as a repository of information (whether administrative or technical) that employees of the organization can access and perform searches on. For example, a company may have a large library of human resources documents that together define (hopefully in a consistent manner) the HR policies and procedures of the enterprise. Users, such as company employees, may search through a collection of documents to answer a question (e.g., "How much vacation time am I entitled to?").

[0004] Depending on the level of specificity of the submitted query, the question-answering system may generate a large number of search results (even if the QA system performs some initial filtering to eliminate responses that do not meet a minimum relevance criterion). The user may then be presented with an overwhelming number of possible answers whose actual relevance and responsiveness to the submitted query can be ascertained only by reading through the answers, whether by reading short snippets or summaries presented on the search result user interface or by accessing the underlying documentation associated with the results. Summary of the Invention [Means for solving the problem]

[0005] The present disclosure is directed to a question answering system configured to identify concepts (abstract concepts describing content, metadata information, entity identifier information, etc.) related to answer results returned by the question answering system, and to determine disambiguation information to help reduce the number of answers through removal of answers that are deemed less relevant (in terms of disambiguation information) than others of the returned answers. The disambiguation information may be automatically generated based on available context information processed by the question answering system (entity names, extracted concepts derived on captured content segments, etc.) or may be obtained through dynamic interaction facilitated by the question answering system to let the user provide additional information (e.g. in the form of a response to the generated query) based on concepts identified by the QA system that may be used to disambiguate the available answers and remove less relevant answers.

[0006] The identification of related concepts may be performed by the QA system based on contextual information related to the answer results generated in response to a query (e.g., contextual information stored during the initial incorporation and processing of source documents into the QA searchable content) and based on other available contextual information (e.g., information related to the user, information related to previously submitted queries, etc.). The identification of concepts may be performed via a learning machine configured to identify / extract concepts from the search results (or a portion thereof). As discussed in more detail below, two answers may be considered ambiguous (and require disambiguation to resolve the existing ambiguity) if they are determined to relate to the same or similar concepts but have different (contradictory) concept values.

[0007] Advantageously, the proposed techniques and solutions described herein avoid the need to populate searchable content with an exhaustive set of possible contextual metadata capturing a large universe of possible contexts in which the content may be used or searched (as such vast contextual information would be too difficult to capture and difficult to predict what information will ultimately be useful for disambiguation).The proposed techniques and solutions implement an efficient framework that includes a search phase followed by a disambiguation phase (to perform queries on previously captured content).

[0008] Thus, in some variations, a method is provided that includes receiving query data at a local device from a remote device, the query data representing a question related to source content of one or more source documents; and causing a search of a data repository that maintains data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions maintained in the data repository. The method additionally includes identifying one or more concepts associated with the set of a plurality of matches, where at least one of the one or more identified concepts is associated with at least some of the plurality of matches and includes distinct respective values ​​associated with at least some of the plurality of matches; obtaining disambiguation information associated with at least one of the one or more identified concepts; and selecting at least one of the plurality of matches based on the obtained disambiguation information associated with at least one of the one or more identified concepts.

[0009] Some embodiments of the method may include at least some of the features described in this disclosure, including one or more of the following features.

[0010] Obtaining the disambiguation information may include obtaining query context information of recent query transactions conducted in association with the source content, and selecting at least one of the plurality of matches may include selecting at least one of the plurality of matches based at least in part on the query context information of recent query transactions conducted in association with the source content.

[0011] Obtaining the disambiguation information may include generating prompt data for prompting a user to provide clarification information, and selecting at least one of the plurality of consistency may include selecting at least one of the plurality of consistency based at least in part on the clarification information provided by the user in response to the generated prompt data.

[0012] Generating prompt data for prompting the user to provide clarification information may include, for example, automatically generating an output prompt based on one or more of: generating a list having selectable items corresponding to different values ​​of one or more context categories; applying natural language processing to the identified plurality of matches to generate a prompt having a list of selectable items from which the user selects one or more of the selectable items; and / or selecting one or more items from a set of pre-defined prompts.

[0013] Selecting at least one of the plurality of matches may include excluding one or more of the plurality of matches based on disambiguation information provided by the user. In such an embodiment, the method further includes iteratively generating refined prompt data based on non-excluding matches from the set of identified matches to prompt the user to iteratively provide another disambiguation information to identify an optimal match from the identified plurality of matches.

[0014] Generating the prompt data may include rendering a graphical representation of a map to prompt the user to indicate a geographic location, and selecting at least one of the plurality of matches based at least in part on the disambiguation information may include selecting at least one of the plurality of matches in response to at least one of the plurality of matches determined to be associated with the geographic location indicated by the user.

[0015] Each of the plurality of consistency matches may be associated with content context information associated with the data portion maintained in the data repository. Identifying one or more concepts associated with the plurality of consistency matches may include identifying the one or more concepts based at least in part on the content context information associated with each of the plurality of consistency matches.

[0016] The content context information associated with each data portion may be generated, for example, by one or more of the following: a) applying one or more pre-processes to one or more source documents to generate document context information representative of the structure and content of the one or more source documents, and transforming the one or more source documents based on the context information to generate one or more question-answer searchable documents; b) segmenting the one or more source documents into a plurality of document segments, identifying, for at least one segment of the plurality of document segments, at least one segment descriptor including one or more of at least one entity associated with the at least one segment, at least one task associated with the at least one segment, or at least one subject descriptor associated with the at least one segment, and tagging the at least one segment with the at least one descriptor; and / or c) adding user annotations to one or more of the data portions.

[0017] The content context information for each consistency of the plurality of consistency may include data representing values ​​of a plurality of context categories, and identifying one or more concepts associated with the plurality of consistency includes determining whether at least two of the plurality of consistency are associated with different values ​​of a particular context category from the plurality of context categories.

[0018] Having a search of the data repository determine a set of matches between the query data and data portions maintained in the data repository may include placing matches in the set of matches into groups that each share one or more of a plurality of context categories.

[0019] The query data may include query context data, and having a search of the data repository determine a set of matches may include having a search of the data repository identify data portions related to the query context data included in the query data.

[0020] The query context data may include geographic location data defined by a user via the graphical representation of the map, and selecting at least one of the plurality of matches based at least in part on the disambiguation information may include causing a search of the data repository to identify data portions related to the user-defined geographic location data.

[0021] The query context data may include category data defining one or more categories from a plurality of context categories, and inducing a search of the data repository may include causing the search of the data repository to identify matches relating to the defined one or more categories from the plurality of context categories defined in the query context data.

[0022] The data portions maintained in the data repository may include transformed portions of the source content transformed according to one or more content transformation procedures, and inducing a search of the data repository maintaining the data portions may include transforming the query data into transformed query data that is compatible with the transformed source content, and searching the transformed content maintained in the data repository to identify one or more candidate portions within the transformed content that match the transformed query data according to one or more criteria.

[0023] The transformed portion of the source content may include a data portion that has been transformed according to a "Bidirectional Encoder Representations from Transformers" (BERT) process.

[0024] The one or more transformations may include, for example, one or more of a coarse linearization transformation to generate coarse numerical vectors representing the content of multiple document segments of the source content, or a fine-detail transformation to generate fine-detail transformed content records representing the content of the multiple document segments.

[0025] Obtaining disambiguation information related to at least one of the one or more identified concepts may include, for example, obtaining the disambiguation information according to one of the following: i) a first disambiguation policy that specifies a predetermined ordering of concepts selected from the one or more identified concepts in which a relevance of a plurality of matches to each of the plurality of concepts is determined; ii) a second disambiguation policy for selecting a concept from the one or more identified concepts that optimizes an objective function for reducing a level of ambiguity among the plurality of matches; and / or iii) a third disambiguation policy that visually prompts a user for feedback related to the one or more identified concepts for selecting at least one of the plurality of matches.

[0026] In some variations, a system is provided that includes a communication unit configured to receive query data representing a question related to source content of one or more source documents from a remote device and a controller electrically coupled to the communication unit, wherein the controller is configured to: cause a search of a data repository maintaining data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions maintained in the data repository; identify one or more concepts associated with the plurality of matches, where at least one of the one or more identified concepts is associated with at least some of the plurality of matches and includes distinct respective values ​​associated with at least some of the plurality of matches; obtain disambiguation information associated with at least one of the one or more identified concepts; and select at least one of the plurality of matches based on the obtained disambiguation information associated with at least one of the one or more identified concepts.

[0027] In some variations, a non-transitory computer-readable medium is provided that is programmed with instructions executable on one or more processors of a computing system to receive, at a local device, query data representing a question related to source content of one or more source documents from a remote device, and to cause a search of a data repository that maintains data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions maintained in the data repository. The instructions further cause the computing system to: identify one or more concepts associated with the plurality of matches, where at least one of the one or more identified concepts is associated with at least some of the plurality of matches and includes distinct respective values ​​associated with at least some of the plurality of matches; obtain disambiguation information associated with at least one of the one or more identified concepts; and select at least one of the plurality of matches based on the obtained disambiguation information associated with at least one of the one or more identified concepts.

[0028] In some variations, a computing apparatus is provided that includes a communications unit configured to receive query data from a remote device representing a question related to source content of one or more source documents, and one or more programmable devices for performing a question answering process according to any of the method steps described above.

[0029] In some variations, a non-transitory computer-readable medium is provided that is programmed with a set of computer instructions executable on a processor that, when executed, causes operations including any of the various method steps described above.

[0030] Some embodiments of the above systems, devices, and / or computer-readable media may include at least some of the features described in this disclosure and may be combined with any other embodiment, variation, or feature of the method.

[0031] Other features and advantages of the invention will become apparent from the following detailed description, and from the claims.

[0032] These and other aspects will now be described in detail with reference to the following drawings. [Brief description of the drawings]

[0033] [Figure 1] 1 is a diagram of an exemplary question-and-answer document processing and response generation system. [Diagram 2] 1 is a flow diagram of a framework for determining disambiguation information for use during execution of a query (of a QA system). [Diagram 3] 1 is a diagram of an exemplary document capture procedure. [Figure 4] 1 is a flowchart of the context clarification and disambiguation procedure of the question answering process.

[0034]

[0035]

[0036] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Like reference symbols in the various drawings indicate like elements.

[0038] Disclosed are implementations of a question-answering system (also called a question-answering system or QA system) that dynamically determines disambiguation information used to evaluate the relevance of a query / search results and selects or excludes search results based on the determined disambiguation information. The disambiguation information may be determined through an interactive process (visual / graphical, textual and / or verbal interaction) to derive from user feedback that may resolve result ambiguities between multiple answers (or groups / clusters of answers) generated for a particular query, and / or through obtaining contextual information related to the query and / or related to various answers generated in response to the query.

[0039] The solutions and techniques proposed herein include a process that begins by adding metadata to unstructured content to indicate the context in which the information is relevant. For example, with respect to HR data, the employee's status and the state in which the employee resides may be needed to answer a specific question about medical leave. This information may be implicit, for example, in the URL or file path. This information may be separated from the text that answers the question, for example, in the document title or section heading. This information may also be extracted from the document content based on guidance provided by the content manager. When a user asks a question and an answer is returned, contextual information from the valid answers is collected. In some examples, some implementations may determine what information (if available) (e.g., resolving conflicts between values ​​of contextual elements) may disambiguate the answer, and the user may be queried to determine such information so that an answer may be more specifically selected to be relevant to their needs at this time. The disambiguation query (or question) may be a set of multiple values ​​(visually presented) where the user is asked to select one or more of the questions automatically generated by the system; or questions selected from a set already developed for the domain. This approach of interactively presenting the user with a set of disambiguating queries that are efficiently gleaned from a set of plausible answers themselves tells the user what information is needed to arrive at the best answer.

[0040] The proposed methods can be considered as a hybrid interactive system that combines methods related to unstructured QA search and structured search on the content itself by using the dialogue to guide the search of metadata related to the content to achieve better and more relevant results. These methods partially realize the technical solution of dynamic disambiguation of search results by partially adding structured metadata to the unstructured data, partially by determining the needed contextual information (extracted concepts or categories) by inspecting the metadata in a set of plausible responses, and partially by using this metadata to guide the disambiguation dialogue to deliver the user to the most relevant response to his / her question.

[0041] Thus, the proposed technique includes a method including receiving query data from a remote device at a local device, the query data representing a question related to source content of one or more source documents; and causing a search of a data repository that maintains data portions related to the one or more source documents to determine a set of multiple matches between the query data and the data portions maintained in the data repository. The method further includes identifying one or more concepts associated with the multiple matches, where at least one of the one or more identified concepts is associated with at least some of the multiple matches and includes different respective values ​​associated with at least some of the multiple matches; obtaining disambiguation information associated with at least one of the one or more identified concepts; and selecting at least one of the multiple matches based on the obtained disambiguation information associated with at least one of the one or more identified concepts. As noted, the disambiguation information may be determined based on available context information (including context information associated with the query itself or previously submitted queries) that depends on, for example, correlation between temporal proximity queries or spatial proximity queries (e.g., queries submitted from the same terminal). As also noted, the disambiguation information may be determined by interacting with the user (e.g., by presenting the user with a list with selectable options) to prompt the user to provide disambiguation information. The disambiguation information may then be used to select or eliminate one or more of the multiple generated answers to the query, and this process may be applied iteratively to the refined set of answers until the initial set of answers has been whittled down to some threshold number of answers (e.g., one answer, two answers, or any other number of answers).

[0042] The techniques and solutions described herein may be implemented on any computing framework that has search capabilities (whether in the form of question-answering or otherwise). For illustration purposes only, and without limitation, some exemplary embodiments of the techniques and solutions will be discussed with respect to the architecture depicted in Figure 1, which will be discussed in more detail. A discussion of an exemplary QA framework used to illustrate an embodiment of the dynamic disambiguation technique will then be followed by a more detailed discussion of the dynamic disambiguation technique (provided with reference to Figures 2 and 4).

[0043] Example Question-Answering System Architecture 1, a diagram of an exemplary system 100 for document processing and response generation is provided. Further details regarding system 100 are provided in International Application No. PCT / US2021 / 039145, entitled “DOCUMENT PROCESSING AND RESPONSE GENERATION SYSTEM,” the contents of which are incorporated herein by reference in their entirety.

[0044] System 100 is configured to ingest source documents (e.g., a customer's large library of documents or other repository of data, such as email data, collaboration platform data, etc.) to convert the documents into document objects (called Document Object Model (or DOM) documents) that represent a mapping from source documents to searchable result objects (transformed result) documents. Those document objects may be stored in a DOM repository (also called a knowledge distillation (or KD) repository). Users (e.g., employees of the customer) associated with the customer that provided the document library may then submit queries (e.g., natural language queries such as "How many days of vacation does a two-year employee get per year?") that are processed by system 100, and in situations where a quick answer is not otherwise available from a cache for commonly asked questions, the queries are processed and converted to a format compatible with the format of the ingested documents to identify portions within one or more of the ingested documents that may contain an answer to the user's query. The system then returns output data to the user, including, for example, pointers to locations in one or more of the source documents (corresponding to the identified one or more imported documents), which the user then directly accesses to retrieve the answer to the query. The output may alternatively or additionally, in some embodiments, include an answer to the user's query and / or a portion of the document (e.g., a paragraph) that includes the answer. Advantageously, the output returned to the user need not include the specific information sought by the user (although in some instances it may, if desired), but rather only includes a pointer to a portion of the source document stored in a secure site that cannot be accessed by parties not authorized to access the source document. This answer determination approach thus enhances the security features of transmitting sensitive information (e.g., confidential or private information).As discussed herein, in situations where a query generates multiple answers (some of which may have conflicting values), the output may include dynamically generated prompts asking the user to provide feedback to resolve ambiguities in the returned answers.

[0045] In some embodiments, searching a document object repository to find answers to a query typically involves two operations: (1) first, a process called Fast Match (FM) process is performed, and (2) the Fast Match process is followed by a process called Detailed Match (DM) process (also referred to herein as "dense detailed" search). Both the FM and DM processes may be based on the Bidirectional Encoder Representations from Transformers (BERT) model. In the FM case, the model (in some implementations) reduces to, for example, one vector for the query and one vector for a paragraph (e.g., a 200 word window that may also include contextual data). In DM, there are typically multiple vectors per query or paragraph, for example proportional to the number of words or subwords in the query or paragraph.

[0046] It should be noted that in some embodiments, the transformation of the query and / or source document may occur in the customer's network, and the transformed query and / or transformed content is then communicated to the central server. Such an embodiment may improve privacy and security for communicating sensitive data across the network, since the result vector (derived through transformation of the content or query data) is generated within the customer's (client's) secure space, and thus only the result transformation vector (rather than the actual content or query data) is available or resides in the centralized cloud server. The transformation of the content or query data at the client's device may act as a type of encryption applied to the transformed data, thus resulting in a secure process that protects the data from attacks against the server cloud. In some embodiments, the transformed data at the client's network may additionally be encrypted (whether it is source data or query data) to provide for further enhanced secure communication of the client's data.

[0047] As depicted in FIG. 1, the system 100 typically includes a document processing agent 110 (which may be an AI-based agent) in communication with a customer network 150a (one of n customer networks / systems that access the document processing agent 110 in the exemplary system 100). The document processing agent 110 may be implemented as an independent remote server serving multiple customers, such as customer systems 150a and 150n, and may communicate with such customers via network communication (whether it is a private network or a public network such as the Internet). Communication with the customer units is achieved through a communication unit including one or more communication interfaces (such as a server interface 120, an admin interface 125, an interactive user query interface 130 with an ambiguity resolver, and / or an expert interface 132, all of which are shown diagrammatically in FIG. 1), which may typically include a communication module (e.g., a transceiver for wired network communication and / or for wireless network communication, configured according to various suitable types of communication protocols). Alternatively, the document processing agent 110 need not be located at a remote location, but may be a dedicated node within the customer network (e.g., the document processing agent 110 may be implemented as a process running on one of the customer's one or more processor-based devices, or may be a logically remote node implemented on the same computing device as a logically local node; i.e., it should be noted that the term "remote device" may refer to a customer station while a "local device" may refer to the document processing agent 110, or vice versa). An arrangement in which the agent 110 runs from a customer's network (such as any of the customer networks 150a-n) may improve data security, but may be more expensive to run privately.

[0048] Furthermore, in other alternative embodiments, some parts of the system (e.g., an ingestion unit configured to perform pre-processing and vectorization (parameterization) operations on source documents and / or queries submitted by users) may be located inside the firewall of the customer's network, while the storage of the ingested documents (and optionally, a search engine for searching the ingested content) may be located outside the firewall of the customer's network (e.g., on a centralized cloud server). In such alternative embodiments, data sent to the cloud server (e.g., for performing searches at a centralized location) may already have been processed (e.g., via vector processing that may have been performed via a coarse transformation (e.g., applied to fixed-size input segments) and / or a fine-grained numerical transformation applied to portions smaller than those processed by the coarse transformation) into encoded (ingested) content that is incomprehensible to third parties not authorized to utilize the data, thus adding another measure of privacy and security protection to the data processed by using the system 100. In these alternative embodiments, an initial part of the processing of the input query may also be processed inside the firewall of the customer's network. In addition to performing transformations (of source content and / or queries) within the client's firewall, such transformed data may be further encrypted (using symmetric or asymmetric encryption keys) before being sent to the document processing agent 110, thus increasing the level of security / privacy achieved for communications between the customer's network and a centralized document processing agent (serving multiple customers).

[0049] Exemplary customer network 150a may be a distributed set of stations, possibly with dedicated secure gateways (protected by firewalls and / or other security measures) that may be controlled (from station 152) by an administrator. Customers have typically accumulated large amounts of electronic documents, including, for example, technical documents related to the customer's operations, administrative documents such as human resource documents, and all other types of written documents in electronic form. These documents are located in document library 160 (which may be a computing part of customer network 150a) and are accessible by various authorized users at user stations 154a-c in network 150a and by an administrator (via administrator station 154). Any number of stations may be deployed in any particular customer network / system. Administrator station 152 may control access to documents in library 160 by controlling privileges and otherwise managing the same (e.g., access to specific documents in library 160, managing content to hide portions that do not comply with privacy requirements, etc.). As discussed in more detail below, in addition to library 160 (which contains documents related to the operations of entities operating on the network), other sources of data or information may be available from various applications employed by customers for processing via the document processing implementations described herein (e.g., email applications, chat applications such as Slack, customer relationship applications such as Salesforce, etc.).

[0050] The administrator station 152 is configured to communicate with the document processing agent 110, for example, via the administration interface 125. Among other functions, the administrator may provide the document processing agent 110 with information identifying the location of source documents in a repository (library) 160 that maintains a plurality of source documents, control the configuration and operation of the functionality of the document processing agent 110 related to the customer network 150a, review data generated by the agent 110 (e.g., override some answers), provide training data to the document processing agent 110, etc. The communication between the station 152 and the administration interface 125 may be established based on any communication technology or protocol. To enhance security features, the communication between the document processing agent 110 and the administrator station 152 may include authentication and / or encryption data (e.g., using symmetric or asymmetric encryption keys provided to the document processing agent 110 and the administrator station 152). By using the communication link established between the administrator station 152 and the interfaces 120 and 125, the administrator provides the document processing agent 110 with the information necessary to access the document library. For example, the administrator station may send a message providing the document processing agent 110 with the network address of the document library 160 (and / or an identifier for a document within that library that the agent 110 will access and process). The administrator station may in turn receive an encryption key (e.g., a private symmetric key, or a public key corresponding to a private asymmetric key used by the agent 110) used to encrypt the content of the document transferred to the agent 110. Communications between the administrator station 152 and the administration interface 125 (or any of the other interfaces with which the administrator may communicate, such as interfaces 120 and 130) may also be used to establish other configuration settings that control the exchange of data and information between the customer network 150a and the document processing agent 110, as described in more detail below.

[0051] Once the document processing agent has been provided with the location (e.g., expressed as a network address) of the document library 160 and communication features that control the transmission of data between the customer network 150a and the agent 110, the agent 110 may begin receiving data transmissions of documents from the repository (library) 160. The administrator station 152 controls the transmitted content and may perform some pre-transmission processing on the documents sent to the document processing agent 110, including removing sensitive content (e.g., private details), encrypting the content (e.g., by using a public key that corresponds to a private key in the document processing agent 110), authenticating the data to be transmitted, etc. The document processing agent 110 receives the transmitted data from the customer network 150a via the server interface 120 and performs data pre-processing on the received data, including authenticating and / or decrypting the data, format conversion (if necessary), etc. The server interface 120 then passes the data corresponding to the document sent from the document library 160 (subject to any pre-processing performed by the interface 120) to the ingestion engine 126, which processes the received document in order to transform it into a representation (enabling the determination and generation of an answer to a query provided by a user of the network 150a). Typically, prior to applying the transformation, the source document is segmented into several parts (e.g., 200-word parts, or any other word-based segments), and the segmentation is performed according to various rules of adjacent content from various parts of the document into discrete segments. An example of a pre-processing (i.e., pre-transformation) rule is to build the segments by using a sliding window of fixed or variable length that combines one or more headings preceding the content captured by the sliding window and thus creates a contextual association between the one or more headings and the content captured by the window. Such rules ensure that "the transformation performed on the segment combines important contextual information with content located far away (e.g., far away within the source document) from the segment being processed."

[0052] Upon segmenting the source document and / or performing other types of pre-processing (as described in more detail below), the document ingestion engine 126 is configured to apply one or more types of transformations to the document segments (e.g., question / answer searchable segments) to convert the segments into searchable segments. One type of transformation that may be applied to the segments may be based on converting fixed-size (or nearly fixed-size) segments, which typically contain multiple words / tokens, into numerical vectors to perform a fast search process. Such searches are typically coarse searches in that "such searches typically return a relatively large number of results (hits) (in response to a query submitted by a user) because they are based on matching vectors generated from input data containing a relatively large number of words (tokens or features), and as a result, the resolution achievable from such transformations is lower than that achievable from converting smaller segments." Thus, results based on coarse vector transformations may not provide as precise a representation (as other transformations applied to smaller segments) of the textual meaning of the converted content. On the other hand, as the name suggests, a fast search can be performed relatively quickly and can therefore be used to sift the possible candidates of possible answers (to the submitted query) into a size or number that can then be more carefully searched (perhaps via a search based on another type of transformation). Another transformation that can be applied by the ingestion engine is to generate a dense fine-grain vector transformation that is used to more narrowly locate answers having unique answer word sequences in some text segment (e.g., a paragraph). In general, the document segment to which the dense fine-grain transformation is applied can be at a finer granularity (resolution) than the fast search segment, which is generally of a fixed size (e.g., 200 words and therefore cannot usually locate the exact location of an answer within the segment, if any at all.

[0053] More specifically, a fast lookup transform (e.g., implemented via neural nets, filters, etc.) is applied to the document segment to produce a vector having values ​​based on, and thus representative of, the content of the segment. As discussed in more detail below, several techniques may be applied by the document ingestion engine 126 to transform the data according to the fast lookup transform. In one example, the data representing the content may be transformed into a vector representation (e.g., a fixed size vector or a variable size vector). Thus, in such an example, the transform converts the text content into a vector of numerical values ​​that may or may not be associated with metadata (e.g., text-based metadata that provides additional information that may be used for further processing) or other contextual information. The resulting transformed vector may represent possible questions and answers related to the transformed input segment. One example of a transform that produces such vector values ​​representing the content of the input (including contextual relationships) is "Bidirectional Encoder Representations from Transforms" (BERT).

[0054] With respect to the dense-detailed transformation performed by the document ingestion engine 126, the source data (e.g., text-based portions segmented from the source document according to one or more rules or criteria; the segmented portions are typically smaller in size than the source segments used for fast search transformation) are typically transformed into multiple vectorized (numerical / parameterized) transformed contents. The dense-detailed transformation may also be implemented according to BERT. The processing by the document ingestion engine 126 may include natural language preprocessing to determine at least some language-based information (such as detecting and recording the location of named entities (e.g., people's names and company names) in the document), expanding structured data such as tables into searchable equivalent text, transforming information into knowledge representations (such as predefined frame structures), extracting semantic meaning, and so on. In some embodiments, the resulting dense-detailed transformed data may be combined with the original content to be transformed along with derived or provided metadata (such metadata is not critical, but may facilitate intelligent search and question answering performance of documents). In some instances, the combination of transformed content with source segments may be further extended with automatic questions that may be germane to the source segments, such that these generated questions combine with specific segments (or in specific locations within the entire document that includes the entirety of the source content and the corresponding transformed content) or with specific information fields. When processing queries from a user, the similarities between the user's query and such automatically generated queries may be used to answer the user's query by returning information (e.g., pointers or actual user-understandable content).

[0055] Continuing with reference to FIG. 1, the captured content generated by the document ingestion engine 126 is stored in a document object model (DOM) repository 140. The repository 140 is typically implemented (distributed or available at a single local location) on one or more data storage devices that may be accessible from multiple access / interface points between the repository 140 and other modules / units of the document processing agent 110. In the diagram of FIG. 1, the repository 140 is depicted as having two access points (to search the DOM records stored in the repository) and to receive search results that are forwarded to the user who submitted the query (optionally after some further processing, possibly including disambiguation processing via iterative interactive exchanges with the user): one access point that is a unidirectional link between the ingestion engine 126 and the repository 140 (i.e., a link that allows content to be written from the engine 126 to the DOM repository 140); and a bidirectional access point connected to the query processing module 136 that provides query data to the DOM repository 140. In some embodiments, the access point to the repository may be implemented as a single point connected to a module configured to perform query processing and document ingest operations.

[0056] The DOM repository 140 (in conjunction with the document ingestion engine 126 and / or the query processing module 136) is configured to store, manage, and search DOM records 142a-n. The content of a DOM record typically depends on the transformations performed by the document ingestion engine 126. A DOM record may contain data items related to a particular source document or source document portion. For example, one DOM record may be a collection of items including the original portion of the source document, metadata for that source document portion, contextual information related to that source document portion, corresponding coarse vectors resulting from transformations applied to one or more fixed-size (or nearly fixed-size) segments of the original portion of the source document (to facilitate a fast search process), corresponding resulting dense-detail transformed content resulting from the transformed dense-details (to facilitate more accurate and refined text searches), and so on. Thus, if the transformation results in a vector of values ​​representing the textual content of the segment, that vector is stored (appended or embedded into the vector) in the repository, possibly in association with metadata and / or in association with the original content (in situations where the actual original textual content is preserved; in some embodiments, for security or privacy reasons, the source content may be ignored upon ingestion or may only be available at the customer's site). The metadata associated with the transformed content may include contextual information related to the original source content, and document location information indicating the location or position of the source content that gave rise to the transformed content within a larger source document. Such document location information may be provided in the form of pointer information that points to a memory location (or memory offset location) of the source document stored within the customer network; i.e., such that when returned to a requesting user, the pointer information may be used to find a memory location where relevant content may be found that constitutes an answer to the user's query.

[0057] The transformed content (which may include several transformed content items resulting from various transformations applied to the segmented content), metadata, and / or source content stored in repository 140 may together define a unified record structure, where each of the transformed content, metadata, and / or original source content is a field or segment of the unified record structure. The individual records may be related to each other (e.g., by placing them consecutively or via logical or physical links / pointers) if they correspond to discrete document segments of a larger source document to define a larger document portion (e.g., a chapter of a particular document) or to define the entire original document that was captured as a segment.

[0058] As further shown in FIG. 1, the document processing agent 110 further includes a query unit (also referred to as a query stack) configured to receive input (data representing queries from one or more users authorized to submit queries in relation to at least some of the captured documents disposed in the DOM repository 140) and in turn provide returned output data to an initiating user. The query stack includes an interactive user query interface 130 having an ambiguity resolver in communication with a query processing module (also referred to as a query engine) 136 (the interactive user query interface may be similar to the server interface 120 or may be implemented by using the same hardware and software as the server interface 120). As discussed in more detail below, the query processing module may include a transformation engine for applying similar transformations to queries submitted by users to generate transformed query data that is compatible with the transformed content in the DOM records 142a-n maintained in the DOM repository 140. The transformed query may include coarse numeric-to-vector transformation data that can be used to search numeric-to-vector transformation content in repository 140, dense-detail transformation queries (that can be used to search similarly transformed dense-detail transformation content in repository 140), or any other transformation format that can be used to retrieve source documents. As discussed further below (with reference to FIG. 2), interactive interface with disambiguation resolver 130 is configured to receive process query data from a user and provide and return query output to the user, as well as determine disambiguation information (by itself or in combination with other modules of agent 110). Such disambiguation information may include disambiguation information that was originally provided (with the query) to aid in initial search / matching operations (e.g., pre-filtering operations) performed on searchable content (either in DOM repository 140 or in cache 135) managed by agent 110.The disambiguation information may also include post-filtering dynamically generated disambiguation information presented to the user to solicit the user to provide disambiguation information to resolve ambiguities present in two or more of the query results. For example, if two answers relate to the same or similar concepts / categories of information (entity names, related contextual information, or some abstract concepts derived by using natural language processing, or learning machine implementations) but have different concept / category values, an intermediate output may be provided to the user (e.g., as a visual disambiguation prompt or an audio disambiguation prompt) requesting the user to provide disambiguation information defining which of the identified concepts is most relevant to the user's query. The disambiguation information returned by the user may then be used to select one or more of the initial matches (and may eliminate some other matches) and / or to rank the initial or remaining matches (based on the calculated relevance determined by using the input returned by the user).

[0059] In embodiments in which repository 140 includes multiple types of transformed source content, a search of repository 140 may be implemented as a multi-pronged search. For example, because a coarse numeric vector representation is generally more compact and easier to search (although may not be as precise as a dense fine-grained transformed representation, whether implemented by a BERT-based transformation or some other transformation), a first prong of a search to determine an answer to a submitted query may be to transform the query data into a coarse vector representation and then use that first transformed query representation to search for records in repository 140 that match the coarse numeric base transformation of the query data (e.g., according to some closeness criterion that may represent the distance or difference between the transformed vector query data and the transformed vector-incorporated content data). This type of initial search may be referred to as a fast search. The results of the search may result in the identification of one or more candidate answers (e.g., identifying 1000 or any other number of possible segments that may contain an answer word sequence in response to a query submitted by a user). The identified first batch of possible results may then be used to perform a second stage of a search by converting the query into a dense detailed transformed query and by searching for dense detailed transformed content related to the search results identified in the first stage of the search process. This search stage may be referred to as a detailed (or fine-grained) search. It should be noted that in some embodiments, a fast search may be used to identify original portions of source content related to the identified candidates, and the identified portions may then be converted into dense detailed transformed content. In such an embodiment, the repository 140 need not maintain the dense detailed transformed content, but rather, the conversion of the source content is performed based on which portions are identified by the fast search as likely to contain the answer to the query. In an alternative example, searching for an answer to the query may be performed directly across the dense detailed transformed content records without first identifying possible candidate portions of the source content via a fast search of the fast search transformed content records.

[0060] Thus, in some embodiments, the query stack (e.g., query processing module 136) is configured to convert the query data into transformed query data that is compatible with the transformed source content (e.g., compatible with one or more of the transformed content records in DOM repository 140). For example, the fast search compatible transformation can be a coarse BERT-based transformation (e.g., using a learning engine implementing the same or similar trained learning model used to generate the searchable transformed content from the source data) that is applied to the entire query data (e.g., natural language question) to generate a single vector result. The query processing module can launch a fast search process (each numerical vector resulting from the coarse transformation) that identifies, for example, one or more candidate portions in the transformed source content that match the transformed query data according to a first criterion. For example, the matching operation can be based on some closeness or similarity criterion that corresponds to some calculated distance metric between the calculated vector transformed query data and various vector transformed content records in repository 140. As described herein, in some embodiments, the transformed content can include vectors that correspond to possible questions that the user may ask regarding which source content provides possible answers. Thus, in some embodiments, fast search may compare the transformed query results (generally, result vector records) with searchable vector records that represent possible questions that might be asked in relation to the source content from which those searchable vectors were generated.

[0061] The query processing module 136 may further be configured to determine, from the one or more fine-detailed transformed content records corresponding to the one or more candidate portions identified based on those coarse-transformed vectors, at least one fine-detailed transformed content record that matches the fine-detailed transformed data of the query data according to a second criterion (e.g., some other closeness or similarity metric, or the same criterion applied with respect to the coarse-transformed data). Alternatively, in embodiments where a fast search is not performed, the query processing module 136 may be configured to identify one or more candidate portions in the transformed source content having respective fine-detailed transformed content records that match the transformed query data according to the second criterion.

[0062] In some embodiments, the interface 130 and / or the query processing module may be coupled to a query cache 135 and a question generation unit (which may be part of the cache 135 or the query processing module 136 or may be a separate unit). The query cache 135 stores, among other things, answers / content corresponding to frequently asked questions. Such answers / content may include content previously retrieved from DOM documents (and / or from their corresponding raw source content) in response to previously submitted queries. Counters associated with such cached answers may track how frequently particular questions and answers are submitted and / or retrieved. The cache 135 may also be configured to ignore cached content that has not been requested within a certain threshold time interval. The content in the answer cache may also have been stored by an administrator (e.g., operating from a station such as station 152 via admin interface 125) in anticipation of some likely questions that users of customer system (network) 150a are expected to submit or that are expected to override content that may have been retrieved from DOM repository 140 (e.g., content determined based on subsequent feedback from users to be inaccurate or non-responsive to the submitted query). Thus, in some embodiments, the query stack is configured to determine whether the received query data matches one of the predefined questions (which may be stored in the answer cache), and to generate output data based on one or more answer data records (possibly stored in the answer cache) in response to determining that the received query data matches one of the predefined questions. In some embodiments, matching of the query data against past questions and related answers stored in the cache is performed by calculating a score based on a combination of the questions and their answers, and by ranking the calculated scores to identify one or more likely matching candidates.

[0063] The query processing module may also include a question generation engine that may determine (e.g., based on the trained learning engine and / or by using the repository of question data) follow-up or related questions to one or more questions submitted via the query data. Follow-up questions may be generated by paragraphing the submitted query (e.g., by transforming and / or standardizing the submitted query to modify the submitted question, e.g., by using the trained learning engine). In some embodiments, answer data determined for the submitted query (e.g., based on content retrieved from the DOM repository 140 via the query processing module 136) may be processed (by a separate module) to formulate further questions from the answers. Such derived questions may then be resubmitted to the query processing module to search for follow-up answers. This process may be repeated iteratively up to a predetermined number of times. In some circumstances, the content stored in the DOM repository 140 may associate multiple questions (represented in any transformation format applied during the document ingestion stage) with each processed segment of the source document. As noted, the generation of transformed content may include, for each processed segment, data representing a question related to the processed segment, metadata, content that may be provided in a transformed format, and / or original source content. Thus, when a query is submitted (typically in a transformed format calculated, for example, according to a coarse BERT or fine BERT type transformation), at least one DOM record / element will be identified. The search results may possibly be associated with multiple questions (including questions that may have generated a match between the identified results and the submitted query). One or more of the additional questions (i.e., questions other than the one that was matched to the query) may be used as separate queries to be resubmitted for search to identify additional content that may be germane to the original query submitted by the user.

[0064] As further shown in FIG. 1, the determination of the answer to the query may be initiated by a user submitting a query 172 via a link 170 established between the station 154a and the interface 130 (as noted with respect to the link established to transfer the source document for ingestion, the link may be based on any type of communication technology or protocol, including wired and wireless communication protocols). The query 172 may be the actual raw question submitted by the user or may be partially or completely transformed (e.g., for privacy and security reasons). For example, the station 154a may apply a transformation commensurate with that applied by the ingestion engine 126 (in which case it may be unnecessary to perform a similar transformation in the query stack). Alternatively or additionally, authentication and encryption processing may be performed on the query 172. The query (query data) 172 is sent to the document processing agent 110 and received at the user query interface 130. Upon receiving the query, a determination may be made as to whether a suitable answer is available in the cache 135 of predefined answers. If there are predefined question-answers (e.g., if the query data matches one or more predefined questions), one or more of the predefined answers are used to generate output data (shown as output data 174) that is returned to the user via link 170 (or via some other link).

[0065] In general, the query data is transformed by the query stack into transformed query data (if not already transformed at station 154a). The transformed data may provide the query in one or more transformed formats that are compatible with the format of the transformed source content stored in DOM repository 140. In some embodiments, the query data may also be used to generate one or more additional questions (e.g., follow-up questions, or questions related to the original query submitted by the user). In situations where an answer to a query is available from an answer cache, the answer itself may be used as a basis for generating one or more further questions that may be related to the cached answer. The query or transformed query is used to search DOM repository 140 via query processing module 136. As noted, the search may be performed as a multi-way process that follows multiple transformation formats used to store the data in DOM repository 140.

[0066] The output generated in response to a submitted query typically includes a pointer to source content available in the customer network 150a. Because the data stored in the repository 140 is populated based on source documents maintained in a document library available in the customer network to which the user submitting the query has access, and because the source documents may not have been stored in their original format in the document processing agent 110 (e.g., for security reasons to protect sensitive data from being compromised), the output returned to the user does not require that the actual answer data be sent back to the user. Instead, the pointer returned as output of the query may identify the address or location of an answer having the appropriate document available to the user in the user's network 150. For example, in the example shown in FIG. 1, the output data is shown as a pointer to the specific location of the answer within document 162a (stored in library 160 along with documents 162b-d). Such pointers may thus include data representing the document 162a (e.g., data representing a network address or memory location where the beginning of the document is found); a specific location of a portion of the document that represents an answer to a question asked by a user at station 154a (e.g., a relative offset from the beginning of the beginning location of the document 162a or from an actual address or memory location where the beginning of the identified portion is found). The pointer data provided in the output data may be included in a metadata field of a DOM record that included the transformed content data that was determined (e.g., by query processing module 136) to match (according to one or more applied matching criteria) to a query submitted by the user. In some embodiments, the output data may include, in addition to or instead of the pointer data, at least a portion of the source content that corresponds to at least a portion of the transformed content, and / or a summary of the source content that corresponds to at least a portion of the transformed content.

[0067] Disambiguation information for processing multiple answer search results As discussed with respect to FIG. 1, a query is submitted to the document processing agent 110 (e.g., via the interface 130) according to which the document processing agent 110 will return a set of matches. Typically, the returned matches will satisfy one or more relevance or consistency criteria, but the set of matches may nevertheless include multiple answers determined to satisfy the query (e.g., because the query is not specific enough and thus multiple answers from the content stored in the DOM repository 140 were found to satisfy one or more consistency criteria applied during the search of the DOM repository 140). For example, the query "how do I install a browser?" searched within the technical electronic documentation of a company that uses multiple different computing systems may discover dozens of different answers related to the various browsers that may be installed on the various available computing platforms and operating systems used by the company. Thus, a query may be directed to substantially the same concept (i.e., the concept of installing a browser) and may yield multiple answers with various possible legitimate answers (i.e., different values ​​relating to substantially the same concept) (e.g., answers related to installing the Chrome™ browser on various computing systems such as Mac™ systems or Windows™-based systems, answers related to installing Explorer™ on various computing systems, etc.). Similarly, answers to a query such as "What is the battery life of my laptop?" may depend on the laptop brand, model, and other unspecified technical details, and thus may be mapped to multiple possible answers providing information about the battery life of various types of computers. Multiple answers resulting from a query may create ambiguity in terms of what the user submitting the query is searching for, which requires further specificity and disambiguation to be resolved.

[0068] Thus, the solutions and techniques described herein address the problem of answering questions from large unstructured data. Every question has the potential to return many plausible answers, and therefore additional information is required to disambiguate the question and select from a set of plausible one or more answers (and / or rank the answers). This additional information may be collected through an interactive dialogue with the user. There are several parts to the proposed solution, including: 1) determining what information is needed to avoid ambiguity between various plausible answers to the same question; 2) linking that information to the appropriate location within the content where the answer appears; 3) once a question has been asked, determining what contextual information is needed to avoid ambiguity between various answers to that question; 4) determining how to interactively solicit information from users; 5) Selecting relevant answers based on contextual information provided.

[0069] In some embodiments, chatbot technology may address the disambiguation challenge by providing tools for designing dialogues such that clarification questions are designed into the dialogue flow. Depending on the answer to one question, the user may be asked another question or provided with an answer once all contextual information is collected. This approach relies on accurately classifying the intent of the question, which requires building a model based on example questions, designing prompts to elicit the required entities, and writing the answer. The solution proposed herein automates many of the operations to obtain disambiguation information.

[0070] Thus, referring to FIG. 2, a flow diagram 200 is shown in which the framework determines disambiguation information for use during the execution of a query (question-answer) procedure. The flow diagram 200 shows the operations in a computing framework similar to that shown in relation to FIG. 1. However, a similar disambiguation information determination process can be performed for other computing frameworks and configurations. The flow diagram 200 includes several parts that correspond to distinct functionalities and stages of the proposed framework. The first processing stage / section depicted inside box 210 includes a process configured to add structure to unstructured content. The process associated with box 210 can be performed based on an implementation similar to that of the document processing agent 110 of FIG. 1. In block 212, a customer (e.g., an entity or organization) provides information (including unstructured information such as documents to be processed and converted into searchable documents) and structured information. For example, in block 212, a communication link is established between an administrator station (such as station 152) and a processing agent (such as processing agent 110 of FIG. 1). An administrator interacting through an administrator station may provide the document processing agent (through an administration interface, such as interface 125 of FIG. 1, which implements connectivity with remote or local computing systems) with information identifying the location of source documents in a repository (such as document library 160 of FIG. 1) that maintains a plurality of source documents, and control the configuration and operation of the functionality of the document processing agent 110 related to the customer's network. Information about the documents to be processed may be in the form of a network address and access information for accessing the repository of documents. Alternatively, the information may include the actual content (raw or transformed in some form). The communication between the administrator station and the administration interface 125 may be established based on any communication technology or protocol, and may include authentication and / or encryption data. For example, the administrator station may send a message providing the document processing agent 110 with the network address of the document library 160 (and / or an identifier of a document in that library that the agent 110 will access and process).

[0071] Customer-provided information can also be provided for other content sources (whether within the customer network 150a or elsewhere), including: a) data stored within collaboration systems (such as Slack, MS Teams, MS productivity Suite (Office 360), Google G-Suite, etc.) (traditional email servers that store email communications between designated senders and recipients can also be processed to capture relevant data); b) data stored within enterprise SaaS applications such as SalesForce, ServiceNow, etc.; c) data within web pages of various websites and various web applications (customer-facing web applications, employee-facing web applications, etc.).

[0072] Once the customer-provided information is received, an ingestion process is performed on the received data (e.g., source documents) at block 214, for example, by using a system similar to document ingestion engine 126 of FIG. 1. As discussed, the ingestion process may include pre-processing applied to the source documents (including segmentation and contextual information identification), followed by conversion of the pre-processed documents into searchable (e.g., searchable QA) content. The ingestion content generated by block 214 is then stored as part of a library of document object model (DOM) objects (shown as knowledge distillation (KD) repository 216). The customer-provided information received at block 212 may also be used to generate (at block 218) an index table of information. The index table may include one or more identifiable concepts or entities (people, places, or other nameable concepts) to later assist in entity / concept-based queries as well as subsequently arranging and ranking the determined search results (in response to future queries). The index table data may be derived by using a learning machine or alternatively (or additionally) by using a natural language process (performed in either block 218 or natural language understanding block 222). The natural language process may use an ontology to recognize and derive semantic and syntactic information from the input data (e.g., user-provided information in this case). Thus, in some embodiments, at least some of the concept identification process may be performed during ingestion time. That is, the determination of concepts associated with the ingested content segments is performed prior to the submission of any query, and typically for most or all of the ingested content segments. Alternatively, the determination of concepts associated with the ingested content segments may be performed after a search has been performed to generate the N best results.For example, extraction of identified concepts may be based solely on, for example, a closeness / proximity criterion between the query vector representation and the vector representation of the content segment and / or a closeness between the semantic meaning of the query and the semantic meaning associated with the captured content segment.

[0073] The ingestion process in block 214 may include various pre-processing operations performed on the content (e.g., to split the source document into manageable-sized segments) while preserving more closely related contextual information. Thus, the document ingestion engine is configured to receive the source document, apply one or more pre-processes to the source document to generate contextual information representative of the structure and content of the source document, and transform the source document based on the contextual information to generate question / answer searchable documents.

[0074] Ingestion of documents may be based on a particular source of data and / or based on a desired or intended presentation of information (e.g., presentation of response data returned in response to query data submitted by a user). For example, if the source of data (content) is from some specialized application (Salesforce, Slack, etc.), ingestion of source content may be configured to perform specific or specialized pre-processing required for the particular source (e.g., converting data located in a specific format record such as chat data or Salesforce records into prose or some other format more conducive to the transformation applied to segmented portions of the source content).

[0075] In some embodiments, document capture may be based on (or may take into account) the particular manner in which the response data is presented. Consider the following three example ways to achieve the presentation of data: In a first exemplary approach, data is presented according to an API-based methodology, where, for example, the answer / paragraph is included in addition to the location (such as the page number or start / end location of the answer snippet) provided to a renderer of various format types (HTML, PDF, Word document, etc.). The renderer may be implemented as a macro or plug-in / extension that allows finding the answer snippet and paragraph in the document and performing special processing of the rendered segment (e.g., by bolding or highlighting parts of the data, segments, etc.). Another exemplary approach for presenting in response to data is to save (e.g., via the capture engine) screenshots of segments in the document that are candidates for presentation during the document processing phase (e.g., effectively pre-rendering the output content). During subsequent presentation of the data identified in response to the query, the client application may pick the most appropriate screenshot that holds the snippet / paragraph. In the third approach to presenting the query results, after a suitable segmentation of the presentation has been generated, every segment of the processed document, which may be available in various formats (e.g. Word document, html, etc.), is converted into a PDF document format containing segments allowing heading and highlighting of the answer snippets (by appropriate connection to the searched segments a one-to-one mapping between the segments is achieved and the start / end positions of the answer snippets are passed from the API to a common PDF renderer).

[0076] An example of a pre-processing procedure is the segmentation of the source content of a source document into multiple document segments. Such segmentation may be performed according to hierarchical rules that semantically relate one portion of the source document to one or more other portions of the source content. For example, a sliding window of fixed or variable size (e.g., 200 words) may be applied to the source content to generate manageable size segments to which content transformations are applied. However, when segmented into small chunks, the content segments may lose important contextual information that would otherwise be available for larger size segments. For example, a passage in the middle of a section of a document may not, in isolation, contain important contextual information such as the section heading, the location of the passage relative to an earlier passage within the section (e.g., if the current passage is a footnote), font size relative to other passages not captured by the particular segment, etc. Thus, in some embodiments, contextual information (e.g., section heading, chapter title, document title, location, font type and size, etc.) may be combined with one or more of the document segments. This pre-processing procedure is illustrated in FIG. 3, which provides a diagram of an exemplary document ingestion procedure 300 with pre-processing to extract contextual information. In FIG. 3, source content 310 (which may be a portion of a source document) has been segmented into segments 320a-n. Each segment has its own individual segmentation content (resulting from applying a segmentation window to the source content) that is combined with contextual information associated with each segment (which may be textual information, numerical information, or both). As can be seen, at least some of the contextual information (i.e., document identifier ("Doc a"), chapter information (chapter S), and heading information (section x)) is common to the segments illustrated in FIG. 3. This allows a transformation subsequently applied to the segments to preserve at least some of the contextual information and thus preserve some of the relevance of the segments that are transformed to themes.

[0077] In some examples, to simplify the segmentation process (to facilitate more efficient search and retrieval), the source document may be segmented to generate overlap between consecutive document segments (without including contextual information added to each segment separately). Thus, for example, in a situation where segments are generated by windows of some particular (fixed or variable) size, the window may be shifted from one position to the next by some predefined fraction of the window size (e.g., 3 / 4 (which would be 150 words for a 200 word window). As a result of the fractional shift, a transform (e.g., vectorization of a BERT-based transform) applied to the overlapped segments may result in some correlation between the segments, which may preserve the relevance between consecutive segments for subsequent QA retrieval. In some embodiments, headline information (and other contextual information) may be added directly to the partitioned segments. Alternatively, headline and contextual information may either be converted to vectors that are then added to vectors resulting from a transform operation applied to the content extracted by the sliding window, or may be combined with the content extracted by the window before the transform is applied to the resulting combined data. By relating neighboring segments to each other (e.g., via a slight shift of the window over the document to form the segments), identification of relevant paragraphs (in response to the submitted query) for retrieval and presentation of top paragraphs and related answer snippets is improved.

[0078] Another pre-process that may be applied during segmentation of the source document relates to handling of tabular information (i.e., when the original content is arranged in a table or grid). This pre-processing is used to expand structured data arranged in a table (or other type of data structure) into a searchable format such as an equivalent text. For example, upon identifying that a portion of the source document is a multi-cell table, a substitute portion is generated to replace the multi-cell table, each of the multiple substitute portions including respective sub-portion content data and contextual information related to the multi-cell table. Additional examples of pre-processing include procedures for associating contextual information with one or more portions of the source document, for example, a) based on information provided by a user in response to one or more questions related to the source document presented to the user, and / or b) based on one or more ground truth examples of question-answer pairs.

[0079] In some instances, contextual information may not be explicitly included with the segment, but instead may need to be discovered and included with the document segment as extended information (in this case, extended contextual information). For example, entity discovery (determining identifiers of relevant entities referenced within a document) may be used to speed up searches during inference (fast-match (FM) searches or detailed match (DM) searches) and to help improve search accuracy and generate improved schemes.

[0080] Consider the following example implementation: • Each search unit (e.g. 200 word window, paragraph, document, etc.) is analysed for unique entities associated with the search unit and also for metadata associated with task specific entities (e.g. HR, author, organisation, etc.). Each search unit is tagged with appropriate intrinsic and metadata entities. During the search, various heuristics may be used that may eliminate many of these search units by briefly identifying them as irrelevant to the query. For example, in one use case where a user's question is determined with a high degree of confidence to be related to some particular subject (e.g., due to user explicit identification of the subject (e.g., a question stating "I have a financial question") or the subject can be inferred via rules or classification engines to be related to a particular subject), all documents / document objects of other subjects (HR, security, etc.) may be eliminated from further consideration, and these documents need not be retrieved in response to the submitted query. A by-product of such filtering is to speed up FM and DM searches. In addition, potential answer units from irrelevant categories will not generate false recognition errors, which in turn helps improve the accuracy of searches.

[0081] Information about a particular entity (or entities) relevant to the user's search can also be used to generate more precise follow-up questions (e.g., to determine various ways to paragraph the input query so that additional possible question-answer pairs can be generated), and also to provide additional context that can be used to search the repository of data (whether as a DOM object in a transformed form or in a user-readable data format).

[0082] As discussed in more detail below, during submission of a query to identify relevant matches from a ingested content database (e.g., DOM library / repository 140 of FIG. 1), contextual information may be obtained (automatically or via direct input by a user) to improve the quality of the matches returned. For example, entity context information may also include information about specific items (e.g., various product brands) and elements that provide additional contextual information to refine the search and improve the output results.

[0083] In some embodiments, document processing (e.g., segmentation) may be performed as two separate tasks. In one processing task, the source document is appropriately segmented and organized into small chunks (e.g., paragraphs) with additional augmentations (e.g., a vector sequence representing a section heading may be appended to the vectors of every paragraph in that section). These augmentations are used to improve search accuracy. In a parallel task, the document is segmented in the most appropriate manner for presentation purposes. The two different result segmentation outputs need to be correlated with each other so that top paragraphs and relevant answer snippets are identified during the search process, but what is presented to the user is the presentation content related to the identified answer snippet (rather than the identified answer snippet). In other words, the system may capture specific passages to facilitate a search operation, and separately capture specific passages to facilitate a presentation operation. In this example, identifying a passage as a result of matching a query with searchable captured content results in the output of the presentation content related to the identified passage.

[0084] Upon segmenting the source document into segments, each segment may be provided to one or more content transformers (or transformers) 330a-m that transform the segment (content and optionally contextual information, although in some embodiments the contextual information may be preserved without transformation) into result transformed content associated with questions and answers related to the original content of the respective segment. In the example of FIG. 3, m transforms are shown, each applied to any one of the segments (such as segment 320j). Although the same segment (e.g. 320j) is shown provided to each of the transforms, in some embodiments different segmentation procedures may be applied to obtain segments of different sizes and configurations as may be required by each individual transform (e.g. coarse-fast-search transform 330a may be configured to be applied to segments of a first segment size, while dense-detail transform 330b may be configured to be applied to segments of a second different size (e.g. strings of several words)).

[0085] As noted above, one example of a transformation that may be applied is a fast lookup (also called fast match or coarse search) transformation based on transforming a fixed-size (and usually large) segment of input data into a vector (which may also be, but need not be, of uniform dimension). The resulting transformed vector may represent possible questions and answers related to the transformed input segment. The resulting vector typically provides a starting point for narrowing the number of possible document objects that need to be more thoroughly searched (e.g., by using content transformed according to another, finer-grained transformation). For example, searching a transformed content repository (e.g., DOM repository 140) based on a match between the fast lookup transformation results and the query data transformed into a representation compatible with the fast lookup transformed content, the resulting search may yield, for example, 1000 possible candidates (or any other number of candidates). A more refined content match may then be performed on the transformed content objects that correspond to the candidates identified by searching the fast lookup transformed content. The fast lookup (coarse) transformation may be performed according to the BERT approach. Another transformation shown in FIG. 3 as being performed by transform unit / module 330b is a dense detail transformation (which in the example of FIG. 3 is also based on the "Bidirectional Encoder Representations from Transforms" (BERT) technique).

[0086] Under the BERT approach, when a query is received, relevant sequences within a document can be quickly identified by identifying portions of the document (e.g., paragraphs) that may contain the answer, and by identifying spans of words within that portion of the document that contain the particular answer (perhaps from a set of objects that may have been identified earlier, e.g., by using fast transformation processing). In some examples, under the BERT approach, the question and answer are concatenated (tokenized, e.g., by using WordPiece embeddings with suitable markers separating the question and the answer) and processed together in a self-attention-based network. The output of the network indicates a score for each possible start position of the answer and a score for each possible end position of the answer, and the overall score of the answer span is the sum of those of the corresponding start and end positions of the answer. That is, a self-attention method is used, where the embedding vectors of the paragraph and the query are mixed through many layers followed by a decision maker layer and segmenter logic that provides an efficient way to determine whether the question can be answered by the paragraph, and if so, where exactly the answer span is within the paragraph.

[0087] In the BERT-based approach, the network is first trained on a masked language model task where certain words are omitted from the input, and the output layer provides a probability distribution over the words of the vocabulary, which can be predicted by the network. Once the network is trained on the masked language model task, the output layer is removed, and for the question answering task, a layer is added to produce a start output, a finish output, and a confidence output, and the network is further trained (e.g., fine-tuned transfer learning) on ​​supervised training data of the target domain (e.g., by using the Stanford Question Answering Dataset (SQuAD)). Once the network is trained for question answering in the target domain, further training can be used to adapt the network to the new domain. Another training strategy used for BERT is next-sentence prediction, where the learning engine is trained to determine which of two input segments (e.g., such segments can be neighboring sentences of a text source) is the first of the two segments. When training the model, both the masked language and next-sentence training procedures may be combined by using an optimization procedure that seeks to minimize a combined loss function. Alternatively or additionally, other training strategies (to achieve context recognition / understanding) may be used separately or in conjunction with one of the aforementioned training strategies of BERT.

[0088] In an exemplary embodiment based on the BERT approach, an implementation called the Two-Leg BERT approach may be used, where much of the processing of a query is separated from the processing of some parts of a document (e.g., paragraphs) where an answer to the query may be found. In general, in the two-leg-BERT approach, a neural net architecture has two "legs": one leg for processing the query; and one leg for processing the paragraphs, and the output of the two legs is an embedding / encoding of some sequence of words of the query and words of the paragraph. These sequences are passed to a question-answering network. The particular way in which this approach is used is to pre-compute BERT embedding sequences of paragraphs and complete the question-answering computation when the query is available. Advantageously, since much of the processing of the paragraphs is done before the query is received, the answer to the query may be computed with less delay compared to using a network where the query and each paragraph are then concatenated and processed together. Paragraphs are typically much longer than queries (e.g., 200-300 words vs. 6-10 words), so preprocessing is particularly effective. When sequential queries are applied to the same paragraph, the overall amount of computation can be reduced because the output of the paragraph legs can be reused for each query. Low latency and reduced total computation can also be advantageous in server-based solutions. As noted, in the implementations described herein, the BERT-based processing of the source document generates transformed content that is typically stored in a repository (such as the DOM repository 140 of FIG. 1). The underlying documents from which the BERT-based transformed content is generated can be similarly retained and associated with the resulting transformed content (as well as with corresponding transformed content obtained via other transformations).

[0089] In some embodiments, a BERT-based transformer (e.g., used for fast coarse transform and / or fine detail transform) may be implemented according to an encoder-based configuration. For example, a BERT-based transformer structure may include multiple stacked encoder cells, where an input encoder cell receives and processes an entire input sequence (e.g., a sentence). By processing the entire input sentence, a BERT-based implementation may process and learn contextual relationships between individual parts (e.g., words in the input sequence). The encoder layers may be realized by one or more self-attention heads (e.g., configured to determine relationships between various parts (e.g., words in a sentence of input data)) followed by a feed-forward network. The outputs of various layers in the encoder implementation may be directed to a normalization layer to obtain an appropriately structured result output for further processing by subsequent layers.

[0090] It should be noted that in some embodiments, a fast lookup vector transform (which converts segments into compact-sized numeric vectors) may be applied to the tokenized version of the text (e.g., some transforms, such as those achieved via the BERT process, may already have been performed to generate the intermediate (e.g., tokenized) content to which the fast lookup transform is then applied).

[0091] The transformation module (fast transformation, BERT-based, or any other type of transformation) may be implemented via a pre-trained neural net to generate transformation content related to question-answer pairs. Other transformation implementations may be realized by using filters and algorithmic transformations. Training of the neural net implementation may be achieved with a large training sample of question-answer ground truth that may be publicly available or may be developed internally / privately by a customer using the system 100 to manage its document library.

[0092] Returning to FIG. 2, in some embodiments, part 1 processing 210 may be configured to determine (e.g., by using ingestion process block 214) dimensions of ambiguity in the data (which may be done as part of intra-document processing operations implemented, for example, by an ingestion engine similar to document ingestion engine 126). The dimensions may be in the form of categories with multiple values. These dimensions may arise from knowledge workers or may be learned from analyzing the data by using one or more ontologies of concepts (e.g., by using a learning machine implementation), such as the source content of the segments in combination with associated metadata (e.g., metadata such as document titles, section headings, etc. that provide contextual information regarding the relevance of the analyzed segment to a particular concept). The ontology of concepts may be as specific or coarse as desired depending on the training data and the desired implementation and availability of ontologies. For example, in the example discussed above in which the query "how do I install a browser?" is posed, the ingestion time (alternatively, query time) analysis may identify browser as a previously learned concept (parametrically or textually represented) that can be searched for in a repository of searchable content (in a DOM repository or in some other database or repository of searchable content). Such a dimension (also called a concept) may be formulated as a set of category / value tuples. Concept-related contextual information (in the form of metadata) that indicates where a passage is specific to a category / value pair attaches the metadata to the content. Two passages in the content identified as possible answers to the query are ambiguous along a particular dimension if they have metadata with the same category and different values. Content may be annotated by using techniques such as extracting entities from text within paragraphs or headings; gathering information from titles, locations, and other data about files; or collecting tags added by content owners or others who annotate the data. Concept identification analysis may also assign values ​​to the concepts that represent the relationship and relevance between the identified concept and the segment being analyzed.For example, a source content segment may be associated with a heading or document title indicator that the document pertains to a Windows™-based computing device. The concept of "browser" may then be assigned a value that indicates that, for the particular segment being analyzed, the concept of "browser" pertains to a Windows™-based computing device.

[0093] The second part of the framework depicted in FIG. 2 involves the submission of a query by a user and the analysis of the results of the unstructured search. In the second part (part 2), standard search techniques may be performed, which may then return a set of plausible answers to the question based on one or more matching criteria for the query, or a representation thereof (e.g., the captured representation based on a transformation of the source query content similar to that applied to the source document), and the searchable content managed by the system 200. In this part of the process in FIG. 2, metadata related to the identified or matched answers are collected, and the answers are returned along with a set of categories / concepts and a value determined for each category / concept. As noted, the concepts / categories may have been derived during ingestion time or query time. Categories with answers having various values ​​are ambiguous along the categorical dimension. It should be noted that the categories / concepts may either have been computed during the ingest process (as described above) or may be computed after retrieval of a set of possible plausible answers (i.e., categories (concepts) and the computation of value tuples may be done at search time (and may be done in a manner similar to that described above). Determining concepts relevant to answer matches during search time has the advantage of needing to determine concepts of only a limited set of answers (i.e., the N best answers) instead of needing to determine concepts of every ingested content segment, thus providing computational efficiency (in terms of computational effort that would otherwise be required if extracted concepts were derived for all ingested content segments).

[0094] As shown, a user provides a query input in the form of a question or alternatively as a more structured search query (e.g., by specifying search terms / values ​​for specific fields) via a user interface 230. The user interface 230 may include a user-side interface with which the user directly interacts (e.g., a graphical interface implemented as an API or in a browser-based implementation, a voice-based interface, etc.) in communication with a server-side interface (e.g., in an implementation where a central document agent serving multiple clients is used) such as the user-input interface 130 depicted in FIG. 1 running on the user's device in communication with a communication module that receives and processes the input from the user. The query data (represented by block 220) may be optionally processed, for example, in a natural language understanding block 222, to extract entity information (including determining entity information / concepts related to the query data (e.g., according to a similar ontology used to perform NLP on the customer-provided information delivered in block 212)), based on which a search operation is performed on the query data. For example, the entity information, and possibly the determined concepts for the query, may be used to perform a search (according to one or more predefined matching criteria) to identify answers that match the content of the query (e.g., based on the vector representation of these queries and the retrieved content segments) and / or that match the determined concepts and metadata of the query and the retrieved content segments. The entity information and the determined concepts may also be used to determine the degree or level of match between the query data and the searchable content, according to the matching criteria used. The extracted entity information, including related concepts / categories derived from the query data, as well as other contextual data related to the query or user (including location information and personal and device information related to the submitted query or user, etc.) are collected and included with the dialogue context data represented by the dialogue context block 224.The interaction context block can be dynamically updated as more context information becomes available during query execution and as search results are iteratively refined.

[0095] The query data is then transformed (represented by block 226) into processed query data that is compatible with the transformed source content (e.g., compatible with one or more of the transformed content records in the KD 216 repository). The processed query data thus includes the resulting transformed vectors, and may also include discovered index types and values ​​(derived based on the NLP operations performed in block 222 and based on other discovery processes performed to determine relevant concepts and other contextual data related to the query data).

[0096] The processed query data is then used to search a repository of searchable content, for example, according to a content retrieval / matching process similar to that performed by the query processing module 136. For example, and as indicated by the QA alignment block 240, the searchable content is compared to the query vector (where the query data may have been transformed into multiple vectors (e.g., vectors for fast (coarse) search, vectors for fine search, etc.)) to identify content vectors resulting from the ingestion process (e.g., pre-processing and BERT-based transformation) performed on the source content corresponding to passages / excerpts of the source document. When searching the searchable content of the source document, the QA alignment block 240 may apply one or more match criteria to identify valid search results. For example, the distance between the query vector and the searchable content vector may be required to be sufficiently close (i.e., the vector distance must be less than a certain threshold). Other match or closeness criteria between the transformed vectors (query transformed vector and content transformed vector) may be used. Other matching criteria that may be required to be satisfied in identifying relevant search results may include criteria that require a certain proximity between the contextual query data and the contextual information associated with the identified content records. For example, in addition to the vector closeness criterion between the query vector and the content vector being satisfied, the matching process may also require that the query and the content records share the same (or similar) entity information (or concept / category identifiers). Additional matching criteria may also be required.

[0097] According to the performance of the search / match process by block 240, the N best matches are identified and provided as intermediate result output 242. The search results may include vectors or parameter values ​​(associated with the identified / matched records), source content passages / segments associated with the vector / parameter representation resulting from the transformation applied to the segmented content, contextual information (including entity identifiers, concepts / categories determined for related content), etc. As noted, the identification of multiple plausible result records (i.e., in a situation where N>1) may indicate that there is ambiguity in the results of the search, perhaps because the search was not specific enough to result in multiple legitimate answers. The search results output 242 may then be processed (e.g., for filtering via disambiguation processing and / or based on other techniques described herein) to generate a filtered set of answers (schematically depicted as block 244) that may be provided as output to the user.

[0098] The determination of the ambiguity of the identified matches is performed by the query ambiguity detector 250. In some embodiments, determining whether two or more of the results / answers are generated in response to the query may be performed by identifying one or more concepts associated with the two or more results / answers (e.g., by applying a concept ontology by a learning machine (performed by the processing stage 210), by applying the concept ontology to the analyzed source content, and / or by the natural language understanding block 222 of the framework 200) and determining that at least one of the one or more identified concepts is associated with at least some different respective values ​​of the multiple matches. In the example of the query "how do i install a browser?", two possible answers each associated with the concept "install a browser" may have different values ​​of the identified concept, which may correspond, for example, to an answer related to a Mac™-based computing system and an answer related to a Windows™-based system. In some embodiments, the presence of multiple answers may not necessarily be considered to create an ambiguity. For example, the framework of FIG. 2 may be configured to detect an ambiguity condition only if the number of matches exceeds a certain answer threshold. On the other hand, if the search returns fewer matches than the answer threshold number, the user may be permitted to consider all found matches.

[0099] Some of the matches may be associated with different concepts / categories, in which case there would not necessarily be ambiguity between the matches (e.g., because there would not necessarily be a common dimension shared by some of the matches (having different (conflicting) values ​​of a shared concept)). In those situations, framework 200 may provide some or all of the matches to the user, or may attempt to eliminate some of the answers by obtaining disambiguating information from the user to determine a particular concept of interest to the user (e.g., by presenting the user with a visual prompt related to at least some of the concepts determined to be matched).

[0100] During the disambiguation process, following removal of some of the matches, the remaining answers in the refined (disambiguated) set of matches may be further disambiguated by identifying another (secondary) concept that is shared by some of the remaining answers and that is related to different (i.e., conflicting) concept values ​​of the other identified concepts. For example, in the case of the query "how to install a browser?", a first disambiguation iteration may remove Mac™-based computing devices, but leave as dependent a number of possible answers related to Windows™-based browser installations. Under the techniques described herein, a second concept (e.g., operating system version number) may be identified for the remaining answers, and the user would be prompted with another request to specify the version number of the operating system on which the browser is installed.

[0101] In some embodiments, multiple disambiguating concepts may be identified for a particular set of matches, and then the user is prompted to provide disambiguation information for all (or some) of the identified concepts. For example, in an initial disambiguation iteration, the user may be asked to provide the operating system, version, and hardware information of the user's device (to disambiguate browser installation queries). When the user is prompted to provide responses to multiple disambiguating concepts, the user does not necessarily have to provide responses for all prompted concepts, but may instead provide response data for fewer concepts than were prompted. Any response data provided by the user may perform some level of disambiguation, thereby reducing some of the information entropy of the set of matches (e.g., may remove one or more matches that are deemed not relevant in light of the user's response data). The user may also decide to forgo any disambiguation process and simply be provided with all returned matches.

[0102] Obtaining the disambiguation information (e.g., by query ambiguity detector 250 or by some other component / process of framework 200) may be accomplished through several ways. In some embodiments, the disambiguation information may be based on available contextual information. Such contextual information may include information associated with recent query transactions (including contextual information related to recent queries submitted by the user submitting the current transaction). For example, if the user has previously submitted a query seeking technical information related to Windows™-based systems, framework 200 may consequently weight (when selecting or ranking answers resulting from the current query) those answers related to Windows-based systems more heavily (e.g., in response to the query "how to install a browser", weighting more heavily answers related to installing a browser on a Windows™-based system).

[0103] Other readily available (i.e., not requiring solicitation of other information from the user) contextual information may include any other information captured at the time the user submits a query (including the user's location information and information indicative of what the user is considering (e.g., what the user is currently looking at)), based on which contextual inference may be made to select one or more of the generated answers. Another example of location-related contextual information that may be used to disambiguate matches (either during the query matching process or after query results have been identified) is the use of map-based information. In some embodiments, an interactive map representation (of a geographic area) may be provided alongside or separate from a visual disambiguation interface (presenting a prompt to solicit a user's response to the identified concepts). The map may be rendered in response to a determination that one of the disambiguation dimensions of match is one of location (e.g., the result includes location-based entity data, or the concept identification process determines that the concept associated with the result is one of location). Alternatively, the map may be displayed in response to a specific user selection to include a map representation with the query results and an interactive interface used to submit the disambiguation results. The user may then use the map to zoom in or out or to select a particular location on the map, and thus to indicate a location value associated with the geographic or location dimension of the returned matches. Based on the selection facilitated via the map rendering, the framework 200 (more specifically, the interaction process block 252 of FIG. 2) may select one or more of the matches whose location dimension values ​​are close (based on application of a proximity or proximity criterion) to the location value indicated by the user's selection on the visual map. For example, in response to a selection on the map of the New York City area, matches with context / entity values ​​or location-related concept values ​​(within some predetermined radius of the New York City area) would be selected (and perhaps ranked according to the closeness of their location values ​​to the value indicated by the user's selection).

[0104] In another example of incorporating available contextual information into the disambiguation process, consider a situation in which a user interacts with an augmented reality system equipped with cameras (and / or other types of sensors), where one or more of the cameras will be pointing in the direction where the user is looking. Information within the scene captured by the sensor devices (e.g., image data (which may be processed, for example, by a learning machine, to identify objects and items appearing in the scene)) may be used to provide contextual information to a query incidentally initiated by a user. For example, if a user looks down (and the camera of the augmented reality system similarly follows the direction and orientation of the user's head to point at the scene being viewed by the user), looks at a MagSafe charger (for wireless charging) on ​​his / her phone, and asks "How do I charge my phone?", the QA system (e.g., based on the implementations described herein) will identify different answers to this question (resulting from a search of the DOM repository) (than would be identified if the user had looked down and looked at the car). In this case, the sensors of the augmented reality system are used to determine (or discover) contextual information (e.g., the user's proximity to a MagSafe charger versus proximity to the car), which may be used to filter answers already generated, or even to limit the search (performed in block 240) to only the determined context.

[0105] In some embodiments, the orientation, orientation, and / or location of the sensor device (in this case the camera) itself (which may be determined based on positioning techniques using satellite or ground-based signal analysis) may provide relevant and important contextual information for selecting an answer from a set of matches, or for searching repository data, or for providing feedback in response to a disambiguation prompt. For example, pointing the camera downwards may mean that the information sought via the query pertains to objects that are close to the ground. In another example, the location of the sensor device may be used to restrict a search to answers that have relevance to a particular geographic location of the sensor device (e.g., to determine details related to a particular conference room in which the user is found). Thus, a query such as "how do I turn on my video conferencing camera?" may be modified (or constrained) to retrieve answers (e.g., from relevant manuals or other source documents stored by the company) regarding video cameras located in a particular conference room in which the querying user is found.

[0106] Another example of an augmented reality system (or other type of system with sensors) being used in conjunction with the document processing (e.g. QA type processing) implementations described herein involves a situation where a factory worker (who may be fitted with a streaming body cam (or hard-hat cam)) may pose a query question that is modified by contextual information extracted from the captured video stream. In one situation, the user may ask for information about the functionality or operation of a "machine" or about a certain "product". The captured image or video by the device carried by the user may identify a specific brand or model of machine, and if the user is seeking some specific information about the operation of the machine, the specific model identified via the augmented reality sensor may be used to constrain the search to documents (e.g. user manuals) related to the identified specific machine model. Thus, the streaming by the camera when used in an augmented reality system adapted to assist factory workers may be used to modify the query (e.g. asking for information about a "machine") to take into account the specific machinery identified according to the video stream generated by the camera (used in conjunction with a learning machine to identify objects and items in the scene). In another example related to a factory worker (or technician) scenario, a user may pose a query (e.g., via a voice-based interface, such as an AI-assisted app running on a mobile device carried by the user) asking about the connectivity of a wiring harness. The query may be modified (or constrained) to retrieve answers that may be specific to wiring harnesses that appear in a captured image (or video) of the scene (where a particular model or type of harness may be identified).

[0107] Yet another example where captured image-based data may be used during context discovery or to perform disambiguation is when a scene contains a recognizable code (QR code, barcode, etc.) that can be decoded to extract meaningful contextual information therefrom. For example, in the wiring harness example above, the wiring harness may contain a label with a QR code or barcode that can be automatically decoded when captured by an image capture device carried by a user. Queries posed by the user related to the wiring harness would then be modified (or constrained in some way) such that the answers obtained are related to the QR or barcode identified during context discovery.

[0108] It should be noted that some of the exemplary systems (e.g., augmented reality systems) described herein may be implemented by using augmented reality goggles, while other systems may be implemented by using a camera installed on a smartphone that the user moves to point the camera in the direction of the relevant scene. Some embodiments of such phone-based augmented reality systems may also include an artificial intelligence (AI)-assisted app (e.g., Siri, Alexa, Cortana, etc.) where the user may provide their own queries that are modified based on contextual information determined from the augmented reality system. It should also be noted that other types of mixed-mode input sources (which may combine input from one or more of text entry sources, voice capture sources, image capture sources, etc.) for formulating queries (in the course of searching the QA data repository) may be used.

[0109] As noted, another approach to obtaining disambiguation information (e.g., in addition to using available context information, or if the available context information has not sufficiently removed the matches back to a manageable level) is to dynamically interact with the user to solicit from the user the disambiguation information needed to aid in selecting one or more answers from the initial or remaining answers. As shown in FIG. 2, upon determining (at 250) an ambiguity in the set of matches for a query, a visual interaction process is performed to determine which values ​​(whether entity values ​​or some extract of information contained in the answer) match what the user is looking for (e.g., for one or more of the ambiguous categories or concepts associated with the current set of answer matches). This can be accomplished, for example, by: a) presenting the user with a set of values ​​and a general prompt: "Choose from the following" ("visual form dialogue"); b) Using natural language generation techniques to generate appropriate prompts (e.g., based on the identified concepts associated with the possible answers generating the ambiguity). For example, to avoid ambiguity between two answers that relate to the same (or substantially similar) concept or category but have different values, the visual interaction process may generate a question to the user asking the user to provide more specificity regarding the ambiguous concept. For an example query "How do I install a browser?" that results in two possible answer matches associated with the browser installation concept (one answer match for Mac™-based computing systems and one answer match for Windows™-based systems), the interactive message to the user may ask the user "On which computing system (or operating system) do I need to install the browser?" Alternatively, the message to the user may list various computing systems and ask the user to specify which computing system the user needs information for. In an embodiment where there are multiple concepts / categories identified within a set of matches, the disambiguation process may seek to obtain disambiguation information for the most general concept determined for the multiple matches. Alternatively, other criteria or policies for identifying promising concepts or categories for disambiguation (e.g., by identifying concepts, entity types, or categories that are most likely to distinguish between various answers (i.e., have the greatest disambiguation impact or highest information gain for eliminating most answers)) may be used. c) selecting from a list of pre-written prompts, for example for chatbots in relevant domains; and d) continuing to interact with the user until at least some of the ambiguities are resolved (eg, a certain threshold number of answers is reached, or in some instances all ambiguities are resolved).

[0110] Thus, the dynamic interactive process implemented by the interaction block 252 (which may implement a visual interface, an audio interface, etc.) is configured to generate output data for prompting the user to provide clarification information, and to select at least one of a plurality of consistency based at least in part on the clarification information provided by the user in response to the generated prompt data. The interactive process configured to generate output data for prompting the user to provide clarification information may be configured to automatically generate an output prompt based on, for example, one or more of the following: generating a list having selectable items corresponding to different values ​​of one or more context categories; applying natural language processing to the identified plurality of consistency to generate a prompt having a list of selectable items from which the user selects one or more of the selectable items; and / or selecting one or more items from a set of predefined prompts.

[0111] The interactive disambiguation process (in conjunction with the query ambiguity detection process, which may be implemented in part in block 250 of FIG. 2) may be configured to exclude one or more of the multiple matches based on disambiguation information provided by the user. This process may be repeated until some or all of the ambiguities are resolved by iteratively generating refined output data based on matches that have not been excluded from the set of identified matches, to prompt the user to iteratively provide another disambiguation information to identify the best match from the identified multiple matches. Thus, the data provided by the user through the interactive process (represented in FIG. 2 by data block 254) is used to filter the content to provide the most relevant answers.

[0112] In some embodiments, the user's additional interactive input may include a specific selection of one of the matches presented to the user (as an intermediate set of matches or a disambiguating set of matches presented to the user as display data 258 via the submit answer process 256 depicted in FIG. 2 ) (which may cause further removal of some of the displayed matches having conflicting concept values ​​of the concepts associated with the selected match). The selection of a specific match may also cause a re-ranking of the displayed matches. For example, concepts associated with the user-selected match that are most commonly shared with other displayed matches are identified. The presented matches may be reordered or re-ranked such that matches (of the identified concepts) having concept values ​​that are closest (in a semantic or numerical manner) to the concept value of the selected match are ranked higher than matches whose concept values ​​of the identified concepts are more distant.

[0113] The disambiguation process may be performed according to one of several possible policies. Such policies include: i) a pre-fixed policy (i.e. what should be disambiguated and in what order is pre-fixed), ii) a policy that seeks to optimize an objective function, e.g., using the one that reduces the maximum amount of entropy among the ambiguous concepts, or iii) a policy that implements a visual display of multiple concepts and lets the user decide which concepts the user considers more important. In some situations, different policies may be utilized at different points in the execution of a query. For example, when a query is initially submitted, the framework 200 may automatically seek to evaluate which contextual information (entity identifiers, extracted concepts, etc.) is likely to be most useful. For example, the “author” context may not provide much information that can be used to logically place answers or to eliminate answers. On the other hand, the “operating system” may split the initial generated answers in a 60:40 ratio. Only after the most relevant contextual information (if available) is selected, the user may be asked about the word cloud (size=potential importance for disambiguation), etc., either with a yes / no or more open-ended question. Thus, in such a situation, in the first stage of processing the query answer, a policy may be applied first that seeks to optimize an objective function to reduce the information entropy of the initially returned matches (e.g., policy type (ii) above). After the objective function policy is applied, a policy may be applied that implements a visual display of multiple concepts to solicit disambiguation information from the user regarding the matches of the refined set.

[0114] It should be noted that when the user is prompted to provide selection / disambiguation data in response to the generated message (from the interaction process 252), the set of matches is set to be "post-filtered" (some of the previous matches are removed based on the disambiguation data) to generate a refined set of matches. If the original query itself includes disambiguation data (e.g., specifying a priori values ​​of one or more concepts or categories), the list of matches (provided in the output 242 block) may be generated based on that originally provided disambiguation data, and the result set of matches is said to be "pre-filtered." At least some of the disambiguation processes, techniques, and operations for post-filtering implementations described herein may also be implemented to pre-filter some operations.

[0115] Referring now to FIG. 4, a flow chart of an exemplary procedure 400 for context clarification and disambiguation of a question answering process is shown. The procedure includes receiving 410 query data representing a query related to source content of one or more source documents at a local device (e.g., document processing agent 110 of FIG. 1) from a remote device (e.g., a customer station that may be physically implemented on the same computing system as the local device but may be a logically separate unit). The procedure 400 further includes invoking a search 420 (e.g., by a third-party search engine or by a locally available embedded search engine) of a data repository (e.g., repository 140 of FIG. 1 or KD repository 216 of FIG. 2) that maintains data portions related to one or more source documents to determine a set of matches between the query data and the data portions maintained in the data repository. In some embodiments, the data portions maintained in the data repository may include transformed portions of source content that have been transformed according to one or more content transformation procedures. In such an embodiment, inducing a search of a data repository that maintains the data portions may include: transforming the query data into transformed query data that is compatible with the transformed source content; and inducing a search of the transformed content maintained in the data repository to identify one or more candidate portions within the transformed source content that match the transformed query data according to one or more criteria. The transformed portions of the source content include data portions that have been transformed according to a "Bidirectional Encoder Representations from Transformations" (BERT) process. In some examples, the one or more transformations may include one or more of, for example, a coarse linearization transformation to generate a coarse numerical vector representing the content of multiple document segments of the source content, or a dense fine-grain transformation to generate a dense fine-grain transformed content record representing the content of the multiple document segments.

[0116] 4, the procedure 400 further includes identifying 430 one or more concepts (e.g., extracted categories of subject matter determined based on ontologies used in machine learning and / or various natural language processing procedures, recognizable entities and their semantic equivalents / synonyms, etc.) associated with the plurality of matches, where at least one of the one or more identified concepts is associated with at least some of the plurality of matches and includes distinct respective values ​​associated with at least some of the plurality of matches. It should be noted that in some embodiments, the identifying operation may follow an initial determination that the set of matches includes multiple matches.

[0117] In response to determining that the set of matches includes a plurality of matches, the procedure 400 additionally includes obtaining 440 disambiguation information associated with at least one of the one or more identified concepts, and selecting 450 at least one of the plurality of matches based on the obtained disambiguation information associated with at least one of the one or more identified concepts.

[0118] In some examples, obtaining the disambiguation information may include obtaining query context information of recent query transactions conducted in association with the source content. In such examples, selecting at least one of the plurality of matches may include selecting at least one of the plurality of matches based at least in part on the query context information of recent query transactions conducted in association with the source content.

[0119] In some embodiments, obtaining the disambiguation information may include generating prompt data for prompting the user to provide clarification information. In such embodiments, selecting at least one of the multiple consistency may include selecting at least one of the multiple consistency based at least in part on clarification information provided by the user in response to the generated prompt data. Generating prompt data for prompting the user to provide clarification information may include automatically generating an output prompt, for example, based on one or more of the following: generating a list having selectable items corresponding to different values ​​of one or more context categories; applying natural language processing to the identified multiple consistency to generate a prompt having a list of selectable items from which the user selects one or more of the selectable items; and / or selecting one or more items from a set of predefined prompts. Selecting at least one of the multiple consistency may include excluding one or more of the multiple consistency based on clarification information provided by the user. In such embodiments, procedure 400 further includes iteratively generating refined prompt data based on non-excluded consistency from the set of identified consistency to iteratively prompt the user to provide another clarification information to identify an optimal consistency from the identified multiple consistency. Generating the prompt data may include rendering a graphical representation of a map to prompt the user to indicate a geographic location, and selecting at least one of the plurality of matches based at least in part on the disambiguation information may include selecting at least one of the plurality of matches in response to at least one of the plurality of matches determined to be associated with the geographic location indicated by the user.

[0120] In some embodiments, each of the plurality of consistency matches may be associated with content context information associated with the data portion maintained in the data repository. In such embodiments, identifying one or more concepts associated with the plurality of consistency matches may include identifying the one or more concepts based at least in part on the content context information associated with each of the plurality of consistency matches. The content context information associated with each data portion may be generated, for example, by one or more of the following: a) applying one or more pre-processes to the one or more source documents to generate document context information representative of the structure and content of the one or more source documents, and transforming the one or more source documents based on the context information to generate one or more question-answer searchable documents, b) segmenting the one or more source documents into a plurality of document segments, identifying, for at least one segment of the plurality of document segments, at least one segment descriptor including one or more of at least one entity associated with the at least one segment, at least one task associated with the at least one segment, or a thematic descriptor associated with the at least one segment, and tagging the at least one segment with the at least one descriptor, and / or c) adding user annotations to one or more of the data portions. The content context information of each match of the plurality of matches may include data representing values ​​of a plurality of context categories, and identifying the one or more concepts associated with the plurality of matches includes determining whether at least two of the plurality of matches are associated with different values ​​of a particular context category from the plurality of context categories. In such an example, searching the data repository to determine a set of matches between the query data and data portions maintained in the data repository may include placing matches in the set of matches into groups that each share one or more of a plurality of context categories.

[0121] In some examples, the query data may include query context data, and causing the search of the data repository to determine a set of matches may include having the search of the data repository identify data portions related to the query context data included in the query data. This is referred to as a pre-filtering operation, where the context data, including, for example, extracted concepts, entity identifiers (names, places, items), location data and other available data related to the query, the user who submitted the query, the station through which the query was submitted, etc., may be used to assist the search to determine more relevant search results. The query context data may include geographic location data defined by a user via a graphical representation of a map, and selecting at least one of the multiple matches based at least in part on the disambiguation information may include having the search of the data repository identify data portions related to the geographic location data defined by the user. The query context data may include category data defining one or more categories from a plurality of context categories, and causing the search of the data repository to identify matches related to the defined one or more categories from a plurality of context categories defined in the query context data.

[0122] Obtaining disambiguation information related to at least one of the one or more identified concepts may include, for example, obtaining the disambiguation information according to one of the following: i) a first disambiguation policy that specifies a predetermined ordering of concepts selected from the one or more identified concepts in which a relevance of a plurality of matches to each of the plurality of concepts is determined; ii) a second disambiguation policy for selecting a concept from the one or more identified concepts that optimizes an objective function to reduce a level of ambiguity among the plurality of matches; or iii) a third disambiguation policy that visually prompts a user for feedback related to the one or more identified concepts for selecting at least one of the plurality of matches.

[0123] In implementations based on learning machines, various types of learning architectures, configurations, and / or implementation techniques may be used. Examples of learning machines include neural nets, including convolutional neural nets (CNNs), feedforward neural networks, recurrent neural networks (RNNs), and the like. A feedforward network includes one or more layers of nodes ("neurons" or "learning elements") with connections to one or more portions of input data. In a feedforward network, the connectivity of the inputs of the nodes and layers is such that the input data and intermediate data propagate in a forward direction toward the output of the network. There are typically no feedback loops or cycles in the configuration / structure of a feedforward network. Convolutional layers allow the network to efficiently learn features by applying the same learned transformation to subnodes of data. Other examples of learning engine techniques / architectures that can be used include generating an autoencoder; and using dense layers of networks to correlate with the probability of future events via a support vector machine; building regression or classification neural network models that prescribe specific outputs from the data (based on training that reflects correlations between similar records and identified outputs), and so forth.

[0124] Neural networks (as well as other network configurations and implementations for implementing the various procedures and operations described herein) may be implemented on any computing platform (including computing platforms that include one or more microprocessors, microcontrollers, and / or digital signal processors that provide processing functionality as well as other computational and control functionality). The computing platform may include one or more CPUs, one or more graphic processing units (e.g., GPUs (such as NVIDIA GPUs) that may be programmed according to the CUDA C platform, and may also include special purpose logic circuitry (e.g., FPGAs (field programmable gate arrays), ASICs (application specific integrated circuits), DSP processors, accelerated processing units (APUs), application processors, customized special purpose circuits, etc.) to at least partially implement the processes and functionality of the neural networks, processes and methods described herein. Computing platforms used to implement neural networks also typically include memory for storing data and software instructions for executing the programmed functionality within the device. Generally speaking, a computer-accessible storage medium may include any non-transitory storage medium that is accessible by a computer during use to provide instructions and / or data to a computer. For example, a computer-accessible storage medium may include storage media such as magnetic or optical disks, and semiconductor (solid state) memory, DRAM, SRAM, and the like.

[0125] The various learning processes implemented through the use of neural networks described herein can be configured and programmed using TensorFlow, an open source software library used for machine learning applications such as neural networks. Other programming platforms that can be employed include keras (an open source neural network library) building blocks, NumPy (an open source programming library useful for implementing modules to process arrays) building blocks, and the like.

[0126] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly or customarily understood. As used herein, the articles "a" and "an" refer to one or more (i.e., at least one) of the grammatical object of the same article. As an example, "an element" means one element or more than one element. As used herein in reference to measurable values ​​such as amounts, time durations, and the like, "about" and / or "approximately" include deviations of ±20% or ±10%, ±5% or +0.1% from the stated value, as such deviations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein. As used herein in reference to measurable values ​​such as amounts, time durations, and the like, "substantially" also includes deviations of ±20% or ±10%, ±5% or +0.1% from the stated value, as such deviations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein.

[0127] As used herein (including in the claims), "or" used in a list of items prefaced by "at least one of" or "one or more of" refers to a disjunctive list, such as a list of "at least one of A, B, or C" meaning A or B, or C or AB or AC or BC or ABC (i.e., A and B and C), or a combination of two or more features (e.g., AA, AAB, ABBC, etc.). Also, as used herein, unless otherwise stated, a statement that a function or operation is "based on" an item or state means "the function or operation is based on the stated item or state, and may be based on one or more items and / or states in addition to the stated item or state."

[0128] Although specific embodiments have been disclosed in detail herein, this has been done as an example for purposes of illustration only, and is not intended to be a limitation of the scope of the invention as defined by the appended claims. Any of the features of the disclosed embodiments described herein can be combined with each other, rearranged, etc., within the scope of the invention to generate more embodiments. Several other aspects, advantages, and modifications are contemplated to fall within the scope of the claims provided below. The claims presented represent at least some of the embodiments and features disclosed herein. Other unclaimed embodiments and features are also contemplated.

Claims

1. receiving query data at the local device from the remote device representing a question related to source content of the one or more source documents; causing a search of a data repository maintaining data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions maintained in the data repository; identifying one or more concepts associated with the set of a plurality of consistency matches, at least one of the one or more identified concepts being associated with at least some of the plurality of consistency matches and including distinct respective values ​​associated with the at least some of the plurality of consistency matches; obtaining disambiguation information associated with the at least one of the one or more identified concepts; and selecting at least one of the plurality of matches based on the obtained disambiguation information associated with the at least one of the one or more identified concepts. A method comprising:

2. obtaining the disambiguation information includes obtaining query context information of recent query transactions conducted in association with the source content; 2. The method of claim 1, wherein selecting at least one of the plurality of consistency matches comprises selecting at least one of the plurality of consistency matches based at least in part on the query context information of the recent query transaction performed in association with the source content.

3. obtaining the disambiguation information includes generating prompt data to prompt a user to provide disambiguation information; 3. The method of claim 1 or 2, wherein selecting at least one of the plurality of consistency comprises selecting at least one of the plurality of consistency based at least in part on the clarification information provided by the user in response to the generated prompt data.

4. 4. The method of claim 3, wherein generating the prompt data for prompting the user to provide the clarification information comprises automatically generating an output prompt based on one or more of the following: generating a list having selectable items corresponding to different values ​​of one or more context categories; applying natural language processing to the identified plurality of matches to generate a prompt having a list of selectable items from which the user selects one or more of the selectable items; or Selecting one or more items from a set of predefined prompts.

5. 4. The method of claim 3, wherein selecting at least one of the plurality of matches comprises eliminating one or more of the plurality of matches based on the disambiguation information provided by the user, The method further includes iteratively generating refined prompt data based on non-excluded matches from the set of identified matches to prompt the user to iteratively provide further clarification information to identify an optimal match from the identified plurality of matches.

6. generating the prompt data includes rendering a graphical representation of a map to prompt the user to indicate a geographic location; 4. The method of claim 3, wherein selecting the at least one of the plurality of matches based at least in part on the disambiguation information comprises selecting the at least one of the plurality of matches in response to the at least one of the plurality of matches being determined to be relevant to the geographic location indicated by the user.

7. Each of the plurality of consistency is associated with content context information associated with the data portion maintained in the data repository; 7. The method of claim 1, wherein identifying the one or more concepts associated with the plurality of matches comprises identifying the one or more concepts based at least in part on the content context information associated with each of the plurality of matches.

8. The method of claim 7, wherein the content context information associated with each data portion is generated by one or more of the following: a) applying one or more pre-processes to the one or more source documents to generate document context information representative of structure and content of the one or more source documents, and transforming the one or more source documents based on the context information to generate one or more question-answer searchable documents; b) segmenting the one or more source documents into a plurality of document segments, identifying, for at least one segment of the plurality of document segments, at least one segment descriptor comprising one or more of at least one entity associated with the at least one segment, at least one task associated with the at least one segment, or a thematic descriptor associated with the at least one segment, and tagging the at least one segment with the at least one descriptor; or c) adding user annotations to one or more of said data portions;

9. the content context information for each consistency in the plurality of consistency includes data representative of values ​​of a plurality of context categories; Identifying the one or more concepts associated with the plurality of matches comprises: The method of claim 7 , comprising determining whether at least two of the plurality of matches are associated with different values ​​of a particular context category from the plurality of context categories.

10. Having the search of the data repository determine the set of matches between the query data and the data portions maintained in the data repository includes: The method of claim 9 , comprising placing the matches in the set of matches into groups that each share one or more of the plurality of context categories.

11. the query data includes query context data; 8. The method of claim 7, wherein having the search of the data repository determine the set of matches comprises having the search of the data repository identify data portions included in the query data that are relevant to the query context data.

12. the query context data includes geographic location data defined by a user via a graphical representation of a map; 12. The method of claim 11, wherein selecting the at least one of the plurality of matches based at least in part on the disambiguation information comprises causing the search of the data repository to identify data portions related to the geographic location data defined by the user.

13. the query context data includes category data defining one or more categories from a plurality of context categories; 12. The method of claim 11, wherein initiating the search of the data repository includes causing the search of the data repository to identify matches related to the defined one or more categories from the plurality of context categories defined in the query context data.

14. the data portions maintained in the data repository include transformed portions of the source content that have been transformed according to one or more content transformation procedures, and causing the search of the data repository that maintains the data portions comprises: converting the query data into converted query data that is compatible with the converted source content; and 14. A method according to any one of claims 1 to 13, comprising searching the transformed content maintained in the data repository to identify one or more candidate portions within the transformed source content alignment according to one or more criteria to the transformed query data.

15. The method of claim 14 , wherein the transformed portion of the source content comprises a data portion transformed according to a Bidirectional Encoder Representation From Transform (BERT) process.

16. 15. The method of claim 14, wherein the one or more transformations include one or more of a coarse linearization transformation to generate coarse numerical vectors representing content of a plurality of document segments of the source content, or a dense-detail transformation to generate dense-detail transformed content records representing the content of the plurality of document segments.

17. The method of any one of claims 1 to 16, wherein obtaining the disambiguation information related to the at least one of the one or more identified concepts comprises obtaining the disambiguation information according to one of the following: i) a first disambiguation policy defining a predetermined ordering of concepts selected from the one or more identified concepts for which relevance of the matches to the respective concepts is determined; ii) a second disambiguation policy for selecting a concept from the one or more identified concepts that optimizes an objective function to reduce a level of ambiguity among the plurality of matches; or iii) a third disambiguation policy that visually prompts a user for feedback related to the one or more identified concepts for selecting at least one of the plurality of matches.

18. a communication unit configured to receive query data from a remote device, the query data representing a question related to source content of one or more source documents; and 11. A system including a controller electrically coupled to the communication unit, the controller configured to: causing a search of a data repository maintaining data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions maintained in the data repository; identifying one or more concepts associated with the plurality of consistency matches, at least one of the one or more identified concepts being associated with at least some of the plurality of consistency matches and including distinct respective values ​​associated with the at least some of the plurality of consistency matches; obtaining disambiguation information associated with the at least one of the one or more identified concepts; and Selecting at least one of the plurality of matches based on the obtained disambiguation information associated with the at least one of the one or more identified concepts.

19. the controller configured to obtain the disambiguation information is configured to obtain query context information of recent query transactions performed in association with the source content; 20. The system of claim 18, wherein the controller configured to select at least one of the plurality of matches is configured to select at least one of the plurality of matches based at least in part on the query context information of the recent query transaction conducted in association with the source content.

20. the controller configured to obtain the disambiguation information is configured to generate prompt data to prompt a user to provide disambiguation information; 20. The system of claim 18 or 19, wherein the controller configured to select at least one of the plurality of consistency is configured to select at least one of the plurality of consistency based at least in part on the clarification information provided by the user in response to the generated prompt data.

21. 21. The system of claim 20, wherein the controller configured to generate the prompt data for prompting the user to provide the clarification information is configured to automatically generate an output prompt based on one or more of the following: generating a list having selectable items corresponding to different values ​​of one or more context categories; applying natural language processing to the identified plurality of matches to generate a prompt having a list of selectable items from which the user selects one or more of the selectable items; or Selecting one or more items from a set of predefined prompts.

22. the controller configured to select at least one of the plurality of matches is configured to exclude one or more of the plurality of matches based on the disambiguation information provided by the user; 21. The system of claim 20, wherein the controller is further configured to iteratively generate refined prompt data based on non-excluded matches from the set of identified matches to prompt the user to iteratively provide further clarification information to identify an optimal match from the identified plurality of matches.

23. the controller configured to generate the prompt data is configured to render a graphical representation of a map to prompt the user to indicate a geographic location; 21. The system of claim 20, wherein the controller configured to select the at least one of the plurality of matches based at least in part on the disambiguation information is configured to select the at least one of the plurality of matches in response to the at least one of the plurality of matches being determined to be associated with the geographic location indicated by the user.

24. Each of the plurality of consistency is associated with content context information associated with the data portion maintained in the data repository; The system of any one of claims 18 to 23, wherein the controller configured to identify the one or more concepts associated with the plurality of matches is configured to identify the one or more concepts based at least in part on the content context information associated with each of the plurality of matches.

25. 25. The system of claim 24, wherein the content context information associated with each data portion is generated by one or more of the following: a) applying one or more pre-processes to the one or more source documents to generate document context information representative of structure and content of the one or more source documents, and transforming the one or more source documents based on the context information to generate one or more question-answer searchable documents; b) segmenting the one or more source documents into a plurality of document segments, identifying for at least one segment of the plurality of document segments at least one segment descriptor comprising one or more of at least one entity associated with the at least one segment, at least one task associated with the at least one segment, or a thematic descriptor associated with the at least one segment, and tagging the at least one segment with the at least one descriptor; or c) adding user annotations to one or more of said data portions;

26. the content context information for each consistency in the plurality of consistency includes data representative of values ​​of a plurality of context categories; The controller configured to identify the one or more concepts associated with the plurality of matches, 25. The system of claim 24, configured to determine whether at least two of the plurality of matches are associated with different values ​​of a particular context category from the plurality of context categories.

27. the controller configured to cause the search of the data repository to determine the set of matches between the query data and the data portions maintained in the data repository; 27. The system of claim 26, configured to arrange the matches in the set of matches into groups that each share one or more of the plurality of context categories.

28. the query data includes query context data; 25. The system of claim 24, wherein the controller configured to cause the search of the data repository to determine the set of matches is configured to cause the search of the data repository to identify data portions related to the query context data included in the query data.

29. The system of any one of claims 18 to 28, wherein the controller configured to obtain the disambiguation information related to the at least one of the one or more identified concepts is configured to obtain the disambiguation information according to one of the following: i) a first disambiguation policy defining a predetermined ordering of concepts selected from the one or more identified concepts for which relevance of the matches to the respective concepts is determined; ii) a second disambiguation policy for selecting a concept from the one or more identified concepts that optimizes an objective function to reduce a level of ambiguity among the plurality of matches; or iii) a third disambiguation policy that visually prompts a user for feedback related to the one or more identified concepts for selecting the at least one of the plurality of matches.

30. A non-transitory computer-readable medium programmed with instructions executable on one or more processors of a computing system to: receiving query data at the local device from the remote device representing a question related to source content of the one or more source documents; causing a search of a data repository maintaining data portions related to the one or more source documents to determine a set of a plurality of matches between the query data and the data portions maintained in the data repository; identifying one or more concepts associated with the plurality of consistency matches, at least one of the one or more identified concepts being associated with at least some of the plurality of consistency matches and including distinct respective values ​​associated with the at least some of the plurality of consistency matches; obtaining disambiguation information associated with the at least one of the one or more identified concepts; and Selecting at least one of the plurality of matches based on the obtained disambiguation information associated with the at least one of the one or more identified concepts.

31. a communication unit configured to receive query data from a remote device, the query data representing a question related to source content of one or more source documents; and A computing apparatus including one or more programmable devices for performing a question answering process according to the steps of any one of claims 1 to 17.

32. A non-transitory computer readable medium programmed with a set of computer instructions executable on a processor which, when executed, causes operations including the method steps of any one of claims 1 to 17.

Images