Directed probing of memory networks for knowledge base construction

Abstract

A system for maintaining a knowledge base, the system comprising devices for: (i) generating a first interface to receive queries for transmission to a question-answering system and providing a response including a list of one or more suggested triples; (ii) after selecting a specific triple, generating a second interface to: provide at least one evidence record, the evidence record including a text span supporting the specific triple, and providing one or more control elements associated with each evidence record, the one or more control elements including at least one of: a first control element selectively referencing its corresponding evidence record and text span as supporting the specific triple, or a second control element selectively preventing its corresponding evidence record and text span from being referenced as supporting the specific triple; and (iii) generating a data structure based on the selection of the one or more control elements, thereby updating the knowledge base.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 63 / 010,359, filed April 15, 2020, entitled “Directed Probing of Memory Networks for Knowledge Base Construction,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to systems and / or methods for building and / or maintaining knowledge bases, and more specifically, to building and / or maintaining knowledge bases using editing devices. Background Technology

[0004] Traditional computers include a processor that accesses memory to execute program instructions stored in the memory. The processor executes program instructions and uses data stored in memory as input to compute a result output. Neural networks, on the other hand, include multiple interconnected processor nodes that operate in parallel and are organized by layers (e.g., an input layer, one or more hidden layers, and an output layer). The input to each layer is the output of one or more previous layers (e.g., input layer to a first hidden layer, to a second hidden layer, to an output layer, and / or similar layers). Generally, each processor node is connected to some or all of the other nodes in a weighted manner. The output level from the connected processors, multiplied by the weight of the connection from the first processor to the second processor, constitutes a portion of the input signal to the second processor. The total input signal to the second processor is the sum of the weighted outputs of the processors connected to the input.

[0005] These weights are updated according to various types of learning rules. For example, the most common learning rule requires that the network know the output to be learned for each of several inputs. The network is trained by applying the inputs, computing the final output, comparing the final output with the expected output, and then changing the weights to slightly reduce the difference (error) between the actual output and the expected output. This is repeated many times until the difference between the actual output and the expected output is minimized across all input and output patterns used.

[0006] Neural networks are typically adaptive because they modify themselves during training as each new dataset is processed. Therefore, each new dataset received as input to the input layer allows the network to evaluate its accuracy and update its weights to learn how to handle that input. Consequently, systems and methods for selecting relevant datasets (e.g., authoritative, valid, high-quality, and / or similar datasets) to receive as input at the input layer are desired.

[0007] Generally, neural networks are designed to identify patterns in input data, classify data, cluster data, and / or make predictions based on data. Therefore, neural networks can be used to acquire information about unstructured text (e.g., text lacking metadata, text that is not easily mapped to database fields, text provided in natural language, and / or similar text). A neural network is initially trained using input data and known outputs. During training, connectors that contribute to the correct known outputs may receive more weights, while those that fail to contribute to the correct known outputs will have their weights reduced. After training, the weights are fixed, and the neural network can receive queries (e.g., natural language queries and / or similar queries) and return a response based on the data input into the neural network.

[0008] Traditionally, knowledge bases have been "edited" from the front end. For example, a person (such as an editor, subject or domain expert, trained specialist, and / or similar) would manually research materials on a topic of interest (such as books, scientific articles, journals, and / or similar materials), read what they found, and fill out and submit forms to incorporate any relevant information into the knowledge base. Unfortunately, this approach is time-consuming, relies on an individual's research capabilities (e.g., finding materials relevant to the topic of interest) and their ability to comprehensively evaluate (e.g., potentially hundreds of pages) each piece of material found (e.g., its relevance to the topic of interest), and then summarize all of this as changes or additions to the information in the knowledge base. Therefore, systems and / or methods should not only have more efficient front-end systems (e.g., for generating selective input datasets) but also back-end systems for curating neural network outputs to build and / or maintain a knowledge base associated with the neural network. Summary of the Invention

[0009] In one implementation, the system for maintaining a knowledge base includes a device having a processor and a memory storing program instructions that, when executed by the processor, cause the device to: generate a first interface configured to: receive queries for transmission to a question-answering system and provide a response to the queries from the question-answering system, wherein the response includes one or more suggested triples from a list of suggested triples. When executed by the processor, the program instructions further cause the device to: after selecting a specific triple from the list of suggested triples, generate a second interface configured to: provide at least one evidence record associated with the specific triple, wherein each evidence record includes a text span supporting the specific triple; and provide one or more control elements associated with each evidence record, wherein the one or more control elements include at least one of: a first control element selectable to reference its corresponding evidence record and text span to support the specific triple; or a second control element selectable to prevent its corresponding evidence record and text span from being referenced to support the specific triple. When executed by the processor, the program instructions further cause the device to: generate a data structure based on the selection of the one or more control elements, thereby updating the knowledge base.

[0010] In another embodiment, the system for maintaining the knowledge base includes a question-answering system having a neural network, a knowledge base, a suggestion triple database, a processor, and a memory. The memory stores program instructions that, when executed by the processor, cause the question-answering system to: read multiple texts received from an editing device as input to the neural network; store the neural network outputs corresponding to the multiple texts as suggestion triples in the suggestion triple database; respond to queries received from the editing device based on the suggestion triples stored in the suggestion triple database; and update the knowledge base based on one or more data structures received from the editing device.

[0011] In yet another implementation, the method of maintaining a knowledge base includes: inputting a query for transmission to a question-answering system via a first interface of an editing device; receiving a response to the query from the question-answering system via the first interface, wherein the response includes one or more suggested triples from the suggested triple list; receiving at least one evidence record associated with a specific triple selected from the suggested triple list via a second interface of the editing device, wherein each evidence record includes a text span supporting the specific triple; selecting a control element associated with each evidence record via the second interface, wherein the control element includes one of: a first control element for referencing the corresponding evidence record and text span to support the specific triple; or a second control element for preventing the corresponding evidence record and text span from being referenced to support the specific triple; and generating a data structure based on the selected control element to update the knowledge base.

[0012] These and other features provided by the embodiments described herein will be more fully understood in light of the following detailed description, taken in conjunction with the accompanying drawings. Attached Figure Description

[0013] The embodiments illustrated in the accompanying drawings are illustrative and exemplary in nature and are not intended to limit the subject matter as defined in the claims. The following detailed description of the illustrative embodiments will be understood when read in conjunction with the following drawings, in which similar structures are indicated by similar reference numerals, wherein: Figure 1 A block diagram of an illustrative system according to one or more embodiments shown or described herein is provided.

[0014] Figure 2 The embodiments shown or described herein are described. Figure 1 Descriptive interface for editing devices in the system;

[0015] Figure 3 The embodiments shown or described herein are described. Figure 1 Another descriptive interface for the editing equipment in the system;

[0016] Figure 4 The embodiments shown or described herein are described. Figure 1 Another descriptive interface for the editing device in the system;

[0017] Figure 5 The embodiments shown or described herein are described. Figure 4 A horizontal scrolling view of the descriptive interface;

[0018] Figure 6A flowchart is described illustrating an illustrative method for constructing a knowledge base using a neural network according to one or more embodiments shown or described herein; and

[0019] Figure 7 A flowchart is described, illustrating a method for using a knowledge base constructed and / or maintained using a neural network, according to one or more embodiments of this disclosure. Detailed Implementation

[0020] Various embodiments of this disclosure relate to computer-based systems and methods for generating and / or maintaining knowledge bases using neural networks. Depending on the specific implementation, the knowledge base can be generated and / or maintained by performing targeted queries on a neural network of input text. Maintaining the knowledge base using the systems and / or methods described herein can improve its coverage, timeliness, and accuracy.

[0021] The various implementation schemes described herein provide users (such as editors, subject or domain experts, trained experts and / or similar personnel) with systems and methods that not only evaluate existing assertions in a particular knowledge base, but also administratively control the growth of assertions in said particular knowledge base.

[0022] In one instance, when a knowledge base query does not yield an argument and / or the accuracy of existing arguments is questionable, the editing device described herein enables the user to efficiently find text corresponding to the query (e.g., relevant text, text with relatively high evidentiary value, and / or similar text) to input into the neural network. Furthermore, the editing device and the various interfaces described herein enable the user to effectively review each argument (e.g., corresponding to the input text) output by the neural network before it is added to the knowledge base. More specifically, the editing device and the various interfaces described herein enable the user to evaluate supporting evidence text associated with each argument and approve or disapprove the addition of each argument to the knowledge base.

[0023] In another instance, when the knowledge base query results are narrow and / or limited assertions, the editing device described herein enables users to expand existing assertions. More specifically, the editing device and various interfaces described herein enable users to query the knowledge base with targeted questions and / or enhanced questions to expand the assertions associated with the query.

[0024] This document describes various embodiments with reference to flowchart illustrations of methods, apparatus (systems), and computer program products. Each block of the flowchart illustration, and combinations of blocks in the flowchart illustration, can be implemented via executable computer program instructions. These computer program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a special-purpose machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create a system to implement the functions specified in the flowchart blocks and / or combinations of blocks.

[0025] Computer program instructions may also be stored in a non-transitory computer-readable storage medium that can instruct or cause a computer or other programmable data processing apparatus to operate in a particular manner. In these respects, computer program instructions stored in a non-transitory computer-readable storage medium can define a computer program product (e.g., an industrial article). When executed by a processor of a computer or other programmable data processing apparatus, the computer program instructions of the computer program product can implement the functions specified by the blocks in the flowchart description and / or various combinations of blocks in the flowchart description described herein.

[0026] Computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the computer program instructions, which execute on the computer or other programmable apparatus, provide steps to implement the functions specified in the blocks of the flowchart description described herein and / or various combinations of blocks in the flowchart description.

[0027] The various embodiments described herein may include a specially configured computer (e.g., a server) or a computer configured with the necessary hardware, software, and / or firmware. The computer may include a processor, input / output hardware, network interface hardware, data storage components, and memory components configured as volatile or non-volatile memory, including RAM (e.g., SRAM, DRAM, and / or other types of random access memory), flash memory, registers, optical disc (CD), digital versatile optical disc (DVD), and / or other types of storage components. Depending on the above, the memory components may also include operating logic that, when executed, facilitates the operations described herein. The processor may include any processing component configured to receive and execute instructions (such as from the data storage components and / or memory components). The network interface hardware may include any wired / wireless hardware known to those skilled in the art for communicating with other networks and / or devices.

[0028] Figure 1A block diagram of an illustrative system 100 according to one or more embodiments of the present disclosure is described. The system 100 may include a client device 102, a question-and-answer system 104, an editing device 110, and a text source device 124. Each of the client device 102, question-and-answer system 104, editing device 110, and text source device 124 may include a computer or other programmable data processing apparatus as described herein (e.g., a specially configured processor and / or similar apparatus for executing computer program instructions). Furthermore, each of the client device 102, question-and-answer system 104, editing device 110, and text source device 124 may be communicatively coupled via one or more networks, thereby enabling the communication described herein to occur via one or more networks. The one or more networks may include, but are not limited to, wide area networks (WANs) such as the Internet, local area networks (LANs) such as Ethernet, mobile communication networks, public service telephone networks (PSTNs), personal area networks (PANs), metropolitan area networks (MANs), virtual private networks (VPNs), and / or other networks. Therefore, each of the client device 102, the question-and-answer system 104, the editing device 110, and the text source device 124 can be located remotely or locally. In one instance, the question-and-answer system 104 and the editing device 110 can be located locally, while the client device 102 and the text source device 124 can be located remotely. In another instance, the question-and-answer system 104, the editing device 110, and the text source device 124 can be located locally, while the client device 102 can be located remotely. Although the editing device 110 is... Figure 1 While shown as a single device, it should be understood that multiple editing devices may each operate in a manner similar to editing device 110 described herein. For example, multiple editing devices may be configured as an editing workflow system, as described herein, to arrange work among multiple users (e.g., editors, reviewers, subject or domain experts, trained experts, and / or similar personnel) and / or their managers. Similarly, client device 102 may include multiple client devices, each operating in a manner similar to client device 102 described herein.

[0029] Given Figure 1The question-answering system 104 can be configured to receive a query 106 via an interface 103 of a client device 102. The question-answering system 104 can be configured to transmit a response 108 back to the client device 102 based on the current state of the knowledge base 120 for display on the interface 103. Based on each response 108, the client device 102 can be configured to generate a data structure 107 and transmit said data structure 107 to the question-answering system 104 to establish and / or update the knowledge base 120, as described herein. According to various aspects described herein, the question-answering system 104 may include a neural network 105. According to various aspects, the neural network 105 may include a memory network (e.g., a machine understanding network, a reading comprehension network, and / or a similar network). In some aspects, the neural network 105 and the knowledge base 120 may be separate components that can be selectively loaded within the question-answering system 104 (e.g., from respective files stored internally or externally to the question-answering system 104), thus enabling the system 100 to load, edit, and save multiple knowledge bases. In these respects, each knowledge base 120 may be loaded into a database management system (e.g., a question-answering system 104), which is configured to edit each knowledge base 120. According to some aspects, various neural networks 105, the selection of text from the text source device 124, and the knowledge base 120 may complement each other so that they are typically loaded together. The knowledge base 120 is a technique configured to store complex structured and unstructured information. According to various aspects described herein, the neural network 105 may be a Bidirectional Attention Flow (BiDAF) network. An example BiDAF network is described in the document entitled “Bidirectional Attention Flow for Machine Understanding” by Seo, Minjoon, et al., the entire contents of which are incorporated herein by reference.

[0030] Refer again Figure 1Editing device 110 can be configured to read multiple texts 112 as input to the neural network 105 of question-answering system 104. Depending on some aspects, the multiple texts 112 may include multiple pre-selected texts (e.g., the full text of a book, a specific chapter of a book, the full text of a journal, a specific paragraph of a journal, and / or similar content). Here, reading the pre-selected texts allows neural network 105 to focus on relevant material, and including such relevant material reduces the likelihood of any false positives. In these aspects, material may be included in or excluded from the pre-selected texts according to predefined selection criteria applied by editing device 110. Instance selection criteria may include selecting texts that are recognized as authoritative in a particular field, texts verified as peer-reviewed, and / or similar texts. This approach may be advantageous if the size of neural network 105 is limited. However, further in these aspects, editing device 110 may apply predefined data cleaning techniques to the multiple texts 112 before reading them into neural network 105. One instance of data cleaning technique may include accessing a dictionary to correct inaccurate or incomplete data. According to other aspects, editing device 110 may apply predefined data cleaning techniques to the multiple texts 112 after reading them into neural network 105. As an example, editing device 110 may evaluate the frequency of one or more received responses to assess the presence of noise in the responses based on the multiple texts 112. As another example, editing device 110 may apply at least one verification rule to one or more received responses to verify that the correct response type is received based on the multiple texts 112 (e.g., for the subject, predicate, and / or object of a triple, as described herein). According to various embodiments, the multiple texts 112, or at least a portion thereof, may be stored in a non-textual manner in knowledge base 120. According to various aspects, the output of question-answering system 104, generated from reading the multiple texts 112 into neural network 105 and receiving queries 106, is a plurality of suggested triples, as described more fully herein.

[0031] Depending on other aspects, the multiple texts 112 may not include pre-selected texts. In these aspects, the text source device 124 may include a database 126 storing text bodies 128 (e.g., full-text books related to health, authoritative full-text journals related to health, and / or similar content). In some aspects, the text bodies 128 may be massive (e.g., searchable via a state-of-the-art full-text search engine). In some aspects, the body of text 128, including each full text and / or its respective sections (e.g., chapters, paragraphs, and / or similar content), may be indexed. In some aspects, the full-text search engine 135 may implement search engine indexing techniques (e.g., including BM-25 relevance functions, TD-IDF, and / or similar relevance mechanisms) for the editing device 110 to retrieve multiple texts 112 and read them as input to the neural network 105 of the question-answering system 104. For example, editing device 110 may be configured to send a text query 130 (e.g., find a “X” segment related to “endocrine system”) to a full-text search engine 135 within text source device 124 and receive a text response 132 (e.g., including a “X” segment associated with “endocrine system”) from text source device 124. As another example, editing device 110 may be configured to send a text query 130 to text source device 124 and receive a text response 132 from text source device 124 including a predetermined number of results (e.g., the first 100 results). In such an aspect, editing device 110 may read the text of text response 132 as input to neural network 105 as multiple texts 112. Depending on various aspects, the output obtained by reading the text of text response 132 into neural network 105 is natural language text that provides evidence for multiple suggestion triples, as described more fully herein.

[0032] Given Figure 1The editing device 110 may be further configured to transmit multiple queries 114 to the question-answering system 104. Depending on various aspects, the multiple queries 114 may include targeted queries and / or enhanced queries, as described more fully herein. Each of the multiple queries 114 may be in plain language form (e.g., without a special query language syntax). The editing device 110 may be further configured to receive multiple responses 116 from the question-answering system 104 (e.g., in response to each query). More specifically, each of the multiple responses 116 may include a text extract (e.g., a text response) from text read as input to the neural network 105 (e.g., text from multiple texts 112), as described herein. For example, each response may include an extract from indexed, highly relevant text (e.g., a text response). Similar examples can be found in a document entitled “End-to-End Open Domain Question Answering with BERTserini” written to Yang, Wei, et al., the entire contents of which are incorporated herein by reference. According to other aspects, each of the plurality of responses 116 may include new text generated by the question-answering system 104 based on text read as input to the neural network 105 (e.g., text of the plurality of texts 112). Based on each of the plurality of responses 116, the editing device 110 may be further configured to generate a data structure 118 and transmit the data structure 118 to the question-answering system 104 to establish and / or update the knowledge base 120, as described herein.

[0033] According to various embodiments of this disclosure, the editing device 110 may include a plurality of interfaces 122 configured to transmit a plurality of queries 114 to and receive a plurality of responses 116 from the question-and-answer system 104. Depending on various aspects, the data structure 118 may be generated based on control element inputs received via the plurality of interfaces 122.

[0034] Figure 2 An illustrative interface 122A of an editing device 110 according to one or more embodiments of this disclosure is described. (See also...) Figure 2 Interface 122A can be configured to accept queries in the form of triples (e.g., resource description framework triples). Each triple typically consists of three parts: a subject, a verb, and an object (e.g., described here using the notation: subject-verb-object). For example, a triple could include "diabetes -- is a disorder of the endocrine system." The subject "diabetes" represents one resource, the object "endocrine system" represents another resource, and the verb "is a disease" represents the relationship between the subject and the object. Other instances of verb forms may include, but are not limited to, "is a treatment for," "is a cause of," "is associated with," "is an alternative treatment for," and / or similar statements, and / or variations thereof. Figure 2In this example, the subject is generally referred to as Entity 1 (202), the predicate as relation (204), and the object as Entity 2 (206). In this case, Entity 1 (202) is associated with the first Entity 1 text box (208), the second Entity 1 text box (210), and the Entity 1 dropdown menu (212). Similarly, relation (204) is associated with the first relation text box (214), the second relation text box (216), and the relation dropdown menu (218), and Entity 2 (206) is associated with the first Entity 2 text box (220), the second Entity 2 text box (222), and the Entity 2 dropdown menu (224).

[0035] refer to Figure 2 According to various aspects of this disclosure, a user (e.g., an editor, reviewer, subject or domain expert, trained expert, and / or similar person) can search for an entity 1 202 of interest (e.g., subject) by entering a human-readable name (e.g., “glaucoma”) in the second entity 1 text box 210. Similarly, a user can search for a relation 204 of interest (e.g., predicate) by entering a human-readable name in the second relation text box 216, and a user can search for an entity 2 206 of interest (e.g., object) by entering a human-readable name (e.g., “blindness”) in the second entity 2 text box 222. Here, depending on different aspects, the subject, predicate, and / or object of interest can be based on triples already existing in the knowledge base 120 (e.g., glaucoma -- progresses to -- blindness). In these aspects, the interface 122A of the editing device 110 can be further configured to probe the knowledge base 120 (e.g., via communication link 115) to verify or expand one or more existing triples.

[0036] Refer again Figure 2According to various aspects of this disclosure, each part of each triple can be associated with a unique, machine-readable identifier (e.g., subject identifier (SID), predicate identifier (PID), object identifier (OID)). In this case, a user can search for entity 1 of interest 202 by entering a machine-readable identifier (e.g., SID "2791370") in the first entity 1 text box 208 (e.g., if the user knows it). Similarly, a user can search for relation 204 of interest by entering a machine-readable identifier (e.g., if the user knows it) in the first relation text box 214, and a user can search for entity 2 of interest 206 by entering a machine-readable identifier (e.g., OID "2790966") in the first entity 2 text box 220 (e.g., if the user knows it). Depending on the aspects, interface 122A can be configured to automatically populate its corresponding machine-readable identifier (e.g., in the first entity 1 text box 208, the first relationship text box 214, and / or the first entity 2 text box 220) after and / or during the input of a human-readable identifier (e.g., in the first entity 1 text box 208, the first relationship text box 214, and / or the first entity 2 text box 220), and to automatically populate its corresponding human-readable name (e.g., in the second entity 1 text box 210, the second relationship text box 216, and / or the second entity 2 text box 222) after and / or during the input of a machine-readable identifier (e.g., in the first entity 1 text box 208, the first relationship text box 214, and / or the first entity 2 text box 220). Interface 122A can be configured to automatically populate human-readable names or machine-readable identifiers by transmitting (e.g., via communication link 115) an input human-readable name or an input machine-readable identifier to question-answering system 104 in real-time or near real-time, and receiving (e.g., via communication link 115) a corresponding machine-readable identifier or human-readable name from question-answering system 104 in real-time or near real-time. In these respects, question-answering system 104 can be configured to access lookup table 133, as described more fully herein.

[0037] According to a further aspect of this disclosure, if a user wishes not to specify or define a text box, the user can enter a wildcard in the text box of interface 122A. Given... Figure 2For example, a user can search for relation 204 by entering a wildcard (e.g., "*") in the first relation text box 214. In some aspects, interface 122A can be configured to accept wildcards in the first entity 1 text box 208, the first relation text box 214, and / or the first entity 2 text box 220. In other aspects, interface 122A can be configured to accept wildcards in the second entity 1 text box 210, the second relation text box 216, and / or the second entity 2 text box 222. In a further aspect, interface 122A can be configured to accept wildcards in any one of the first entity 1 text box 208, the second entity 1 text box 210, the first relation text box 214, the second relation text box 216, the first entity 2 text box 220, and / or the second entity 2 text box 222. Depending on other aspects, interface 122A can be configured to accept blank text boxes in addition to and / or instead of wildcards, as a way to make the text box unspecified or undefined.

[0038] According to a further aspect of this disclosure, interface 122A can be configured to receive selections of semantic groups via entity 1 dropdown menu 212 and / or entity 2 dropdown menu 224. Depending on various aspects, semantic groups can restrict wildcard search results in entity 1 textbox 208 to only those concepts in knowledge base 120 that belong to members of said semantic group. For example, selecting the semantic group "disease" will exclude any non-disease concepts from consideration. Similarly, interface 122A can be configured to receive selections of knowledge base relations via relation dropdown menu 218. Given... Figure 2 Continuing with the example, Elsevier's Merged Medical Classification (EMMeT) is a medical classification knowledge base that can be loaded as knowledge base 120 for continuous revision. Therefore, the relation dropdown menu 218 can be configured to display to the user each relation 204 (e.g., predicate) currently defined in knowledge base 120 (e.g., EMMET), from which the user can select. This aspect leads the user to select the relation 204 suitable for knowledge base 120. However, furthermore, given... Figure 2 Interface 122A can be configured to accept user contact information (e.g., email address and / or similar information) via user contact text box 234.

[0039] Still refer to Figure 2After selecting search control element 226 (e.g., icon, button, and / or similar), interface 122A can be configured to transmit query 114 to question-answering system 104 to probe neural network 105 for textual evidence of suggested triples corresponding to query 114. Here, neural network 105 may output multiple text fragments as evidence supporting or opposing that query 114 is a triple that should be added to knowledge base 120, in response to multiple texts 112 being read as input into neural network 105. Question-answering system 104 may also query knowledge base 120 to understand whether one or more of the multiple suggested triples have already been established as facts in the knowledge base. Depending on various aspects, the multiple suggested triples may be stored in suggested triple database 134 of question-answering system 104 until evaluated by the user. In these respects, after selecting search control element 226, interface 122A can be configured to present one or more suggested triples (e.g., stored in suggested triple database 134) in a suggested triple list 228 of a portion of interface 122A (e.g., the bottom portion). Given Figure 2 The list of suggested triples 228 used for user evaluation may be relatively large (e.g., 1925 triples). One or more suggested triples may be received as a response 116 to query 114, wherein one or more suggested triples correspond to the query (e.g., wildcards and / or blanks filled in for query 114). See again Figure 2 The list of suggested triples 228 may include a predetermined number of triples out of the total number of triples (e.g., 100 triples out of 1925 triples), which the user can scroll through using scroll bar 230. Depending on various aspects, the list of suggested triples 228 can be prioritized to a predetermined order, as described herein. This prioritization effectively separates valid and / or correct information in response to query 114 from potentially invalid and / or erroneous information in response to query 114 (e.g., due to the nature of automated systems). Such prioritization improves the efficiency of users (e.g., editors, reviewers, subject or domain experts, trained experts, and / or similar personnel).

[0040] In one respect, it is suggested that the list of triples 228 can be sorted by date (e.g., in relation to the base text source, the time the triple was generated, and / or similar content). This predetermined order (e.g., from latest to earliest, from earliest to latest, and / or similar order) allows users to assess any changes and / or shifts in perspective over time (e.g., whether the understanding of a “fact” or “argument” has changed over time).

[0041] In another respect, it is suggested that the list of triples 228 be ordered according to the source of the underlying text. For example, triples associated with text sources that have been copyrighted could take precedence over triples associated with text sources that have not yet been copyrighted.

[0042] In another respect, it is suggested that the list of triples 228 be ordered according to the strength of evidence from the underlying text sources (e.g., considering evidence pyramids, evidence-based medical source pyramids, and / or similar content). For example, triples associated with text sources of critical evaluation could be prioritized over triples associated with text sources of expert opinion.

[0043] In another respect, it is suggested that the list of triples 228 be sorted according to the domain of the underlying text source. For example, if the query is related to the endocrine system, triples associated with text sources limited to the field of endocrinology can be prioritized over triples associated with text sources not limited to the field of endocrinology.

[0044] In another aspect, the list of suggested triples 228 may be ordered based on predefined features in the underlying text source. Here, predefined features may include one or more features of interest (e.g., subject, predicate, object, complex factors, and / or similar factors) defined by a user (e.g., editor, reviewer, subject or domain expert, trained expert, and / or similar person). For example, if “gluten allergy,” “endocrine disorder,” and “Italian youth” are user-defined features, then triples associated with text sources involving such features of interest (e.g., endocrine disorders involving Italian youth who also suffer from gluten allergy) may be preferred over triples associated with text sources that do not involve such features of interest. In this case, depending on various aspects, one or more user-defined features of interest may be further used as pre-filters to limit the number of texts 112 input to neural network 105 (e.g., to be searched with query 114). On the other hand, continuing the ordering based on predefined features, the list of suggested triples 228 may be further ordered based on the relative frequency of specific terms or groups of terms within the underlying text source. For example, if "endocrine disorder" and "Italian youth" are user-defined features, a triple associated with a first text source (e.g., endocrine disorder in Italian youth) (where the first text source contains a first specific term or a first group of terms) (e.g., "diabetes" as a disease) may take precedence over a triple associated with a second text source that relates to this feature of interest, where the second text source contains a second specific term or a second group of terms with a relatively lower frequency (e.g., "osteoporosis" as a disease). Here, the first specific term or the first group of terms (e.g., diabetes) may have a higher priority for knowledge base 120 in the context of a youth population than the second specific term or the second group of terms (e.g., osteoporosis).

[0045] Still referencing Figure 2It is suggested that each triple in the list of triples 228 can be associated with a corresponding unique, machine-readable triplet identifier (TID) (e.g., TID 232). Continuing with the example, while each triple in the list of triples 228 is associated with SID "2791370" corresponding to "glaucoma" and OID "2790966" corresponding to "blindness" (e.g., as entered in query 114), each respective triple may be associated with a different TID and a different PID. For example, TID 2801026 is associated with PID 8622, which corresponds to "progress to," TID 2800003 is associated with PID 4856, which corresponds to "treated with," TID 2800625 is associated with PID 1642, which corresponds to "caused," and so on. In these respects, see [reference to...]. Figure 2 Each TID (e.g., TID 232) in the list of suggested triples 228 can be configured as a hyperlink (e.g., an evidence hyperlink) for a user to optionally view the respective textual evidence supporting each of the suggested triples 228. Based on these aspects, the list of suggested triples 228 provides visual material for users (e.g., editors, reviewers, subject or domain experts, trained experts, and / or similar individuals) to evaluate multiple texts 112 read as input to neural network 105. For example, a relatively large number of suggested triples may indicate that multiple texts 112 are relevant to query 114, while a relatively small number of suggested triples may indicate that multiple texts 112 are irrelevant to query 114, and more text (e.g., text corresponding to query 114) may be needed as input to neural network 105. Depending on some aspects, the list of suggested triples 228 may include unsuggested triples. In this respect, unsuggested triples may indicate that more text (e.g., corresponding to query 114) may need to be read as input into neural network 105 so that at least one triple and / or its corresponding supporting textual evidence can be identified. Such a focused search for more text can be handled via text query 130 and text response 132 to text source device 124, as described herein. Furthermore, given interface 122A, a user can quickly determine (via inspection) whether one or more suggested triples require further investigation (e.g., the suggested triples appear to be incorrect), a user can verify the one or more suggested triples, a user can determine whether the list of suggested triples 228 corresponding to their query 114 needs to be expanded (e.g., more suggested triples corresponding to query 114 are needed), and / or similar matters.

[0046] Figure 3Another illustrative interface 122B of an editing device 110 according to one or more embodiments of this disclosure is described. Specifically, the editing device 110 may present interface 122B (e.g., after the user selects a specific TID from a list of suggested triples 228 (e.g., via their respective hyperlinks)). Figure 2 Depending on various aspects, Interface 122B can confirm that a text evidence request associated with a specific TID (e.g., TID 2800625:2791370 (glaucoma)-1642 (causing)-2790966 (blindness)) has been submitted. Depending on some aspects, Interface 122B can provide a user with Instruction 302, indicating that the user will receive an email containing a Uniform Resource Locator (URL) for indirect access to the text evidence results associated with the selected TID. Depending on further aspects, Interface 122B can provide a URL 304 for direct access to the text evidence results associated with the selected TID.

[0047] Figure 4 Another illustrative interface 122C of an editing device 110 according to one or more embodiments of the present disclosure is described. In particular, the editing device 110 can be configured after the user selects a URL provided via email at the entered address ( Figure 1 (e.g., via the user contact text box 234) and / or after the user selects a URL provided via interface 122b (e.g., Figure 3 The URL 304) renders the interface 122c. Depending on other aspects, the editing device 110 can display the user's input from a list of suggested triples 228 (e.g., ...). Figure 2 After selecting a specific TID (e.g., via its respective evidence hyperlink), interface 122C is rendered directly without rendering interface 122B.

[0048] Reference Figure 4 Continuing with the example, the selected triple (e.g., TID 2800625: 2791370 (glaucoma) - 1642 (cause) - 2790966 (blindness)) is associated with the first text evidence record 402 and the second text evidence record 404. According to various aspects of this disclosure, each text evidence record may include a record identifier 406, an inferred relation 408, a known relation 410, and a text span 412 (e.g., sentence W of paragraph X, paragraph Y of paragraph Z, and / or similar content). According to the aspects described herein, each text span 412 has been extracted from the text of the plurality of texts 112 and read as input into the neural network 105. Given... Figure 4Interface 122C can be configured to visually distinguish each part of a selected triple within each text span 412. For example, in the first text span 412A associated with the first text evidence record 402, the subject “glaucoma” is distinguished from the surrounding text by a first dashed box of a first color (e.g., red), the predicate “cause” is distinguished from the surrounding text by a second dashed box of a second color (e.g., blue), and the object “blindness” is distinguished from the surrounding text by a third dashed box of a third color (e.g., green). Similarly, in the second text span 412B associated with the second text evidence record 404, where the subject “glaucoma” is distinguished from the surrounding text by a first dashed box of a first color (e.g., red), neural network 105 has inferred that “…risk” provides evidence for the predicate “cause” in the context of the rest of the text. Therefore, the phrase "...risk" is distinguished from the surrounding text by a second dashed box of a second color (such as blue), while the object "blindness" is distinguished from the surrounding text by a third dashed box of a third color (such as green). Other methods can also be used to distinguish each part of the selected triples (e.g., highlighting). Thus, interface 122C allows the user to quickly and efficiently locate the respective components of each triple being evaluated. Furthermore, interface 122c allows the user to view each triple being evaluated in the text context that is the input to neural network 105 (e.g., sentences including each part of the selected triples, sentences before and after the sentence including the selected triples, paragraphs including each part of the selected triples, and paragraphs before and after the paragraph including the selected triples, and / or similar content). Given... Figure 4 Knowledge base 120 has assigned a first text variant of the first text span 412A (e.g., “cause”) to the queried predicate (e.g., 1642(cause)) and a second text variant of the second text span 412B (e.g., “risk of…”) to the queried predicate (e.g., 1642(cause)).

[0049] Figure 5 The present disclosure describes one or more embodiments of the present disclosure. Figure 4 The illustrative interface 122C provides a scrollable view (e.g., via scrollbar 414). Given... Figure 5 Each of the first text evidence record 402 and the second text evidence record 404 may further include a source identifier 502 and a user operation interface 504. Specifically, the source identifier 502 may reference a source (e.g., a book, guideline, clinical trial report, journal article, and / or similar content) associated with its corresponding text span (e.g., text span 412A). Furthermore, the user operation interface 504 of interface 122C may be configured to include one or more control elements (e.g., checkboxes, buttons, and / or the like) to perform a specified operation. Figure 5 As shown, the user interface 504 for each text evidence record can be configured to include a "Citing Evidence" control element 506, a "Suppressing Evidence" control element 508, and / or a "Add New Triplet" control element 510. In these respects, for example, if a user (e.g., an editor, reviewer, subject or domain expert, trained expert, and / or similar person) determines that the first text span 412A supports the selected triplet (e.g., TID 2800625:2791370 (glaucoma)-1642 (cause)-2790966 (blindness)), the user can select the "Citing Evidence" control element 506. Additionally, if the user determines that the first text span 412A is irrelevant to the selected triplet, that all or part of the first text span 412 is erroneous, that the first text span 412 is duplicated, and / or similar, the user can select the "Suppressing Evidence" control element 508. Furthermore, if the user determines that the first text span 412A supports one or more additional triples, the user can select the "Add New Triple" control element 510. For example, given... Figure 4 The first text span 412A further supports “Cataracts – a leading cause of blindness,” “Glaucoma – a leading cause of blindness,” “Age-related macular degeneration – a leading cause of blindness,” and / or similar content. In these respects, after selecting the “Add New Triplet” control element 510, interface 122C can be configured to present an interface (e.g., similar to the upper part of interface 122A, not shown) for the user to manually input one or more further supported triplets.

[0050] Depending on various aspects, selecting the "Suppress Evidence" control element 508 can prevent the specific associated text span (e.g., the first text span 412A) from being added to knowledge base 120 to support the selected triple. In this respect, selecting the "Suppress Evidence" control element 508 can generate a span rejection record, associating the specific text span with the selected triple. The span rejection record can be added to the span rejection portion of data structure 118, which can be used to prevent the specific associated text span from being added to knowledge base 120 to support the selected triple. Depending on other aspects, selecting the "Suppress Evidence" control element 508 can prevent all text spans 412 associated with the source text (e.g., "glaucoma," Medical Diagnosis and Treatment, 2015, Vol. 1, 2nd Edition, Khouri, Albert S. & Fechtner, Robert D.) from being added to knowledge base 120 to support the selected triple. In this respect, selecting the "Suppress Evidence" control element 508 can generate a source rejection record that associates source text with the selected triple. The source rejection record can be added to the source rejection portion of data structure 118, which can be used to prevent any text span associated with the source text from being added to knowledge base 120 to support the selected triple. According to other aspects, selecting the "Suppress Evidence" control element 508 can generate a partially completed record that associates the specific text span (e.g., first text span 412A) with the selected triple. In these aspects, the partially completed record can be added to data structure 118 for transmission to question-answering system 104 for addition to the suggested triple database 134 (e.g., for subsequent evaluation via editing device 110). In some aspects, the partially completed record may include a description from a user (e.g., the user who selected the "Suppress Evidence" control element 508) describing one or more questions that led to the selection of the "Suppress Evidence" control element 508. Depending on the circumstances, any record generated after selecting the “Suppress Evidence” control element 508 (e.g., span rejection record, source rejection record, partially completed record, and / or similar record) may be transferred to another user (e.g., another editor, another reviewer, another subject or domain expert, another trained expert, and / or similar person) and / or supervising user for re-evaluation (e.g., in the workflow).

[0051] Furthermore, given Figure 5Interface 122C may include an "Add to KB" control element 512 and a "Cancel" control element 514 (e.g., an icon, button, and / or the like). In these respects, interface 122C may be configured to, upon selection of the "Add to KB" control element 512, generate a record that associates the selected triple (e.g., TID 2800625: 2791370 (glaucoma) - 1642 (cause) - 2790966 (blindness)) with each text evidence record (e.g., first text evidence record 402, second text evidence record 404, and / or the like), for which the "Cited Evidence" control element 506 has been selected. Depending on various aspects, the generated record may be added to data structure 118 for transmission to question-answering system 104 for addition to knowledge base 120 (e.g., the EMMET Medical Taxonomy Knowledge Base and / or a similar knowledge base). Then, each text evidence record (e.g., text span and related source) selected in the “Cited Evidence” control element 506 can be tracked in the knowledge base 120 as the source of the added triple.

[0052] Interface 122C can be further configured to generate records after selecting the "Add to KB" control element 512, which associate each added triple with its associated text evidence record for which the "Add New Triple" control element 510 has been selected. Depending on various aspects, the generated records can be similarly added to data structure 118 for transmission to question-answering system 104 for addition to knowledge base 120. In these respects, each text evidence record (e.g., text span and related source) for which the "Add New Triple" control element 510 has been selected can then be tracked in knowledge base 120 as the source of the added triple. According to another aspect, the generated records (e.g., associating each added triple with its associated textual evidence record, for which the "Add New Triple" control element 510 has been selected) can be added to data structure 118 for transmission to question-answering system 104, thereby adding them to the suggested triple database 134 (e.g., for subsequent evaluation when a query concerning the triple is made via editing device 110). In these aspects, question-answering system 104 can be further configured to, upon receiving data structure 118, define a TID for each new triple and associate each defined TID with the SID, PID, and OID corresponding to each corresponding human-readable name via lookup table 133. Furthermore, in these respects, if a portion of the new triple (e.g., subject, predicate, or object) is not already defined in lookup table 133, the question-and-answer system 104 can be further configured to define unique, machine-readable identifiers (e.g., SID, PID, OID, respectively) and store the defined machine-readable identifiers in lookup table 133 in association with their human-readable names (e.g., for future use). Depending on the aspects, after selecting “Add to KB” control element 512, the editing device 110 can be configured to present interface 122A to the user again (e.g., select another suggested triple from the list of suggested triples 228 for evaluation).

[0053] Furthermore, given Figure 5 Interface 122C can be configured to exit interface 122C and invalidate any selected "Cited Evidence" and / or "Suppress Evidence" controls (e.g., without making any changes) after selecting "Cancel" control element 514, depending on the aspects. After selecting "Cancel" control element 514, editing device 110 can be configured to present interface 122A to the user again (e.g., to select another suggested triplet from the list of suggested triplets 228 for evaluation).

[0054] According to the description in this article Figure 1-5The embodiments of this disclosure implement a semi-automated editing system and / or method that dynamically interacts with a user to assist the user in evaluating suggested triples for addition to knowledge base 120. Therefore, the interface 122 of the editing device 110, as described herein, can be used to continuously maintain and / or expand the triples of knowledge base 120.

[0055] Brief Reference Figure 1 The knowledge base 120 of the neural network 105 can represent each relationship between its data as a triple (e.g., as a TID associated with SID, PID, and OID). This relationship can be represented and / or processed in a machine-readable form. In these respects, each machine-readable identifier (e.g., SID, PID, OID) can be associated with a human-readable name in the lookup table 133 of the question-answering system 104 (e.g., SID corresponding to “glaucoma,” PID corresponding to “cause,” and OID corresponding to “blindness”). Accordingly, the question-answering system 104 can be configured to access the lookup table 133 to translate received queries (e.g., query 114 from editing device 110, query 106 from client device 102) and translate transmitted responses (e.g., response 116 to editing device 110, response 108 to client device 102). Similarly, the question-answering system 104 can be configured to access lookup table 133 to translate the received data structure 118, as described herein, before adding it to the knowledge base 120 and / or the suggestion triple database 134.

[0056] According to the aspects described herein, each triple in knowledge base 120 can be considered a "fact" or an "argument". In some aspects, various triples can be combined within knowledge base 120 into disjoint triples, referred to herein as graphs (e.g., H-graphs). Each graph can also be associated with a unique, machine-readable graph identifier (GID). Thus, each GID can be associated with one or more TIDs in query table 133 of question-answering system 104.

[0057] Figure 6A flowchart is described for an illustrative method 600 for constructing a knowledge base 120 using a neural network 105 according to one or more embodiments of this disclosure. In block 602, a topic (e.g., a health-related topic) can be selected. In block 604, text associated with the selected topic (e.g., pre-selected or not pre-selected) can be read as input into the neural network 105 (e.g., via text source device 124 and editing device 110, as described herein). Depending on various aspects, the text associated with the selected topic can be read as input into the neural network 105 in a first order. In block 606, the neural network 105 can be probed with at least one orientation question (e.g., via an interface 103 of a client device 102 configured to accept queries 106, e.g., via a text box, in plain text or natural language, without any specific query language). In some aspects, interface 122A can be similarly configured to include a text box to accept queries 114 (e.g., orientation questions) in plain text or natural language. In a further aspect, interface 122A can be configured to generate orientation questions. Depending on various aspects, the orientation question can be constructed to extend the assertions in knowledge base 120 (e.g., new or existing assertions). Further, the orientation question can be constructed to obtain confirmation or refutation of assertions in knowledge base 120 (e.g., new or existing assertions). For example, an orientation question can be constructed based on an assertion already existing in knowledge base 120 (e.g., a triple) to find evidence of said assertion in the text. On one hand, the assertion in knowledge base 120 can be transformed into a true or false question. On the other hand, one or more parts of the assertion in knowledge base 120 can be edited (e.g., the subject, predicate, and / or object of the triple), and a question (e.g., a fill-in-the-blank question) can be constructed from the remaining parts of the assertion. Such editing can generate a large number of potential questions. For example, an existing knowledge base triple (e.g., diabetes – a disease of the endocrine system) can be rewritten as: (____ – a disorder of the endocrine system). In this instance, the constructed orientation question could be a natural language question: "What is a disorder of the endocrine system?". Accordingly, this aspect can be another way of presenting wildcard or blank queries. Based on the various aspects described herein, at least one orientation question may include a series of orientation questions. For example, the initial orientation question in a series of orientation questions may include "What are the treatment options for coronary artery disease?" (derived from the triple ____-- is the treatment for-- coronary artery disease). Subsequent orientation questions in a series of orientation questions may include variations of the initial orientation question. For example, subsequent orientation questions may include: "What are the first-line treatment options for coronary artery disease?", "What are the recommended treatments for coronary artery disease?", "Which intervention is recommended for coronary artery disease?", and / or similar questions.Depending on the specifics, subsequent orientation questions in a series of orientation questions may include variations of such questions using alternative names and / or synonyms (e.g., ischemic heart disease, atherosclerotic heart disease, atherosclerotic vascular disease, coronary heart disease, and / or similar diseases).

[0058] Depending on the aspects, alternative names or synonyms derived, received, or extracted from knowledge base 120 may be used. Continuing with the example of the endocrine system, a follow-up question to the orientation question “What is an endocrine system disorder?” could include “What is a hormone-regulated disease?”. In other aspects, partial statements in knowledge base 120 (e.g., diabetes is an endocrine system disorder) can be replaced with broader or more general terms or narrower or more specific terms derived, received, or extracted from knowledge base 120. For example, a follow-up question to the orientation question “What is an endocrine system disorder?” could include “What disease is an endocrine system disorder?”. In a further aspect, a combination of alternative names or synonyms and broader or narrower or more specific terms may be used. For example, a follow-up question to the orientation question “What is an endocrine system disorder?” could include “What disease is a hormone-regulated disease?”. Depending on the aspects, in block 606, neural network 105 can probe more than once for any particular orientation question, as described herein. For example, the same orientation question can be used to probe the neural network 105 more than once to detect further responses associated with the selected topic, as described herein. Here, while the methods described pertain to the interpretation of endocrine system disorders and / or coronary artery diseases, it should be understood that the methods are equally applicable to other diseases and / or other query topics.

[0059] In block 608, it can be determined whether a response has been received for each orientation question (e.g., response 108 via interface 103 of client device 102). If one or more orientation questions have not received a response, method 600 can return to block 604 (e.g., shown as optional in the phantom) to read more, more focused text (e.g., corresponding to the orientation question) into neural network 105. It should be noted that if no response is received for an orientation question, it is possible that the question has no answer. The method can suggest additional decision points, such as "Try more text again?" or "Exit". If at least one response is received for each orientation question, method 600 can proceed to block 610.

[0060] In block 610, responses from question-answering system 104 to each directed question can be compiled (e.g., via interface 103 of client device 102). As described herein, each response may include triples and / or a text span corresponding to the triples. Depending on certain aspects, responses to directed questions (e.g., directed questions constructed based on assertions present in knowledge base 120) may confirm or refute assertions present in knowledge base 120. For example, a response to “What is an endocrine disorder?” might include triples and / or text spans that not only represent “diabetes” (e.g., confirming an existing knowledge base assertion), but also “type 1 diabetes,” “type 2 diabetes,” “osteoporosis,” “thyroid cancer,” “adrenal insufficiency,” “Addison’s disease,” “Cushing’s disease,” “Cushing’s syndrome,” “Graves’ disease,” “acromegaly,” “hyperthyroidism,” “hypothyroidism,” “Hashimoto’s thyroiditis,” “hypopituitarism,” “multiple endocrine neoplasia I,” “multiple endocrine neoplasia II,” “polycystic ovary syndrome,” “precocious puberty,” and / or similar diseases (e.g., suggesting new assertions to expand knowledge base 120 assertions). Therefore, the compiled response can represent possible and / or alternative knowledge base answers regarding endocrine disorders. According to certain aspects, in cases where the question-answering system 104 is probed multiple times for the same oriented question, subsequent responses can exclude previously provided responses so that the user can further probe for possible and / or alternative responses. For example, a first set of responses may be received in response to a first query using the targeted question; a second set of responses may be received (e.g., excluding the first set of responses) in response to a second query using the same targeted question; a third set of responses may be received (e.g., excluding the first and second sets of responses) in response to a third query using the same targeted question, and so on. According to other aspects, if the question-answering system 104 has been probed multiple times with the same targeted question, subsequent responses may provide previously provided responses at the end of the response list, rather than excluding previously provided responses from subsequent responses, so that the user can probe for more possible and / or alternative responses sequentially before the previously provided responses. For example, a first set of responses may be received in response to a first query using the targeted question; a second set of responses may be received in response to a second query using the same targeted question (e.g., appending the first set of responses to the end of the second set of responses); a third set of responses may be received in response to a third query using the same targeted question (e.g., appending the first and second sets of responses to the end of the third set of responses), and so on. Here, re-evaluation of previously provided responses can confirm that there are no more possible and / or alternative responses to the same targeted question in the text read by the question-answering system 104.

[0061] Depending on the aspects, in block 608, a series of directed questions allows for the measurement of confidence (e.g., correctness) in a particular response (e.g., response consistency) of a compiled response. Confidence can be based on sources associated with a particular response (e.g., the strength of the sources described in the evidence pyramid). In one instance, confidence can be based on the source of the particular response—e.g., the strength of the source in the evidence pyramid, where a systematic review is considered higher quality evidence than textbooks. In another instance, as described herein, confidence can be based on the frequency of identical or similar responses given variations in the questions and / or submissions of the same question. In yet another instance, confidence can be based on the number of users who agree on a response (e.g., editors, reviewers, subject or domain experts, trained experts, and / or similar individuals) (i.e., the editing device provides a voting mechanism to evaluate the agreement of multiple users). Thus, the methods and systems of this disclosure can increase their users to ensure the supervised and / or controlled construction of knowledge base 120.

[0062] In block 612, the question-answering system 104 can be probed with at least one enhanced question (e.g., via interface 103 of client device 102 configured to accept plain text queries). In some aspects, interface 122A can be similarly configured to include a text box to accept queries 114 in plain text or natural language form (e.g., enhanced questions). In many aspects, interface 122A can be configured to generate enhanced questions. In some aspects, the enhanced questions recognize that assertions (e.g., triples) may not be absolute. Continuing with the example of coronary artery disease, the first assertion (e.g., lifestyle changes and medication—are methods of treating—coronary artery disease), the second assertion (e.g., percutaneous coronary angioplasty—is a method of treating—coronary artery disease), and the third assertion (e.g., coronary artery bypass grafting—is a method of treating—coronary artery disease) may all be user-verifiable treatments for coronary artery disease. However, one treatment method (e.g., lifestyle modifications and medication) may be more preferred than another (e.g., percutaneous coronary angioplasty and / or coronary artery bypass grafting) (e.g., due to established care guidelines from healthcare providers). Furthermore, one treatment method may be superior to another for one group (e.g., a group with a particular characteristic), but not for another group (e.g., another group with a different particular characteristic). Additionally, one treatment method may be available (e.g., due to regulatory approval in one country), while another may be unavailable (e.g., due to regulatory non-approval in one country). In such cases, each enhanced question can be constructed using an enhanced question template. An instance of an enhanced question template may take a specific response to a targeted question (e.g., block 608) and insert one or more portions of said specific response into a new question to focus (e.g., further inquire) on one or more aspects of said specific response. For example, if the response to the orientation question “What is the treatment for coronary artery disease?” is the following triple: percutaneous coronary angioplasty – is a treatment for – coronary artery disease, an enhanced question template can be used to construct an enhanced question including “When should percutaneous transluminal coronary angioplasty not be used as a treatment for coronary artery disease?”. In a similar manner, each enhanced question can focus on features associated with a particular response. In some aspects of this disclosure, said features may include demographic considerations (such as age, sex, race, and / or similar content), complications (such as pregnancy, diabetes, heart disease, and / or similar conditions), alternative treatments (such as hypertension medications and / or similar methods), and / or similar content. According to further aspects, at least one enhanced question may include a series of enhanced questions.For example, the series of enhanced questions could include: “What is the second-line treatment for coronary artery disease?”, “What precautions should be taken when treating coronary artery disease in pregnant women?”, “Is coronary artery bypass surgery contraindicated in diabetic patients?” (e.g., an enhanced question template in the form of: “Is the treatment of the disease contraindicated for the population?”), “Is it possible for people taking hypertension medication to develop coronary artery disease complications?” (e.g., another enhanced question template in the form of: “Is there a risk of complications in the population?”), “What coronary artery disease complications might people over 50 years of age have?” (e.g., another enhanced question template in the form of: “What disease complications might the population have?”) and / or similar questions. Thus, a series of enhanced questions can reveal more information corresponding to a particular assertion (e.g., preference for treatment of the assertion relative to other treatments, considering healthcare provider guidelines; availability of treatment of the assertion relative to other treatments, considering cohort characteristics; regulatory issues; and / or similar questions). Depending on the aspects, in block 612, the question-answering system 104 can be probed more than once with any particular enhanced question, as described herein. For example, question-answering system 104 can probe more than once with the same enhanced question to detect further responses associated with the selected topic.

[0063] In block 614, it can be determined whether a response has been received for each enhanced question (e.g., response 108 via interface 103 of client device 102). If one or more enhanced questions have not received a response, method 600 can return to block 604 (e.g., with phantom display optional) to read more, more focused text (e.g., corresponding to the enhanced question) into question-answering system 104. It should be noted that method 600 can terminate in some cases where there is no response, such as in the absence of an answer. If one or more enhanced questions have received at least one response, method 600 can proceed to block 616. In block 616, the responses from question-answering system 104 for each enhanced question can be compiled (e.g., via interface 103 of client device 102). According to some aspects, if question-answering system 104 has been probed more than once with the same enhanced question, subsequent responses can exclude previously provided responses so that the user can probe for more possible and / or alternative responses, as described herein. According to other aspects, if the question-answering system 104 has been probed more than once by the same enhanced question, subsequent responses may provide the previously provided response at the end of the response list, rather than excluding the previously provided response from subsequent responses, so that the user can sequentially probe more possible and / or alternative responses before the previously provided response, as described herein. Here, re-evaluation of the previously provided response can confirm that there are no further possible and / or alternative responses to the same enhanced question in the text read into the neural network 105.

[0064] In block 618, data structure 107 can be generated to establish and / or update knowledge base 120. Specifically, data structure 107 can correlate responses to orientation questions with corresponding responses to augmentation questions. For example, data structure 107 can associate triples originating from a particular orientation question with one or more triples originating from one or more augmentation questions corresponding to said particular orientation question. Indicators can be used to connect various triples within data structure 107. According to various aspects of this disclosure, due to the orientation and augmentation questions, the generated data structure 107 goes beyond conventional triples, revealing more useful and detailed information about a topic.

[0065] In some respects, the order in which text associated with the selected topic is read into neural network 105 (e.g., in block 604) (e.g., the first order) may affect the returned response. Therefore, in block 620 (e.g., as an optional step, displayed as a phantom), the text associated with the selected topic (e.g., pre-selected or not pre-selected) can be reordered and / or shuffled (e.g., into a second, third, and / or similar order different from the first order), and the methods of blocks 604 through 618 are repeated. Depending on various aspects (e.g., if the optional step of block 620 has been performed), the response associated with the highest confidence can be selected to be included in data structure 107. Such a method allows neural network 105 to avoid giving different responses based on the order in which the text is read as input to neural network 105. Accordingly, including block 620 in method 600 can increase the confidence measure regarding the response. Depending on various aspects, block 620 can be performed a predetermined number of times.

[0066] In block 622, the generated data structure 107 can be transferred to a system (e.g., question-answering system 104) to be added to knowledge base 120. According to various aspects of this disclosure, each directed question response and each corresponding enhanced question response can be further reviewed by the user (e.g., via interface 122C and / or similarly) and cited (e.g., accepted) or suppressed (e.g., rejected) as described herein. According to various aspects, the generated data structure 107 can add to and / or modify specific assertions in the knowledge graph of knowledge base 120 to include related assertions that describe additional factors (e.g., preferences, queues, regulations, and / or similar factors) associated with the specific assertion.

[0067] According to various embodiments of this disclosure, method 600 can be repeated with a related topic (e.g., another health-related topic) to build a knowledge base 120 related to the related topic (e.g., a health-related knowledge base). Furthermore, although... Figure 6 The method references client device 102, but it should be understood that, as described herein, such steps can be similarly performed by editing device 110. In other words, client device 102 and editing device 110, and their respective associated functions, can be combined into a single device.

[0068] Figure 7 A flowchart is described for an illustrative method 700 for using a knowledge base 120 (e.g., constructed and / or maintained using the neural network 105 described herein) according to one or more embodiments of this disclosure. In block 702, a question-answering system 104 may receive a query 106. Depending on various aspects, the question-answering system 104 may receive the query 106 from a client device 102.

[0069] In some respects, client device 102 (e.g., such as...) Figure 1 The service recipient (e.g., searcher and / or similar person) described herein may be associated with a service recipient unrelated to editing device 110 and / or text source device 124. For example, at least one of editing device 110 or text source device 124 may be associated with a service provider that offers search-related services to the service recipient via client device 102. Depending on various aspects, such search-related services may be pre-ordered. Therefore, the service recipient and / or client device 102 may be authenticated via an authentication process before submitting query 106. Furthermore, in these aspects, client device 102 (e.g., associated with the service recipient) may not be configured to generate data structure 107 and / or transmit data structure 107 to question-answering system 104 to add to knowledge base 120 as described herein. In some aspects, client device 102 (e.g., associated with the service recipient) may be configured to send query 106 and receive response 108, as described herein.

[0070] Depending on other aspects, client device 102 may be associated with users (e.g., editors, subject or domain experts, trained experts, and / or similar individuals) associated with question-and-answer system 104, editing device 110, and / or text source device 124. For example, client device 102 (e.g., associated with a user) may be used to test responses 108 to query 106 and to build knowledge base 120 (e.g., Figure 6 In these respects, although as described in this article, Figure 7 The method references client device 102, but it should be understood that, in addition to client device 102 (e.g., associated with a user) and / or instead of client device 102, these steps may be performed by editing device 110.

[0071] Refer again Figure 7 In block 704, neural network 105 can determine response 108 to a query based on its knowledge base 120. Depending on various aspects, question-answering system 104 can access data structures stored in knowledge base 120 (e.g., via...). Figure 6 Method 600) is used to determine a response 108 to query 106. As another example, question-answering system 104 may invoke neural network 105 and lookup table 133 to obtain the information needed to determine a response 108 to query 106. Depending on some aspects, data structures can be browsed to determine a more detailed response to a more detailed query. In block 706, neural network 105 may transmit the determined response 108 to client device 102. In some aspects, response 108 may include raw text read from neural network 105 (e.g., ...). Figure 6The text extract (e.g., text response) of block 604. Depending on other aspects, response 108 may include new text generated based on the original text read into neural network 105 by means of a natural language question submitted to question-answering system 104.

[0072] Although the systems, apparatuses, and methods described herein are interpreted in a medical context, they are not limited to the described field. That is, it should be understood that the systems, apparatuses, and methods described herein can be similarly applied to any field (e.g., agriculture, astronomy, chemistry, humanities, psychology, sociology, zoology, and / or similar fields).

[0073] It should now be understood that the systems, apparatuses, and methods described herein are suitable for building and / or maintaining a knowledge base 120 using editing device 110. More specifically, the systems, apparatuses, and methods described herein provide not only a more efficient front-end system (e.g., for generating selective text input to neural network 105), but also a back-end system (e.g., the editing device 110 and interface 122 described herein) for curating the output of neural network 105 to build and / or maintain a knowledge base 120 associated with neural network 105.

[0074] While specific embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Furthermore, although various aspects of the claimed subject matter have been described herein, these aspects need not be used in combination. Therefore, the appended claims are intended to cover all such variations and modifications falling within the scope of the claimed subject matter.

Claims

1. A system for maintaining a knowledge base, the system comprising: A device including a processor and a memory, the memory storing program instructions that, when executed by the processor, cause the device to: Generate a first interface, wherein the first interface is configured as follows: Receive queries for transmission to the question-and-answer system; and Provide a response to the query from the question-answering system, wherein the response includes one or more suggested triples from a list of suggested triples; After selecting a specific triple from the list of suggested triples, a second interface is generated, wherein the second interface is configured as follows: Provide at least one evidence record associated with the specific triple, wherein each evidence record includes a text span supporting the specific triple; and Provide one or more control elements associated with each evidence record, wherein the one or more control elements include at least one of the following: A first control element, which can be selected to support the specific triple by referencing its corresponding evidence record and text span; or A second control element, which can be selected to prevent its corresponding evidence record and text span from being cited as supporting the specific triple; as well as The knowledge base is updated by generating a data structure based on the selection of one or more control elements.

2. The system of claim 1, wherein the system comprises more than one of the devices.

3. The system of claim 1, wherein the first interface is configured to receive the query as a triple, the triple having a first part, a second part, and a third part.

4. The system according to claim 3, wherein, The first interface is configured to receive the triplet, which includes one or more of an undefined first part, second part, and third part.

5. The system of claim 3, wherein the first interface is configured to receive at least one of the first portion, the second portion, or the third portion in a human-readable or machine-readable form, and wherein the first interface is further configured to automatically populate the first interface with a human-readable form corresponding to the machine-readable form after or during receiving at least one of the first portion, the second portion, or the third portion in a machine-readable form.

6. The system of claim 1, wherein each suggested triple in the list of suggested triples is associated with a corresponding evidence hyperlink, and wherein each corresponding evidence hyperlink can be selected to generate the second interface.

7. The system according to claim 1, wherein, The first interface is configured to provide the list of suggested triples in a predetermined order.

8. The system of claim 7, wherein the predetermined order is based on at least one of the following: text source date, text source origin, text source evidence strength, text source domain, or text source feature of interest.

9. The system of claim 1, wherein the second interface is further configured to visually distinguish the components of the particular triple within each text span corresponding to each evidence record.

10. The system of claim 1, wherein the one or more control elements include at least one of the first control element, the second control element, or the third control element, and at least one of the first control element, the second control element, or the third control element may be selected to add triples supported by corresponding evidence records and text spans.

11. The system of claim 1, wherein, when executed by the processor, the program instructions further cause the device to: Receive a text query for transmission to a text source device, wherein the text query relates to a specific topic; and Multiple texts related to the specific topic are sent to the question-answering system for input into the neural network of the question-answering system, and the received multiple texts serve as text responses to the text query.

12. The system of claim 11, wherein the text source device includes a database storing text bodies and a full-text search engine configured to identify the plurality of texts from the text bodies.

13. The system according to claim 1, wherein, The first interface is configured to receive the query in the form of a question in natural language, the received query being specific to a particular topic.

14. The system according to claim 13, wherein, The first interface is also configured to generate one or more target questions based on at least one of the following: Specific triples stored in the knowledge base, wherein the one or more target questions include at least one of the following: True or false questions derived from one or more of the first, second, and third parts of the specific triple; or Fill-in-the-blank questions derived from the following methods: Edit at least one of the first, second, or third parts of the specific triple; as well as The fill-in-the-blank question is formed from at least one of the first, second, or third parts of the remaining specific triples; or The received query, the one or more target questions, includes at least one of the following: The question of replacing terms with alternative names or synonyms in the received query; or The issue of expanding or narrowing the scope of terms in the received query.

15. The system according to claim 14, wherein, The first interface is also configured to generate one or more enhanced questions based on at least one of the following: A specific set of suggested triplets provided in response to the one or more target questions, the one or more enhanced questions including: The following are the key issues raised: Extract at least one of the first portion, the second portion, or the third portion of the specific suggested triple; and The focus question is formed from at least one of the extracted first part, second part, or third part, to further inquire about at least one of the extracted first part, second part, or third part.

16. The system of claim 15, wherein the extracted further queries regarding at least one of the first portion, the second portion, or the third portion focus on at least one of guidance considerations, queue considerations, or regulatory considerations.

17. A system for maintaining a knowledge base, the system comprising: A question-and-answer system, comprising a processor and a memory, wherein the memory stores: Neural networks; knowledge base; A triplet database is recommended; as well as Program instructions, when executed by the processor, enable the question-and-answer system to: Multiple texts received from the device are read as input to the neural network; The neural network receives one or more natural language questions as input. The neural network is used to generate a neural network output that includes one or more text fragments, wherein the one or more text fragments are evidence for suggesting triples; The neural network output corresponding to the plurality of texts is stored as the suggestion triple in the suggestion triple database; Based on the suggested triples stored in the suggested triple database, respond to the query received from the device; as well as The knowledge base is updated based on one or more data structures received from the device. The specific triples in the suggested triples database can be selected via an interface configured to: Provide at least one evidence record associated with the specific triple, wherein each evidence record includes a text span supporting the specific triple; and Provide one or more control elements associated with each evidence record, wherein the one or more control elements include at least one of the following: A first control element, which can be selected to support the specific triple by referencing its corresponding evidence record and text span; or A second control element, which can be selected to prevent its corresponding evidence record and text span from being cited as supporting the specific triple.

18. The system of claim 17, wherein the memory further stores a lookup table, and wherein the program instructions, when executed by the processor, further enable the question-and-answer system to: Receive at least one of the first, second, or third portions of a triple from the device in a human-readable form; Access a lookup table to convert at least one of the first part, the second part, or the third part into a machine-readable form; and At least one of the first part, the second part, or the third part, converted into a machine-readable form, is transmitted to the device.

19. A method for maintaining a knowledge base, the method comprising: The query is input via the device's first interface and transmitted to the question-and-answer system. The system receives a response to the query from the question-answering system via the first interface, wherein the response includes one or more suggestion triples from the suggestion triple list; Receive at least one evidence record associated with a specific triple selected from the list of suggested triples via a second interface of the device, wherein each evidence record includes a text span supporting the specific triple; The control element associated with each evidence record is selected via the second interface, wherein the control element includes one of the following: A first control element is used to reference the corresponding evidence record and text span to support the specific triple; or A second control element is used to prevent the corresponding evidence record and text span from being cited as supporting the specific triple; as well as The selected control element is used to generate a data structure, thereby updating the knowledge base.

20. The method of claim 19, wherein inputting the query comprises inputting a triple comprising one or more of a first part, a second part, and a third part, and wherein receiving the response comprises receiving the list of suggested triples in a predetermined order based on at least one of a text source date, a text source origin, a text source evidence strength, a text source domain, or a text source feature of interest.

Images