Dialogue Query Answering System

Abstract

Embodiments of the present application relate to a conversational query answering system. Techniques for directing users to content based on semantic interpretation of a query entered by a user involve generating links to specific content in a document collection in response to a user string query, the links being generated based on an answer suggestion lookahead index. The answer suggestion lookahead index references mappings between sets of semantically equivalent terms and corresponding links to specific content in a document collection. These techniques are useful for the generalized task of natural language question answering.

Description

Technical Field

[0001] This specification relates to a query answering system using natural language processing (NLP). Background Art

[0002] Some search assistants can generate possible search strings as a user types words into the search assistant's text box. For example, when a user types the question "How do I remove devil eyes", the search assistant can present a new set of search strings that anticipate the question. Specifically, in response to the user typing "How do I remove", the search assistant can present the string "How do I remove app".

[0003] Conventional search assistants generate possible search strings based on a search operation performed on an aggregation of search strings submitted by a large number of users. For example, to generate possible search strings in response to a user typing "How do I remove", the search assistant searches for all possible search strings containing the words "How do I remove". If the user types additional words in the text box, the search assistant dynamically performs a search operation including the new words to generate a new set of search strings. In response to the user selecting a search string, the search assistant displays a set of search results based on a standard search algorithm.

[0004] When the question typed by the user does not contain words present in the aggregate of the search string, the conventional search assistant as described above cannot return the accurate search string. Attempts have been made to obtain a smarter search assistant that may be able to generate accurate search results in such situations. Unfortunately, such attempts have achieved only marginal success at best. Summary of the invention

[0005] In one general aspect, a method may include receiving document data representing a document collection, each document in the document collection including a corresponding topic and content related to the topic. The method may also include generating answer suggestion forward index data based on the document collection, the answer suggestion forward index data representing a plurality of term / link pairs, each term / link pair in the plurality of term / link pairs including a semantically equivalent term and a corresponding link to the content of the document collection. The method may also include receiving a user query string. The method may also include: in response to receiving the user query string, locating semantically equivalent terms of the term / link pairs in the plurality of term / link pairs, the semantically equivalent terms being located based on semantic equivalence with the user query string. The method may also include outputting a representation of the link of the located term / link pair to an output device.

[0006] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram illustrating an example electronic environment in which the improved techniques described herein may be implemented;

[0008] Figure 2 is a flow chart illustrating an example method for building a semantic look-ahead engine for search assistance according to the improved technique;

[0009] Figure 3 is a diagram illustrating an example portion of a document in a document collection of a help system;

[0010] Figure 4 is a flow chart illustrating an example process for generating a semantic annotation file;

[0011] Figure 5 is a flow chart illustrating an example process of generating a Document Object Model (DOM) from a semantically annotated file;

[0012] Figure 6 is a diagram illustrating an example DOM for a document in a document collection;

[0013] Figure 7 is a flowchart illustrating an example process of generating a knowledge graph for a document based on a DOM for the document;

[0014] Figure 8 is a flow chart illustrating an example process for generating a knowledge graph formatted for a natural language processing (NLP) pipeline;

[0015] Fig. 9 is a diagram showing an example portion of a formatted knowledge graph;

[0016] Fig.10 It shows the index generation for keyword search and suggestion<S,P,O> A flowchart of an example process for a triple;

[0017] Fig.11 is a flowchart illustrating an example NLP pipeline;

[0018] Fig.12 is a flowchart illustrating an example process for generating a serialized knowledge graph with annotations;

[0019] Fig.13 is a flow chart illustrating an example process for generating a master suggestion look-ahead index;

[0020] Fig.14is a flow chart illustrating an example process for generating weights for field-weighted search results;

[0021] Fig.15 is a flow chart illustrating an example process for presenting an answer card to a user on an output device;

[0022] Fig.16A is a diagram illustrating an example "how to" answer card in JSON format;

[0023] Fig. 16B is a diagram showing an example "what is" answer card in JSON format; and

[0024] Fig.17 is a diagram illustrating an example method of performing the improved technique. DETAILED DESCRIPTION

[0025] The above-mentioned traditional search assistants are not suitable as help assistants for specific products and services. For example, a typical search assistant may require thousands or tens of thousands of dedicated servers, which collect and aggregate search strings from a large number of users. Such a large number of servers may be very expensive to operate, making the cost beyond the scope of most service providers. In addition, in many cases, the number of possible search strings containing the words typed into the text box may be large, wherein the search string relevance is little or no relevance. And even when the search string is relevant, the search results based on the search string will probably not solve the problem considered by the user.

[0026] For example, consider a situation where a user wants to remove red eyes from a digital photo, but the user does not know the term "red eyes." Instead, the user types the following into the search assistant: "How do I remove devil eyes?" If this unusual way of expressing a query has never been expressed before, the search assistant may look for a close match to the typed characters, such as "How do I remove evil eyes?" In this case, the search assistant may provide links to astrology and religious websites that mention "the evil eyes," none of which will help the user.

[0027] According to the implementation described herein and in contrast to the above-mentioned conventional search assistants that require a large amount of server infrastructure (which is too much for most businesses and often fails to answer questions as intended by the user), the improved technology involves returning an accurate search string when the question typed by the user does not contain a word present in the search string aggregation. Specifically, the improved technology includes generating a link to specific content in a document collection in response to a user string query, the link being generated based on an answer suggestion look-ahead index. The answer suggestion look-ahead index references a mapping between a set of semantically equivalent terms and corresponding links to specific content in the document collection. In an example scenario, a computer receives a collection of help documents related to a product or product line. Each help document includes a topic and content related to the topic. The computer generates an answer suggestion look-ahead index based on the help document collection. Then, in response to receiving a user query, the computer generates a link to specific content in the help document collection. In some implementations, the computer may also output an answer card corresponding to the link, the answer card being a window displayed on an output device, in which content that answers the user's query is shown.

[0028] Advantageously, users do not need to use the answer suggestion look-ahead index to accurately formulate their queries. Although the traditional search assistant described above (which performs a search based on words typed by the user) will likely return links to content that is unrelated to the user's intention, the improved technology uses a natural language processor to process the input search string. Therefore, a help system based on the above mapping is likely to produce a single result, for example, a link to a document in a collection or a link to a location within a document, which accurately answers the user's query. Such a help system learns popular synonym terms without having to see billions of queries, and can automatically prompt answers based on question intent rather than text keywords, which are processed by traditional search assistants. The help system can also provide accurate answers in a compact answer card format, thereby eliminating the need for users to flip through result documents, thereby eliminating reliance on large and expensive search infrastructure, and thereby giving users more confidence in search results. The help system (referred to as "semantic look-ahead") can also dynamically generate links when users type their queries, which allows users to obtain relevant search results faster than systems that only output search results after full input has been entered.

[0029] To achieve the above features, the system first generates a semantic document object model (DOM) for each content page to capture topic relationships and descriptions. The system then constructs a knowledge graph based on the compiled DOM and exploits the discovered parent / child topic relationships, how-to processes, and related topics within the DOM structure. Before building the final question / answer factual knowledge graph, the system exploits very useful topic structure and domain information through the document understanding process.

[0030] In addition, documents for highly competitive products of a proprietary nature (e.g., documents for sensitive government, pharmaceutical, scientific content, or legal applications) are not accessible by large public search engines. For such documents, it is desirable to be able to deploy intelligent question answering systems locally and completely within a single enterprise or across a federated enterprise environment. With a collection of documents as input, the improved technology specifically provides such capabilities.

[0031] Another advantageous feature is that the above-described system is a process for automatically learning answer indexing weights from example question / answer pairs using machine learning techniques and optimizing the argmax() function. These answer indexing weights form the backbone of the above-described semantic lookahead help system, as various parts of the user input (e.g., subject, predicate, title) are weighted in terms of importance with respect to locating semantic equivalents of those parts of the user input in the knowledge graph. This allows the system to quickly generate accurate answer proposals by directly using product help search query logs from a smaller purpose-driven audience. The answer indexing weights are completely distinguishable from the techniques used in traditional search assistants, as such traditional search assistants do not have semantic lookahead capabilities as described herein. Instead, any "lookahead" implemented by traditional search assistants is used in the context of finding exact matches of words in an input string using words in a search database.

[0032] In summary, the improved techniques described herein are directed to fast, accurate search results based on imprecise user input. These search results are constructed by building a knowledge graph based on indexing of content in a document collection (i.e., semantic look-ahead data). As described above, indexing of content can be performed by appropriate weighting of the various parts of the search string, and is suitable for returning search results in real time as the user types; this is a semantic look-ahead of the user input string. The knowledge graph so constructed includes pairs of semantically equivalent terms and links to content in a document collection. In this way, a user can form a search string imprecisely while the search string is being entered and obtain relevant search results.

[0033] In some implementations, generating answer suggestion look-ahead index data includes generating topical knowledge graph data based on a document collection, the topical knowledge graph data representing a topical knowledge graph, the topical knowledge graph including: (i) a corresponding topic and a sub-topic set for each document in the document collection, and (ii) links to content of the document collection, each link corresponding to a corresponding topic or one of the sub-topics in the sub-topic set for each document in the document collection.

[0034] In some implementations, generating answer suggestion look-ahead index data includes generating semantic annotation data representing respective semantic annotations for a topic and a set of subtopics for each document in a document collection, each semantic annotation including a corresponding predicate in a set of predicates and a corresponding object in a set of objects; for each predicate in the set of predicates, identifying at least one object in a set of objects that, when combined with the predicate, corresponds to a topic for each document in the document collection or one of subtopics in a set of subtopics; and for each object in the set of objects, identifying at least one predicate in the set of predicates that, when combined with the object, corresponds to a topic for each document in the document collection or one of subtopics in a set of subtopics.

[0035] In some implementations, generating subject knowledge graph data includes: for each document in a document collection, generating a corresponding DOM in a document object model (DOM) collection, the corresponding DOM corresponding to the document, the DOM corresponding to the document including a topic, a sub-topic collection, and a link to the content of the document; and performing an aggregation operation on the DOM collection to generate a main link list for the subject knowledge graph, the main link list including multiple unique links to the content of the document collection.

[0036] In some implementations, generating the topic knowledge graph data includes: for each document in the document collection, generating a corresponding DOM in a document object model (DOM) collection, the corresponding DOM corresponding to the document, the DOM corresponding to the document including a topic, a sub-topic collection, and a link to the content of the document; and performing a knowledge graph formatting operation on the DOM collection to generate an aggregated flattened knowledge graph formatted for a natural language processing (NLP) pipeline, the aggregated flattened knowledge graph including each of the topics and sub-topic collections of each DOM in the DOM collection, the NLP pipeline being configured to generate<S,P,O> Triplet,<S,P,O> A triple includes a subject, a predicate, and an object for each of the topics and subtopic sets of each DOM in the DOM set.

[0037] In some implementations, generating subject knowledge graph data includes performing a split operation on an aggregated flattened knowledge graph to produce a plurality of aggregated flattened knowledge graph parts; and performing an NLP operation on each of the plurality of aggregated flattened knowledge graph parts through an NP pipeline to produce a knowledge graph, and an MLP operation is performed on each of the plurality of aggregated flattened knowledge graph parts in parallel.

[0038] In some implementations, generating the DOM in the DOM collection includes performing a discovery operation on each document in the document collection to identify non-information content of the document; and performing a removal operation on the non-information content of the document to produce a collection of topics, sub-topics, and links to content of the document, the removal operation including applying a term frequency-inverse document frequency (TF-IDF) algorithm to the document.

[0039] In some implementations, generating the DOM in the DOM collection includes performing a reformatting operation on each document in the document collection to produce a document formatted in Markdown Markup Language (MDML).

[0040] In some implementations, generating answer suggestion look-ahead index data includes obtaining search query log data, the search query log data representing a mapping between user query data and links to content of a document collection, the user query data representing multiple user queries; and performing a text mining operation on the search query log data to produce a common user query set for one of a topic of a document in the document collection or a corresponding sub-topic in a set of sub-topics, each of the topics and sub-topic sets being associated with corresponding content corresponding to a corresponding link.

[0041] In some implementations, generating answer suggestion lookahead index data includes forming pairs of: (i) corresponding user queries in the set of public user queries, and (ii) corresponding links to content in the set of documents, each pair based on an annotated set of topics and subtopics of the topic knowledge graph, the annotated set of topics and subtopics including topic titles and<S,P,O> Triplet,<S,P,O> The triples include a subject, a predicate, and an object for each of the set of topics and subtopics. In such an implementation, generating a link to a particular content in the set of documents includes identifying a pair of a user query and a link to the content that is a semantic best match of the predicate and the object to the predicate and the object of the received user query string.

[0042] In some implementations, forming pairs of user queries and links to content includes generating respective weights corresponding to each of a subject title, a subject, a predicate, and textual content of a subject title corresponding to the pair of user queries and links to content. In such implementations, identifying pairs of user queries and links to content that are semantically best matches to the predicate and object of the received user query string includes using respective weights corresponding to each of the subject title, subject, predicate, and textual content of a common user query set to generate a linear combination of the subject title, subject, predicate, and textual content that most closely matches the same linear combination of the subject title, subject, predicate, and textual content corresponding to the user query string.

[0043] In some implementations, generating a corresponding weight corresponding to each of a topic title, a subject, a predicate, and text content of a topic title corresponding to a pair of a user query and a link to content includes: generating a matrix having a plurality of rows, each of the plurality of rows having elements equal to the corresponding weights of the topic title, the subject, the predicate, and the text content; forming a merit function for each pair of each user query and the link to content in a public user query set, the merit function using as independent variables: (i) a search function of the user query data, the link to the content of the document collection, and the row of the matrix, and (ii) a specific link to the content of the document collection, the search function returning the link as an output, the merit function having a value of 1 in response to the link output of the search function being equal to the specific link, and having a value of 0 in response to the link output of the search function not being equal to the specific link; and identifying the row of the matrix for which the merit function presents a maximum value as the weight corresponding to each of the topic title, the subject, the predicate, and the text content.

[0044] In some implementations, obtaining the search query log data includes performing a machine learning operation on the raw search query data to produce a mapping between the user query data and links to content of the document collection.

[0045] In some implementations, the knowledge graph also includes links for a topic to content corresponding to other topics.

[0046] In some implementations, the link set includes a single link. In such implementations, outputting the link set to the output device includes displaying an answer card corresponding to the single link, the answer card including a window containing content corresponding to the link.

[0047] In some implementations, generating a link to specific content in the document collection includes locating content including text indicating steps of a process; and identifying a topic to which the content corresponds.

[0048] Figure 1 1 is a diagram illustrating an example electronic environment 100 in which the above-described improved techniques may be implemented. Figure 1 , electronic environment 100 includes computer 120 and display 190 viewed by user 192 .

[0049] The computer 120 is configured to display a user-specified drawing on a display device. The computer 120 includes a network interface 122, one or more processing units 124, a memory 126, and a display interface 128. The network interface 122 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from the network into electronic form for use by the editing computer 120. The group of processing units 124 includes one or more processing chips and / or components. The memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory (such as one or more ROMs, disk drives, solid-state drives, etc.). The group of processing units 124 and the memory 126 together form a control circuit that is configured and arranged to perform various methods and functions as described herein.

[0050] In some embodiments, one or more components of computer 120 may be or include a processor (e.g., processing unit 124) configured to process instructions stored in memory 126. Figure 1 Examples of such instructions shown in include document collection acquisition manager 130, knowledge graph generation manager 140, search query log acquisition manager 150, answer suggestion look-ahead index manager 160, user query acquisition manager 170, answer card presentation manager 180, and display manager 182. In addition, as Figure 1 As shown in , the memory 126 is configured to store various data that are described with respect to various managers that use such data.

[0051] The document collection acquisition manager 130 is configured to acquire document data 132 representing a document collection that includes content that can be presented to a user who has access to the document collection. In some implementations, the document collection acquisition manager 130 acquires the document data 132 via a network connection using the network interface 122. In some implementations, the document collection acquisition manager 130 acquires the document data 132 via a storage medium (e.g., a disk drive, a flash drive, etc.).

[0052] Document data 132 represents a collection of documents to be searched in response to a user query. In some implementations, the document collection includes help documents for a product or product line. In some implementations, the document collection includes documents that form a private intranet for an enterprise. Document data 132 includes topic data 134, content data 138, and link data 139. In some implementations, document data 132 includes subtopic data 136.

[0053] The subject data 134 indicates a subject title. Each document in the document collection includes a subject title indicating subject content disclosed in the document and content related to the subject title.

[0054] Subtopic data 136 represents a subtopic title set of each document in the document set. Each subtopic in the subtopic set of a document is related to the subject of the document. In addition, a portion of the content of the document is related to a corresponding subtopic in the subtopic set of the document.

[0055] Content data 138 represents various content included in each document in the document collection, related to the topic represented by topic data 134 and the collection of subtopics represented by subtopic data 136. In some implementations, the content includes text descriptions. In some implementations, the content includes other media, such as graphic media, video media, audio media, etc.

[0056] Link data 139 represents a link to content that can be accessed by a browser or search engine according to the improved techniques described herein. In some implementations, the link includes a uniform resource locator (URL) address (e.g., a web server, not necessarily computer 120) where the document is stored. In some implementations, the link also includes an anchor that indicates the location within the document where the interesting content related to the subtopic is located.

[0057] The knowledge graph generation manager 140 is configured to generate knowledge graph data 142 based on the document data 132. The operation of the knowledge graph generation manager 140 is as follows: Figures 4 to 12 Described in detail.

[0058] The knowledge graph data 142 represents a knowledge graph. A knowledge graph is a hierarchical arrangement of topics, subtopics, and links that also includes information about the semantic content of the text content of a collection of documents and their relationships with links that can be used to access the content. The knowledge graph data 142 includes semantic annotation data 144 and aggregated link data 146.

[0059] The semantic annotation data 144 represents the grammatical parts of the topic and subtopic titles and the text content extracted by a natural language processor (NLP). For example, NLP can extract the subject, predicate, and object (<S,P,O> As a specific example, consider the topic title "Use the text on shape tool". The predicate may be "use", and the object may be "text" or "text on shape tool". Although not explicitly stated in a command format, the subject may be inferred to be "you" or "user". The knowledge graph generation manager 140 is configured to arrange the output of the NLP into a semantic annotation file for reference by a search engine.

[0060] Aggregate link data 146 represents all links in the links to topics and subtopics in the document collection (ie, document data 132). The knowledge graph generation manager 140 is configured to extract and aggregate these links, thereby eliminating duplication and generating an indexing scheme by which the links are referenced in the knowledge graph.

[0061] The search query log acquisition manager 150 is configured to acquire search query log data 152. In some implementations, the search query log acquisition manager 150 acquires the search query log data 152 via a network connection using the network interface 122. In some implementations, the search query log acquisition manager 150 acquires the search query log data 152 via a storage medium (e.g., a disk drive, a flash drive, etc.).

[0062] Search query log data 152 represents a search query log generated by a user searching a document collection. For example, some users enter a query and, in response, click on a link provided by a search engine. In response, some users may not find a useful link and may take another action, such as trying another query or not clicking on any link. Search query log data 152 includes query data 154 and link data 156.

[0063] The query data 154 represents queries entered by a plurality of users when attempting to find content in a document collection. In some implementations, the query data 154 is in the form of a string of characters formed into discrete words, as typically expressed in a search query. The search query log acquisition manager 150 collects queries entered into the search engine by users and stores them in a central log. In some implementations, the search query log acquisition manager 150 formats the collected queries for natural language processing to extract information such as the content of the query from the query.<S,P,O> Semantic information of triples. Storing queries in such a semantic form provides a connection between the query and the knowledge graph.

[0064] Link data 156 represents links selected in response to a query. Links include URLs indicating locations where documents are stored (e.g., on a remote web server). Each link corresponds to a topic that defines the content of the document. In some implementations, links include anchors corresponding to locations of subtopics in the document. For example, an anchor may be represented by a "#" symbol followed by characters in the subtopic following the URL in the link.

[0065] The answer suggestion look-ahead index manager 160 is configured to generate answer suggestion look-ahead index data 162 based on the knowledge graph data 142 and the search query log data 152. The answer suggestion look-ahead index data 162 represents an answer suggestion look-ahead index that provides a link or a set of links to the content of a document collection to a user in response to a query input by a user. Because the knowledge graph data 142 and the search query log data 152 have semantic annotations (e.g.,<S,P,O> The answer suggestion look-ahead index data 162 includes semantically equivalent term data 164.

[0066] The semantic equivalent term data 164 represents phrases that are found to be semantically equivalent because they are known to be associated with the same link. For example, the phrases "remove red eye", "filter reflections from eyes in pictures", and "eliminate red eyes" can all be associated with a link to the topic "How to remove red eyes from photographs". Each phrase and topic has a semantic annotation that has been analyzed for the semantic annotation of the topic title. By analyzing the semantic content of new input queries such as "How do I remove the devil eyes", the answer suggestion look-ahead index manager 160 is able to display links to the user even if the query includes words that are not in the search query log.

[0067] The user query acquisition manager 170 is configured to acquire user query data 172 representing a user query from a search engine. In some implementations, the user query acquisition manager 170 is configured to extract semantic content (e.g.,<S,P,O> triples).

[0068] The answer card presentation manager 180 is configured to display content in an answer card in response to presenting only a single link in response to a query or a user selecting a link from a plurality of links presented. An answer card is a window that is specifically configured to present such content. In some implementations, an answer card may take different forms, depending on whether the associated topic is a "how to" topic or a "what is" topic.

[0069] In some implementations, display manager 182 is configured to send content within an answer card to display device 190 .

[0070] The components of the computer 120 (e.g., modules, processing unit 124) can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of the computer 120 can be configured to operate within a device cluster (e.g., a server farm). In such an implementation, the functions and processing of the components of the computer 120 can be distributed to several devices of the device cluster.

[0071] The components of computer 120 may be or may include any type of hardware and / or software configured to process attributes. Figure 1 One or more parts of the components of the computer 120 in the embodiment may be or may include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., computer code modules, a set of computer-readable instructions that can be executed on a computer). For example, in some implementations, one or more parts of the components of the computer 120 may be or may include software modules configured to be executed by at least one processor (not shown). In some implementations, the functionality of the components may be included in a computer program that is associated with the computer program. Figure 1 those shown in different modules and / or different components.

[0072] In some embodiments, one or more components of the computer 120 may be or may include a processor configured to process instructions stored in a memory. For example, the document collection acquisition manager 130 (and / or a portion thereof), the knowledge graph generation manager 140 (and / or a portion thereof), the search query log acquisition manager 150 (and / or a portion thereof), the answer suggestion look-ahead index manager 160 (and / or a portion thereof), the user query acquisition manager 170 (and / or a portion thereof), the answer card presentation manager 180 (and / or a portion thereof), and the display manager 182 (and / or a portion thereof) may be a combination of a processor and a memory configured to execute instructions related to a process for implementing one or more functions.

[0073] In some implementations, the memory 126 can be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, the memory 126 can be implemented as more than one memory component associated with a component of the editing computer 120 (e.g., more than one RAM component or disk drive memory). In some implementations, the memory 126 can be a database memory. In some implementations, the memory 126 can be or include a non-local memory. For example, the memory 126 can be or include a memory shared by multiple devices (not shown). In some implementations, the memory 126 can be associated with a server device (not shown) within a network and configured to serve the components of the editing computer 120. Figure 1 As shown in , the memory 126 is configured to store various data, including document data 132 , knowledge graph data 142 , search query log data 152 , semantically equivalent term data 162 , and user query data 172 .

[0074] Figure 2 2 is a flowchart depicting an example process 200 for creating a semantic look-ahead index for a query of content in a document collection. The process 200 is described for a collection of help documents in a help system, but the process 200 can be applied to any document collection, such as an enterprise intranet. The process 200 can be combined with Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0075] At 202 , the document collection acquisition manager 130 acquires a help document collection.

[0076] At 204, the knowledge graph generation manager 140 crawls, filters, and annotates the help documents to produce a document set formatted in markdown / markup language (MDML), filtering out irrelevant text. Further details of this filtering and annotation of the help documents are described in detail in Figure 4 is described in more detail.

[0077] At 206, the knowledge graph generation manager 140 generates a document object model (DOM) for each help document in the document collection. Figure 5 and Figure 6 As described, the DOM of a document is a collection of topics, subtopics, and links to content associated with the topics and subtopics.

[0078] At 208, the knowledge graph generation manager 140 generates a knowledge graph for each help document, i.e., a topic corresponding to the documents associated with the topic from the DOM. Figure 7As described, a knowledge graph refers to an aggregated index of links to content in documents associated with a topic, as well as the content itself and links to other topics that may have some relevance to the topic.

[0079] At 210, the knowledge graph generation manager 140 forms a serialized knowledge graph for the help document collection. The serialized knowledge graph is formatted for input into the NLP pipeline.

[0080] At 212, the knowledge graph generation manager 140 splits the serialized knowledge graph into multiple files for input into the NLP pipeline. The NLP pipeline can then perform natural language processing on each split in parallel.

[0081] At 214, the NLP pipeline performs a map / reduce operation on each segment to generate a<S,P,O> Triplet.

[0082] At 216, the knowledge graph generation manager 140<S,P,O> The triples are merged into the serialized knowledge graph. Through the merging, the semantically annotated documents are linked to the serialized knowledge graph so that each sentence is associated with its triples.

[0083] At 218 , the answer suggestion lookahead manager 160 generates semantic lookahead suggestions based on the serialized knowledge graph with semantic annotations.

[0084] At 220 , the answer suggestion lookahead manager 160 deploys the knowledge graph topic answers for semantic lookahead suggestions.

[0085] At 222, the answer suggestion lookahead manager 160 generates weights for the elastic search text field index. The weights are calculated to maximize the likelihood that the user query will produce links and / or answer cards that accurately address the query intent.

[0086] Figure 3 is a diagram illustrating an example help document 202 to be processed as part of the answer suggestion look-ahead index. Figure 3 The help document 202 shown in FIG. 2 has a topic title 310 ("ADD TEXT") that defines the content in the document. In this case, the topic title 310 has a simple predicate ("add") and a simple object ("text"), and describes a feature or series of features in a software tool. The document 202 describes a "how to" help scenario.

[0087] Below the topic title 310 is the text content 312 corresponding to the topic title 310. The text content 312 includes sentences describing a feature or a set of features. These sentences and the topic title 310 will be reformatted for and processed by the NLP pipeline so that the semantic structure of the topic 310 and the content 312 can be standardized to a common platform, for example,<S,P,O> Triplet.

[0088] Below the content 312 is a subtopic title 320 written as "About Text". The subtopic 320 title is related to the topic title 310 and can be linked to using an anchor, for example, a "#" followed by a character that identifies the location in the document 202 where the subtopic title is placed. There is also text content 322 corresponding to the subtopic title 320. Like the text content 312, the text content 322 includes sentences that will be reformatted for and processed by the NLP pipeline so that the semantic structure of the topic 310 and the content 312 can be standardized to a common platform, such as<S,P,O> Triplet.

[0089] Document 202 also has another subtopic title ("Add Text") and text content 332 corresponding to the subtopic title. In this case, text content 332 includes enumerated steps in a "how to" process. The NLP pipeline processing operation can identify the sentences as steps and format the answer card including the text accordingly.

[0090] Figure 4 2 is a flowchart depicting details of a process 204 for creating a semantic annotation file from a collection of help documents 202. The semantic annotation file 412 is used to extract topic, subtopic, and link information for a semantic look-ahead index used in identifying and presenting answer cards in response to user queries. The process 204 may be performed in conjunction with Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0091] At 404 , the knowledge graph generation manager 140 performs a crawling operation on each document in the document collection 202 to identify a corpus of text that is ultimately organized in a knowledge graph.

[0092] At 406, the knowledge graph generation manager 140 performs a reformatting operation on each document in the document collection in Markdown Markup Language (MDML) to generate a formatted document collection. MDML is a lightweight markup language that is human-readable and convertible to a standard markup language such as XHTML or Rich Text Format (RTF).

[0093] At 408, the knowledge graph generation manager 140 performs a discovery operation on the extracted text corpus of each formatted document in the set of formatted documents to identify non-information content in the document. To achieve this, the knowledge graph generation manager 140 obtains (e.g., learns) a model to filter out such content. Examples of such language include legal disclaimers and contact information.

[0094] At 410, the knowledge graph generation manager 140 performs a filtering or removal operation on the identified non-information content of each formatted document in the set of formatted documents. In some implementations, the model used to filter out such content from the text corpus is a term frequency-inverse document frequency (TF-IDF) algorithm. In some implementations, the knowledge graph generation manager 140 calculates the inverse document frequency (IDF) of the text corpus:

[0095]

[0096] Where t is a candidate noise term (i.e., an N-gram, such as "Contact corporate support"), and N = |D| is the total number of documents in the document collection. The term |{d∈D:t∈d}| in the denominator is the number of documents in which the candidate noise term t appears, i.e., the non-zero term frequency.

[0097] The filtering process then includes the following: (i) collecting and ranking the M candidate noise terms using the above expression for idf(t, D); (ii) summarizing each term with a regular expression for the variable part, such as ((TERM1|TERM2|...|TERM i |...|TERM N )+[]*)\[[0-9]+\]; (iii) collect term patterns in a noise term stop list; and (iv) apply the term encountered from documents in the parsed document collection and, if found, remove the term pattern from the annotation.

[0098] At 412, the knowledge graph generation manager 140 performs a parsing operation on each formatted document in the formatted document collection to identify topic titles, subtopics, text descriptions, and links. The parsing operation generates a semantic annotation file 414, which organizes the identified topic titles, subtopics, text descriptions, and links for reference by the suggested semantic look-ahead index. In some implementations, the knowledge graph generation manager 140 uses the Beautiful Soup package to generate a parse tree for the identification of topic titles, subtopics, text descriptions, and links.

[0099] Figure 54 is a flowchart depicting the details of the process 206 for creating a collection of document object models (DOMs) from semantic annotation files 414. Each DOM includes a top-level topic and related subtopics. The process 206 may be performed by combining Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0100] At 502 , the knowledge graph generation manager 140 identifies topics and subtopics of the semantic annotation files 414 associated with each document in the document collection 202 .

[0101] At 504, the knowledge graph generation manager 140 arranges the identified topics and subtopics from each document in the document collection 202 into a DOM 506. The DOM corresponding to the documents in the document collection 202 has a hierarchical organization of topics and subtopics. In some implementations, the DOM 506 is represented in a recursive JSON format.

[0102] Figure 6 2 is a flowchart depicting details of a process 208 for creating a document knowledge graph (KG) set from each DOM corresponding to a respective document in a document set. Each document KG includes a link identifier that identifies a link to content in each document in the document set. Process 208 may be performed in conjunction with Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0103] At 602, the knowledge graph generation manager 140 aggregates and indexes the links identified in the semantic annotation file 414. When aggregating the links, the knowledge graph generation manager 140 identifies redundant (e.g., non-unique) links and removes them from the master reference link list. From the master list, the knowledge graph generation manager 140 generates a public index of the links from the master reference list.

[0104] At 604, the knowledge graph generation manager 140 generates links between topics. In this way, the KG corresponding to the topic corresponding to the document can provide a list of links to other topics. In this way, the knowledge graph generation manager 140 generates a document KG 606 from each DOM using a link index.

[0105] Figure 7 5 is a diagram showing an example DOM 506 in JSON format. Figure 7As shown in FIG. 1 , the DOM lists a topic title 702 (“Add text”) at the top of the hierarchy. Below the hierarchy are subtopics 704, e.g., “About text,” “Add text,” “Use the Text on Shape Tool,” etc. At the same hierarchical level as the subtopics 704 is a text description 706 of the topic (which corresponds to Figure 3 The text content in 312).

[0106] Each subtopic 704 includes a link reference 708, for example, "1585" having a subtopic of "About text", etc. Each link reference 708 is taken from a common index of links and corresponds to a link that generates a subtopic.

[0107] The text description of the subtopic is shown at 710 ("About text") and 712 ("Add Text"). Each line of the text description corresponds to a step in the process described in the corresponding document in the document collection 202. In addition, some lines of the text description include link references (e.g., "633" in line [0] of the text description of subtopic 710). These link references correspond to actual links in the corresponding document in the document collection 202.

[0108] There is a "See also" entry 714 for other topics. When expanding entry 714, links to other topics and subtopics in other documents in document collection 202 can be observed. In addition, at 716, there is a global link list to other content not included in document collection 202.

[0109] Figure 8 is a flowchart depicting details of a process 210 for creating a formatted KG for natural language processing (NLP). The process 210 may be performed by combining Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0110] At 802, the knowledge graph generation manager 140 performs an aggregation operation on the KG for each document in the document collection 202 to produce a flattened KG 806. The flattened or serialized KG 806 is arranged in a format suitable for submission to an NLP pipeline. Such an NLP pipeline will parse each topic text description (e.g., text description 706) and then perform semantic identification and extraction from the text description (e.g.,<S,P,O> generate).

[0111] At 804, the knowledge graph generation manager 140 records all link references and corresponding links (i.e., URLs and link anchors). This record will allow search results received from users to take users directly to relevant content in documents in the document collection 202. In some implementations, the references and links are recorded in JSON format.

[0112] Fig. 9 As shown above, Figure 8 900 of a portion of the exemplary planarized KG 806. The portion may be Figure 7 506 is compared to the JSON format shown in . Fig. 9 As shown in , the flattened KG 806 lists each topic title, subtopic, and text description on a separate line. Each line has a reference numeral 902 to replace the hierarchical arrangement of the JSON format.

[0113] Flattened KG 806 also has an indicator 904 that indicates that a row in flattened KG 806 is of type "Topix", "Doctx", "Triples", or "Index". A "Topix" ​​type refers to a row associated with a topic title. A "Doctx" refers to a row associated with textual content. A "Triples" refers to a row associated with textual content that will be parsed for semantic content. An "Index" refers to a row associated with a subtopic.

[0114] The flattened KG 806 also has a hierarchical DOM path 906 corresponding to the DOM structure of the documents of the document collection 202 and link references 908 .

[0115] Fig.10 is a flow chart depicting the details of the process 212 for performing semantic analysis on the formatted KG 806. The process 212 may be performed by combining Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0116] At 1002, an NLP pipeline (e.g., Adobe SedonaNLP) performs a MAP / REDUCE split operation on the formatted KG 806 to produce N separate files, where N is the number of processor cores on which semantic analysis is performed. In some implementations, N is at least 8. In some implementations, N is at least 16. In some implementations, N is 52. Such a split is advantageous because the document collection 202 may have hundreds of documents, each of which includes approximately 10-20 subtopics, descriptions, and elaborations. The formatted KG 806 may include hundreds of thousands of topics, subtopics, descriptions, and instructions to be parsed and annotated.

[0117] At 1004 (ie, Figure 2 214), the NLP pipeline performs semantic analysis on each of the N individual files in parallel. This analysis involves generating a MAP / REDUCE script that dispatches each of the N individual files to a corresponding MAP process instance for parallel execution. Executing the MAP / REDUCE script at the MAP process instance produces N mapping files. The N mapping files are then input into the corresponding REDUCE process instance - in some implementations, after a sorting process of the keys generated in the MAP process instance - to create keywords, phrases, and verbs / objects from the flattened KG 806<S,P,O> terms for final keyword search and suggestion indexing.

[0118] At 1006, the NLP pipeline combines all NLP annotations (e.g.,<S,P,O> The NLP annotations are identified by IDs for each topic and related subtopics of each document in the document collection 202. In some implementations, the NLP annotation types are as follows:

[0119] exprType==ID indicates the subject element in KG. The elements are as follows:

[0120] exprPOS - part-of-speech tag element of a sentence / phrase expression

[0121] exprRole - an indicator of whether the expr element is a subject, a predicate (e.g., a verb or preposition), or an object

[0122] exprType - An annotation indicating whether the element is a noun (NX), verb (VX), preposition (IN), or adjective (AX) expression. All topics and subtopics in the entire document collection 202 are then aggregated into a master document index.

[0123] Fig.11 is a flow chart illustrating an example NLP pipeline process 1100. Process 1100 may be performed by combining Figure 1 The described software constructs are executed by the computer 120 , which in some implementations reside in the memory 126 of the computer 120 and are executed by a set of processing units 124 .

[0124] The text content 1102 (e.g., a sentence) provides input text, which is first standardized to UTF-8 by a text conversion module 1104, and then segmented and tokenized into grammatical language units by a segment segmenter 1106, a paragraph segmenter 1108, a sentence segmenter 1110, and a word segmenter 1112. A stop word filter 1114 and a stem 1116 are applied to generate a structured model of the input text content elements 1118. The text model 1118 is then indexed by a text model index 1120 and sent to one or more NLP analysis processes. In some implementations, these processes include an N-gram extractor 1126, an N-gram classifier 1128, a topic extractor 1130, a topic classifier, and an N-gram persistence program 1134. In some implementations, these processes include a partial speech tree library trigger 1136,<S,P,O> Triple extractor 1138 , grammar transformer 1140 , triple database 1142 , named entity extractor 1144 , upper / lower ontology module 1146 , entity classifier 1148 , and language annotation persistence program 1150 .

[0125] In some implementations, the latter process (i.e., 1136-1150) is used to annotate the words of each topic title and description with part-of-speech tags (e.g., using Penn Treebank) at 1136, and then extract <subject, predicate, object> semantic triples from the annotated text using 1138, while tagging all discovered nominal terms using external word dictionary ontologies such as WordNet and custom ontologies 1144 and 1146. The tagged verb, noun, and adjective terms are classified and annotated by entity classifier 1148 to indicate their vocabulary role for a particular topic or subtopic, and the entire annotation output, expressed in JSON-LD form, is then saved at 1150 for subsequent processing.

[0126] Fig.12 1206 is a flowchart depicting details of process 216 for generating a serialized KG with annotations 1206. Process 216 may be performed by combining Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0127] At 1202, the knowledge graph generation manager 140 uses the keyword search and suggestion index 1008 to<S,P,O> Triple to generate a JSON format<S,P,O> Serialized file of triple annotations. Due to the 1002 ( Fig.10 ) is processed by a separate NLP MAP process, so in some implementations, at 1150 ( Fig.11) in the generated JSON file saved at<S,P,O> The resulting annotations are combined back into the final aggregate KG annotation file to generate<S,P,O> Index. The index allows users to:

[0128] Find all verb actions (e.g., create, edit, remove) that apply to a single object (e.g., image, mask), or

[0129] - For a single verb action (eg, update), find all relevant objects that the verb can be applied to (eg, color curve, hue and saturation, color of skin tone).

[0130] Topics generated from the NLP MAP / REDUCE processing stage<S,P,O> The annotations are represented in JSON format. Unresolved parent sections and their corresponding reference ID anchors are resolved. The example NLP and predicate and object annotations for the subtopic "Define path for spin blur" are as follows:

[0131]

[0132] At 1204, the knowledge graph generation manager 140<S,P,O> The annotation is added to the serialized KG 806 to produce the serialized KG 1206 with annotation.<S,P,O> Annotations produce many-to-one relationships for creating verb-to-noun search results. An example search experience for finding all verb commands (e.g., "appear in," "convert between," "create," etc.) for a specific noun object (e.g., "path") is as follows, where the italicized phrases are the topics corresponding to the corresponding search results:

[0133] Paths panel <= appear in–drawing modes

[0134] paths <= Convert between – see also

[0135] Path to selection <= convert – understanding shapes and paths

[0136] paths to selection borders <= Convert – see also

[0137] path<=Create type along or inside–create type along or inside a path

[0138] path<=define-spin blur (path<=define-skin color)

[0139] path <= duplicate – create panorama ocr.png (path <= copy – create panorama ocr.png)

[0140] paths <= edit – tutorials photography jumpstart (path <= edit – photography tutorial installation)

[0141] Path <= fill or stroke – understanding shapes and paths

[0142] Path <= find out how to create type along - tutorials photographyjumpstart (path <= find out how to create type along ... - photography tutorial installation)

[0143] Paths <= learn how to convert between-tutorials photography jumpstart (Paths <= learn how to convert between... - photography tutorial installation)

[0144] Paths <= learn how to edit or adjust-tutorials photography jumpstart (path <= learn how to edit or adjust-photography tutorial installation)

[0145] path<=move-create panorama ocr.png (path<=move-create panorama ocr.png)

[0146] The search experience similarly allows users to find all product noun objects that a verb command (e.g., "define") can be applied to using a one-to-many relationship between the verb and the noun (e.g., "perspective planes," "additional data sets," "constraints"). As shown below, the italicized phrases are the topics corresponding to the corresponding search results:

[0147] define and adjust=>perspective planes--define and adjust perspective planes in vanishing point

[0148] define=>additional data sets–define a data set

[0149] define=>constraints to indicate straight lines-create panoramaocr.png (Define=>constraints to indicate straight lines-create panoramaocr.png)

[0150] define=>data set – define a data set

[0151] define=>feathered edge--soften the edges of selections

[0152] define=>four corner nodes of plane surface-create panorama ocr.png (define=>four corner nodes of plane surface-create panorama ocr.png)

[0153] define=>horizon line--tutorials photography jumpset

[0154] define=>path-spin blur (define=>path-skin color)

[0155] define=>planes--adjust perspective

[0156] define=>point size unit–specify type size

[0157] define=>specific RGB–about color profiles

[0158] define=>variables--import a data set

[0159] define=>variables--define variables (define=>variables--define variables)

[0160] Fig.13 is a flowchart depicting details of the process 218 for generating a primary index suggestion forward index 1310 from a serialized KG 1206 with annotations. The process 218 may be performed by combining Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0161] At 1302, the answer suggestion look-ahead index manager 160 processes the search query log (e.g., Figure 1 The text mining operation is performed on the search query log data 152 in the search query log to identify common search questions. In some implementations, the text mining operation is based on the question or keyword frequency in the search query log. Common questions are used to discover and extract new user vocabularies that are unknown to the search system or document collection 202. For example, using this technology, questions such as "how do I remove devil eyes?" (where "devil eyes" is an unknown term) are easily discovered and can be mapped to equivalent questions about a "red-eye removal" tool.

[0162] At 1304 , the answer suggestion look-ahead index manager 160 uses the serialized KG 1206 to identify and form (answer, link) pairs.

[0163] At 1306, the answer suggestion look-ahead index manager 160 uses the common search question as a candidate term to be associated with the base word in the answer suggestion. In some implementations, this use can be accomplished through the document collection 202 and a new learning vocabulary for topics, titles, and / or sentences.

[0164] At 1308, the answer suggestion lookahead index manager 160 curates mappings from candidate terms to formal document words to produce a primary index suggestion lookahead index 1310. The candidate terms are discovered by text mining. In some implementations, curation can be performed by a content editor. Curation can allow new vocabulary mappings to be commented on and added in a supervised manner.

[0165] The process for automatically mapping candidate vocabulary terms learned from user search logs can be accomplished by increasing the amount of query search log data that is text mined and then statistically determining a term count threshold to decide when to accept new terms to be associated with formal base terms on documents in the document collection 202 corresponding to a particular topic.

[0166] In order to automatically acquire new vocabulary terms used by customers, a set of query log entry pairs consisting of (query term, link) pairs are processed and clustered to determine equivalent words or key terms for each topic or subtopic of documents in the document collection 202 corresponding to a specific topic page. The discovered terms or candidate terms with an occurrence count higher than a sample-based threshold are then selected and included in the knowledge graph topic, subtopic,<S,P,O> Triple annotations and descriptions are indexed.

[0167] return Figure 2 At 220, the answer suggestion lookahead manager 160 deploys the knowledge graph topic answers for semantic lookahead suggestions. The deployment sends the answers to the front-end browser or mobile phone client. This process allows the question / answer suggestion lookahead manager 160 to work independently of the back-end search system if desired. The generated answer suggestion lookahead index 1310 supports broad concept matching, not just strict keyword matching like existing search systems or open source lookahead modules.

[0168] Fig.14 14 is a flow chart depicting details of the process 222 for generating optimal weights 1410 for field weighted search results from the search query log 1402. The process 222 may be performed by combining Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0169] At 1404, the answer suggestion lookahead manager 160 generates a matrix ω corresponding to all possible weight combinations of four types of document text elements in order to utilize field-level indexing during search: subject, object, predicate, and text description. The reason for the weights is that the document text elements identified above have different importance. The weights are used for indexing and query processing. The selection of weights is performed by using question / answer pairs based on actual (query, link) pairs. In some implementations, such pairs are created by the product document quality team. In some implementations, such pairs are looked for in Adobe search logs. These pairs are used to determine the search engine's index field weights during query processing to significantly improve answer or search result relevance.

[0170] Therefore, the matrix ω has four columns and M 4 rows, where for each weight, M is the number of samples between 0 and 1. For example, when M=3, the matrix ω takes the following form:

[0171]

[0172] Each column of the matrix ω corresponds to a corresponding document text element: the subject title field weight ω topic ,<S,P,O> Predicate field weight ω predicate ,<S,P,O> Object field weight ω object and the subject text field weight ω text .

[0173] At 1406, the answer suggestion lookahead manager 160 forms a merit function based on whether the search function output using the weights from the rows of the matrix ω produces a correct ground truth query. The ground truth is a set of pairs (query q i , link L GTi That is, the ground truth represents a set of links selected by the user after the user enters the query into the search engine. i = "add caption to image", user selects link L GTi =https: / / helpx.adobe.com / photoshop-elements / using / add-text.html.

[0174] Considering the above defined weights, the search function of the search engine can be defined as follows:

[0175]

[0176] Where S k , k∈{1, 2, …, m} is the link L returned by the search enginek That is, the search engine returns the top m links according to the scores based on the weights used.

[0177] It is meaningful to determine which weights will return the most correct ground truth query. In some implementations, such a determination can be made using the following merit function:

[0178]

[0179] There are K ground truths, and

[0180]

[0181] L k Indicated by f search The link returned by the function.

[0182] At 1408, the answer suggestion lookahead manager 160 uses a merit function to identify the row of the matrix that maximizes the number of basic fact queries 1410. Thus, ω′ represents the values ​​of the four weights that maximize the number of basic fact queries. In some implementations, it has been found that such weight values ​​may be equal (ω topic ,ω predicate ,ω object ,ω text )=(0.44, 0.22, 0.22, 0.11).

[0183] The above discussion has suggested a look-ahead index and a search engine for forming answers that will provide answers to user queries based on the user's intent. However, the front end of the search engine accepts a query as input and in some implementations provides an answer card that directly provides the answer to the user's intended user query.

[0184] Fig.15 1 is a flow chart depicting details of a process 1500 for presenting an answer card to a user in response to a user query. The process 1500 may be performed in conjunction with Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0185] At 1502, the answer card presentation manager 180 generates "how to" and "what is" answer cards as JSON formatted files. Each answer card is based on a text description corresponding to a topic or subtopic of a document in the document collection 202. The answer card is about Fig.16A and Fig. 16B Described in further detail.

[0186] At 1504, the answer card presentation manager 180 defines a heuristic method to identify the help instruction step. For example, one heuristic method involves placing a " <number>" or "Step <number>" as a step. Other heuristics include learning markers in the help page text that indicate step elements, for example, CSS class names such as "step-title" or "headline".

[0187] At 1506, the answer card presentation manager 180 extracts the process title by extracting the parent title DOM node of the step node. For each step node, the span of the step is included to include more DOM nodes. As an example, consider the JSON How-To representation at 1502. In the list, DOM nodes starting with "1. To apply a Curves adjustment…", "2. Do one of the following", etc. are extracted as steps (through heuristic rules). Then, the span of each step is extended - for example, the span of step 2 is extended to include the next line "Click the Curves Icon" up to step 3. Once the steps are identified, the pipeline extracts their process titles by extracting the parent title DOM node of the step node. In the example above, "Adjust image color and tone with Curves" is extracted as the title of the step.

[0188] At 1508, the answer card presentation manager 180 creates and maintains a knowledge base of answer cards from which the best answer for any given query can be retrieved. Using this knowledge base, a search of the question database uses a semantic similarity search to locate canonical questions and relevant answer cards for any input question intent.

[0189] At 1510, answer card presentation manager 180 presents the answer card to an output device (e.g., Figure 1 The answer card is presented to the user on the display 190 in FIG.

[0190] Fig.16A 1 is a diagram showing an example "how to" answer card 1600 as described above in JSON format. In this format, there is a "steps" field that separates out each step in the process as described above.

[0191] Fig. 16B is a diagram showing an example "what is" answer card 1650 in JSON format. In this case, there is an "answer contents" field with "term" and "similar terms" as subfields, the latter being determined by querying the log.

[0192] Fig.17 1 is a flow chart illustrating an example method 1700 for performing the improved techniques described herein. The method 1700 may be performed in conjunction with Figure 1 The described software constructs are executed by resident in the memory 126 of the computer 120 and executed by a set of processing units 124.

[0193] At 1702, the computer 120 receives document data representing a collection of documents, each document in the collection of documents including a corresponding topic and content related to the topic.

[0194] At 1704 , the computer 120 generates answer suggestion look-ahead index data based on the document collection, the answer suggestion look-ahead index data representing a plurality of term / link pairs, each term / link pair of the plurality of term / link pairs including a semantically equivalent term and a corresponding link to content of the document collection.

[0195] At 1706, computer 120 receives a user query string.

[0196] At 1708 , the computer 120 locates, in response to the user query string, semantically equivalent terms of a term / link pair in the plurality of term / link pairs, the semantically equivalent terms being located based on semantic equivalence of the user query string.

[0197] At 1710 , the computer 120 outputs a linked representation of the located term / link pairs to an output device.

[0198] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0199] It should also be understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it may be directly on, connected to, or coupled to another element, or one or more intermediate elements may be present. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to another element, there are no intermediate elements. Although the terms directly on, directly connected to, or directly coupled to may not be used throughout the detailed description, an element that is shown as being directly on, directly connected to, or directly coupled to may be referred to as such. The claims of the present application may be modified to describe the exemplary relationships described in the specification or shown in the accompanying drawings.

[0200] Although certain features of the described implementations have been shown as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the implementations. It should be understood that they are presented only as examples and not limitations, and various changes in form and detail may be made. In addition to mutually exclusive combinations, any part of the apparatus and / or method described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

[0201] In addition, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be deleted from the described flows, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the following claims.< / number> < / number>

Claims

1. A computer-implemented method comprising: Receiving document data representing a document collection, each document in the document collection including a corresponding topic and content related to the topic; For each document in the document collection, generate a corresponding DOM in the document object model DOM collection, wherein the corresponding DOM corresponds to each document, and the DOM corresponding to each document includes a topic, a subtopic set, and a link to the content of each document; aggregating the DOM collection to generate a master link list for a subject knowledge graph, the master link list comprising a plurality of unique links to the content of the document collection; generating answer suggestion look-ahead index data based on the document collection, the answer suggestion look-ahead index data representing a plurality of term / link pairs, wherein the answer suggestion look-ahead index data comprises the subject knowledge graph and the master link list; Receive user query string; In response to receiving the user query string, locating semantically equivalent terms of a term / link pair in the plurality of term / link pairs, the semantically equivalent terms being located based on semantic equivalence to the user query string; as well as A linked representation of the located term / link pair is output to an output device.

2. The method of claim 1, wherein generating the answer suggestion look-ahead index data comprises: The subject knowledge graph data is generated based on the document collection, the subject knowledge graph data represents a subject knowledge graph, and the subject knowledge graph includes: (i) the corresponding topic and the sub-topic set of each document in the document collection, and (ii) the links to the content of the document collection, each of the links corresponding to the corresponding topic of each document in the document collection or one of the sub-topics in the sub-topic set.

3. The method according to claim 2, wherein generating the answer suggestion look-ahead index data further comprises: generating semantic annotation data representing respective semantic annotations of the topic and the subtopic set for each document in the document set, each of the semantic annotations comprising a corresponding predicate in the predicate set and a corresponding object in the object set; for each predicate in the set of predicates, identifying at least one object in the set of objects that, when combined with the predicate, corresponds to a topic of each document in the set of documents or one of the subtopics in the set of subtopics; as well as For each object in the set of objects, at least one predicate in the set of predicates is identified that, when combined with the object, corresponds to a topic of each document in the set of documents or one of subtopics in the set of subtopics.

4. The method according to claim 2, wherein generating the subject knowledge graph data comprises: The DOM collection is formatted to produce an aggregated flattened knowledge graph formatted for a natural language processing (NLP) pipeline.

5. The method of claim 4, wherein the aggregated flattened knowledge graph comprises at least one of the topics and subtopics of each DOM in the DOM set, and the NLP pipeline is configured to generate<S,P,O> The triplet,<S,P,O> A triple includes a subject, a predicate, and an object for each of the set of topics and subtopics for each DOM in the set of DOMs.

6. The method according to claim 4, wherein generating the subject knowledge graph data further comprises: splitting the aggregated flattened knowledge graph to generate a plurality of aggregated flattened knowledge graph portions; as well as Natural language processing is performed on each of the multiple aggregated flattened knowledge graph parts through the NLP pipeline to generate the knowledge graph, and the natural language processing is performed in parallel on each of the multiple aggregated flattened knowledge graph parts.

7. The method of claim 4, wherein generating the DOM in the DOM set comprises: identifying non-information content of each document in the document collection; as well as The non-information content of the document is removed to generate the topics, the sets of sub-topics, and the links to the content of the document, the removing comprising applying a term frequency-inverse document frequency (TF-IDF) algorithm to the document.

8. The method of claim 4, wherein generating the DOM in the DOM set comprises: Each document in the document collection is reformatted to generate a document formatted in the Markdown markup language MDML.

9. A computer program product comprising a non-transitory storage medium, the computer program product comprising code which, when executed by a processing circuit of a computer configured to direct a user to content based on a semantic interpretation of a query entered by the user, causes the processing circuit to perform a method comprising: Receiving document data representing a document collection, each document in the document collection including a corresponding topic and content related to the topic; Obtaining search query log data, the search query log data representing a mapping between user query data and links to content of the document collection, the user query data representing a plurality of user queries; performing a text mining operation on the search query log data to generate a set of common user queries targeting a topic of a document in the document collection or one of a corresponding subtopic in a set of subtopics, each of the topic and the set of subtopics being associated with a corresponding content corresponding to a corresponding link; generating answer suggestion look-ahead index data based on the document collection, the answer suggestion look-ahead index data representing a plurality of term / link pairs, each term / link pair of the plurality of term / link pairs comprising a link to content of the document collection, wherein the answer suggestion look-ahead index data is generated based at least in part on a set of common user queries obtained using search query log data; Receive user query string; In response to receiving the user query string, locating semantically equivalent terms of a term / link pair in the plurality of term / link pairs, the semantically equivalent terms being located based on semantic equivalence to the user query string; as well as A linked representation of the located term / link pair is output to an output device.

10. The computer program product of claim 9, wherein generating the answer suggestion look-ahead index data comprises: Generate subject knowledge graph data based on the document collection, the subject knowledge graph data represents a subject knowledge graph, the subject knowledge graph comprising: (i) a corresponding subject and a sub-topic set for each document in the document collection, and (ii) links to the content of the document collection, each of the links corresponding to a corresponding subject of each document in the document collection or one of the sub-topics in the sub-topic set.

11. The computer program product of claim 10, wherein generating the answer suggestion look-ahead index data further comprises: Form pairs of: (i) a corresponding user query in the public user query set, and (ii) a corresponding link to the content of the document collection, each pair based on a set of topics and subtopics of an annotation of the topic knowledge graph, the set of topics and subtopics of the annotation comprising a topic title and<S,P,O> The triplet,<S,P,O> A triple includes a subject, a predicate, and an object for each of the topic and the set of subtopics, and Generating a link to specific content in the document collection includes: A pair of a user query and a link to the content is identified that is a semantic best match of the Predicate and the Object to the Predicate and the Object of the received user query string.

12. The computer program product of claim 11, wherein forming the pair of a user query and a link to the content comprises: generating a respective weight corresponding to each of the topic title, subject, predicate, and text content of the topic title corresponding to the pair of the user query and the link to the content, and wherein identifying the pair of the user query and the link to the content as a semantically best match to the predicate and object of the received user query string comprises: A linear combination of the topic title, the subject, the predicate and the text content of the common user query set is generated using the respective weights corresponding to each of the topic title, the subject, the predicate and the text content, the linear combination most closely matching the same linear combination of the topic title, subject, predicate and text content corresponding to the user query string.

13. The computer program product of claim 12, wherein generating the respective weights corresponding to each of the topic title, the subject, the predicate, and the text content of the topic title corresponding to the pair of the user query and the link to the content comprises: generating a matrix having a plurality of rows, each of the plurality of rows having elements equal to respective weights of the topic title, the subject, the predicate, and the text content; forming, for each of said pairs of user queries of said common user query set and links to said content, a merit function with as arguments: (i) a search function of said user query data, said links to said content of said document set and a row of said matrix, and (ii) a particular link to said content of said document set, said search function returning a link as an output, said merit function having a value of 1 in response to said link output of said search function being equal to said particular link and having a value of 0 in response to said link output of said search function being not equal to said particular link; and The row of the matrix that makes the merit function present a maximum value is identified as the weight corresponding to each of the topic title, the subject, the predicate, and the text content.

14. An electronic device configured to direct a user to content based on a semantic interpretation of a query input by the user, the electronic device comprising: Network interface; Memory; as well as a control circuit coupled to the memory, the control circuit being configured to: Receiving document data representing a document collection, each document in the document collection including a corresponding topic and content related to the topic; generating answer suggestion lookahead index data based on the document collection, the answer suggestion lookahead index data representing a plurality of term / link pairs, and wherein the answer suggestion lookahead index data comprises a topic knowledge graph comprising, for one or more topics, links to content corresponding to other topics; Receive user query string; In response to receiving the user query string, locating semantically equivalent terms of a term / link pair in the plurality of term / link pairs, the semantically equivalent terms being located based on semantic equivalence to the user query string; as well as outputting to an output device a representation of the links of the located term / link pairs; as well as An answer card corresponding to the link of the located term / link pair is displayed, the answer card including a window having the content corresponding to the link contained therein.

15. The electronic device of claim 14, wherein the control circuit configured to generate the answer suggestion look-ahead index data is further configured to: Generate subject knowledge graph data based on the document collection, wherein the subject knowledge graph data represents the subject knowledge graph, and the subject knowledge graph includes: (i) the corresponding topics and sub-topic sets for each document in the document collection, and (ii) the links, each of which corresponds to the corresponding topic of each document in the document collection or one of the sub-topics in the sub-topic set.

16. The electronic device of claim 14, wherein the control circuit configured to locate the semantically equivalent terms of the term / link pairs in the plurality of term / link pairs is further configured to: locating content including text indicating steps of a process; and The subject to which the content corresponds is identified.

Images