Techniques for generating multimodal discourse trees

Abstract

A method for generating and utilizing a multimodal discourse tree (MMDT) is provided herein. An extended discourse tree (EDT) may be generated from a corpus of text (e.g., from a discourse tree (DT) or a communication DT (CDT)). Data records (e.g., records containing numerical data) may be linked to the extended discourse tree to generate a multimodal discourse tree. The multimodal discourse tree may link any suitable text / records from heterogeneous sources. For example, entities identified from basic discourse units of the EDT may be matched with entities of the data records. Causal links between EDTs and / or data records may be identified. Rhetorical relationships may be identified for each entity match / causal link match to coalesce the data records with the EDT to generate an MMDT. The MMDT may be used to classify subsequent inputs, generate answers to subsequent questions, navigate the corpus of text and / or data records, etc.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 166,568, filed March 26, 2021, entitled "Multimodal Discourse Tree for Health Management and Security," the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] Technical Field FIELD OF THE DISCLOSURE This disclosure relates generally to linguistics and, more specifically, to generating and utilizing Multi-Modal Discourse Trees (MMDTs) (e.g., for classifying input text, generating answers to questions, navigating text, etc.). [Background technology]

[0003] background The vast increase in processor speed and memory capacity has led to a proliferation of computer-implemented linguistic applications. For example, computer-based analysis of linguistic discourse facilitates numerous applications, such as automated agents that can answer questions received from a user device, or machine learning applications that can classify input. In these and other contexts, current systems may provide text data and potentially related numerical data (e.g., logs, maps, data records, etc.) separately. In some cases, it may be beneficial to utilize both text and various potentially related numerical data to classify similar data and / or answer subsequent related questions. However, such data is difficult to combine in a meaningful way. Summary of the Invention [Means for solving the problem]

[0004] Quick Overview Techniques are disclosed for creating and utilizing multi-modal discourse trees (MMDTs) (e.g., for classifying input text, generating answers to questions, navigating text, etc.).

[0005] In some embodiments, a computer-implemented method for generating a multimodal discourse tree (e.g., for categorizing input, navigating text, identifying answers to a subsequent question, etc.) is disclosed. The method may include obtaining a corpus of text and one or more data records separate from the corpus of text. The method may further include generating an extended discourse tree for the corpus of text. In some embodiments, the extended discourse tree includes a plurality of discourse trees. Each discourse tree may include a plurality of nodes, each terminal node of the discourse tree corresponding to a fragment of text, and each non-terminal node of the discourse tree indicating a rhetorical relationship between the nodes of the discourse tree. In some embodiments, the extended discourse tree includes further links between the plurality of discourse trees indicating further rhetorical relationships between the nodes of the respective discourse trees. The method may further include identifying entity matches between a set of base discourse units of the plurality of discourse trees and the one or more data records. In some embodiments, the entity match may be identified by comparing a first entity identified from a basic discourse unit and a second entity identified from a data record. The method may further include executing a causal link identification algorithm to identify one or more causal links between two data records of the one or more data records. The method may further include determining a corresponding rhetorical relationship for each identified entity match and each of the one or more identified causal links. The method may further include generating a respective node for the expanded discourse tree for each identified entity match and each identified causal link. The method may further include creating a multimodal discourse tree by linking the respective node generated for each entity match and each causal link to a respective node of the expanded discourse tree based at least in part on the determined corresponding rhetorical relationship.

[0006] In some embodiments, generating the extended discourse comprises: i) generating a first discourse tree from a first text of the corpus of text, the first discourse tree corresponding to a first portion of the first text; generating the extended discourse further comprises: ii) generating a second discourse tree from a second text of the corpus of text, the second discourse tree corresponding to a second portion of the second text; generating the extended discourse further comprises: iii) in response to determining a particular rhetorical relationship between respective basic discourse units of the first discourse tree and the second discourse tree, linking the first discourse tree and the second discourse tree using the particular rhetorical relationship.

[0007] In some embodiments, the first discourse tree and the second discourse tree are communicative discourse trees that include respective verb signatures generated for each basic discourse unit of the first discourse tree and the second discourse tree.

[0008] In some embodiments, identifying the entity further comprises comparing the second entity identified from the data record with a third entity identified from a second data record, the entity referring to one of: (i) a person, (ii) a business, (iii) a location, (iv) a name of a document, or (v) a date or time.

[0009] In some embodiments, identifying the entity match further comprises identifying the entity from a predefined ontology.

[0010] In some embodiments, the method may further include classifying a subsequent input based at least in part on the multimodal discourse tree, where classifying the subsequent input includes: i) generating a training dataset including a plurality of multimodal discourse trees, where each multimodal discourse tree corresponds to a respective text corpus and a respective set of data records, where each multimodal discourse tree is associated with a label corresponding to a classification; and classifying the subsequent input further includes: ii) training a machine learning model to classify an input based at least in part on the training dataset and a supervised learning algorithm; and iii) generating a corresponding multimodal discourse tree from the subsequent input, where the subsequent input includes a respective set of text and a respective set of data records; and classifying the subsequent input further includes iv) classifying the subsequent input based at least in part on providing the corresponding multimodal discourse generated from the subsequent input as an input to the machine learning model and receiving an output from the machine learning model indicative of a classification of the subsequent input.

[0011] In some embodiments, the method further includes navigating the corpus of text using the multimodal discourse tree, where navigating the corpus of text includes accessing the multimodal discourse tree and determining from the multimodal discourse tree a first basic discourse unit responsive to a query from a user device, the first basic discourse unit corresponding to a first node of a first discourse tree of the multimodal discourse tree, and navigating the corpus of text further includes determining from the multimodal discourse tree a set of navigation options, the set of navigation options being based on (i) a first rhetorical relationship between the first node of the first discourse tree and a second node of the first discourse tree and (ii) a second rhetorical relationship between the first node and a third node of a second discourse tree of the multimodal discourse tree, or (iii) a second rhetorical relationship between the first node and a third node of a second discourse tree of the multimodal discourse tree. and a third rhetorical relationship between the first node of the multimodal discourse tree and a fourth node of the multimodal discourse tree associated with a corresponding data record, and navigating the corpus of text further includes presenting at least two of the first, second or third rhetorical relationships on the user device, and in response to receiving further user input including a selection of the first rhetorical relationship, the second rhetorical relationship or the third rhetorical relationship, i) presenting a second basic discourse unit corresponding to the second node based at least in part on determining the selection corresponding to the first rhetorical relationship, ii) presenting a third basic discourse unit corresponding to the third node based at least in part on determining the selection corresponding to the first rhetorical relationship, or iii) presenting at least a portion of the corresponding data record based at least in part on determining the selection corresponding to the third rhetorical relationship.

[0012] The example methods discussed herein may be implemented on a system and / or device including one or more processors and / or may be stored as instructions on a non-transitory computer-readable medium. Various aspects of the present disclosure may be implemented using a computer program product that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an example of a computing environment for classifying input using a Multi-Modal Discourse Tree (MMDT), in accordance with at least one embodiment of the present disclosure. [Diagram 2] 2 illustrates an example flow of a method performed by the computing device of FIG. 1 (e.g., by the multimodal discourse tree (MMDT) generation module of FIG. 1) in accordance with at least one embodiment. [Diagram 3] FIG. 2 illustrates an exemplary discourse tree according to at least one embodiment. [Figure 4] FIG. 2 illustrates a flow diagram of a method for generating a discourse tree for communication according to at least one embodiment. [Diagram 5] FIG. 2 illustrates an example of an extended discourse tree according to at least one embodiment. [Figure 6] 1 is a flowchart of an example process for creating an extended discourse tree according to at least one embodiment. [Figure 7] FIG. 2 illustrates relationships between text units of a document at various levels of granularity, according to at least one embodiment. [Figure 8] FIG. 2 illustrates an example of a multimodal discourse tree according to at least one embodiment. [Figure 9]FIG. 9 is another diagram illustrating the multimodal discourse tree of FIG. 8 along with relationships between text units of a document at various levels of granularity and between those text units and their associated data records, according to at least one embodiment. [Figure 10] FIG. 2 illustrates a flow diagram of a method for generating a multimodal discourse tree according to at least one embodiment. [Figure 11] 1 is a simplified diagram of a distributed system for implementing one of the aspects according to at least one embodiment. [Figure 12] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an aspect of a system may be provided as a cloud service, in accordance with at least one embodiment of an aspect of the present disclosure. [Figure 13] FIG. 1 illustrates an example computing subsystem in which various aspects may be implemented in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Detailed Description Aspects of the present disclosure relate to generating and utilizing a multimodal discourse tree (MMDT) to classify input. A multimodal discourse tree (MMDT) refers to a discourse tree that includes associated data (values, sets, and / or records) associated with parts of the discourse tree (e.g., basic discourse units). The MMDT includes relationships between data sets and discourse trees and relationships between data in a given data set. Examples of rhetorical relationships (also referred to herein as "rhetorical relationships") that may exist between data values, sets, and records include reason, cause, enable, contrast, and temporal sequence.

[0015] Discourse analysis plays an important role in constructing the logical structure of ideas expressed in a text. Discourse trees are a means to hierarchically formalize a textual discourse specifying rhetorical relations between phrases and sentences (discourse units). Discourse trees (DTs) are a compromise between a complete logical representation such as a logical form and an informal representation of the original text. Learning DTs has found numerous applications in content generation, summarization, machine translation and question answering. A limitation of the use of DTs in general data analysis tasks is that they are designed to represent textual discourse and not causal relations between constituents of abstract data items.

[0016] The techniques disclosed herein utilize discourse analysis to generate various discourse trees from a corpus of text. For example, a discourse tree may be generated for each text in a corpus of documents (e.g., a corpus of related / associated content). In some embodiments, a discourse tree (DT) may be extended with communicative actions (e.g., verb signatures) to generate a communicative discourse tree (CDT). Rhetorical relationships between units of the CDT (e.g., basic discourse units) may be identified to generate an extended discourse tree (EDT). An EDT may include a combination of CDTs with relationships identified between various units of the CDT and across various granularities (e.g., between sentences of a document, between paragraphs of a document, between sentences / paragraphs of different documents, and / or between entire documents).

[0017] A multimodal discourse tree (MMDT) can be generated from an EDT by combining the EDT with the associated numerical data through discourse abstraction to formalize discourse across various types of numerical data as well as texts and text documents of the corpus. MMDTs can be generated such that the same rhetorical relationships that hold between text fragments also hold between data values, sets, and records. MMDTs can be discourse trees that include associated data (values, sets, and / or numerical data records) that are related to parts of the EDT (e.g., basic discourse units). MMDTs can be generated to include relationships between the associated data and between parts of the associated data and parts of the corpus text. The resulting MMDT can then be used to drive automated answers in the field of autonomous agents and / or classify inputs. In one example, a discourse tree may be utilized that represents the inference structure required to reach a conclusion in a legal case. From this discourse tree, an extended discourse tree can be generated, which can be extended with related information (e.g., data on specific incidents) such as telephone calls, information on people's movements, trade / financial records, and internal relationships between information.

[0018] The disclosed technology improves on existing solutions by providing a better understanding across multiple related information sources. MMDT can be used to extend the capabilities of conversational chatbots (e.g., autonomous agent applications). Traditional discourse trees represent the flow of an author's thoughts at the level of a paragraph or multiple paragraphs. Traditional discourse trees become quite inaccurate when applied to larger text fragments or documents. These discourse trees can be used to generate extended discourse trees and then multimodal discourse trees that can serve as a representation of a set of interconnected documents and corresponding data records covering a topic.

[0019] Specific Definitions "Rhetorical structure theory", as used herein, is a field of investigation and research that has provided a theoretical foundation that may enable the analysis of discourse coherence.

[0020] "Discourse Tree" or "DT" as used herein refers to a structure that expresses rhetorical relations about sentences that are part of a sentence.

[0021] "Rhetorical relation," "rhetorical relationship," "coherence relation," or "discourse relation," as used herein, describes how two segments of discourse are logically connected to each other. Examples of rhetorical relations include elaboration, contrast, and attribution.

[0022] A "sentence fragment" or "fragment" as used herein is a part of a sentence that can be separated from the rest of the sentence. A fragment is a basic discourse unit. For example, in the sentence "Dutch accident investigators say that evidence points to pro-Russian rebels as being responsible for shooting down the plane," the two fragments are "Dutch accident investigators say that evidence points to pro-Russian rebels" and "as being responsible for shooting down the plane." A fragment may contain a verb, but does not necessarily have to contain a verb.

[0023] "Signature" or "frame" as used herein refers to the properties of verbs in a fragment. Each signature may include one or more thematic roles. For example, in the fragment "Dutch accident investigators say that evidence points to pro-Russian rebels", the verb is "say", and the signature for this particular use of the verb "say" may be an "agent-verb-topic". In this case, "investigators" are the agents and "evidence" is the topic.

[0024] "Thematic role," as used herein, refers to a component of a signature that is used to describe the role of one or more words. Continuing with the example above, "agent" and "topic" are thematic roles.

[0025] "Nuclearity," as used herein, refers to which text segment, fragment, or span is closer to the center of the writer's intent. A nucleus is a more central span, a satellite is less central.

[0026] "Coherency," as used herein, refers to that which links two rhetorical relations together.

[0027] A "Communicative Discourse Tree" or "CDT" as used herein is a discourse tree supplemented with communicative actions. Communicative actions are collaborative actions taken by individuals based on their mutual deliberation and discussion. Thus, a Communicative Discourse Tree combines rhetorical information with communicative actions.

[0028] A "communicative verb," ​​as used herein, is a verb that indicates communication. For example, the verb "deny" is a communicative verb.

[0029] "Communicative behavior," as used herein, describes an action performed by one or more agents and the subject of the agents.

[0030] An "entity," as used in this specification, has a separate and distinct existence. Examples include objects, places, and people. An entity can also be a subject or topic, such as "electric cars," "brakes," or "France."

[0031] Turning to the drawings, FIG. 1 illustrates an example of a computing environment for correcting raw text generated using deep learning techniques, according to at least one embodiment. In the example shown in FIG. 1, the computing environment 100 includes one or more of a computing device 102 and a user device 104. The computing device 102 may implement an application (e.g., application 106). In some embodiments, the application 106 may be an autonomous agent (e.g., a chatbot) that engages in a conversation with the user device 104 and interacts in response to input provided by the user device 104 using one or more of the techniques disclosed herein. As another example, the application 106 may be part of a machine learning component of the computing device 102. The application 106 may be configured to generate any suitable number of multimodal discourse trees (MMDTs) from text data (e.g., a corpus of text data obtained from the text data 116) and data records (e.g., numerical data obtained from the record data store 118). An example of the computing device 102 is an example of the distributed system 1000 and client computing devices 1002, 1004, 1006, and 1008 of FIG.

[0032] The user device 104 may be any mobile device, such as a mobile phone, smartphone, tablet, laptop, smartwatch, etc. As shown, the user device 104 includes a user interface 108. The user interface 108 may be configured to accept input from a user (e.g., via a keyboard, microphone input, mouse input, touch screen, etc.) and provide data corresponding to the input to the computing device 102. In some embodiments, the user interface 108, or another component of the user device 104, may be configured to capture voice input and convert the voice input to text before transmitting to the computing device 102. The user device 104 may be configured to transmit the text to the computing device 102. In other embodiments, the text may be obtained using a user interface (not shown) provided by the computing device 102 and / or from a data store (not shown) accessible to the computing device 102. Examples of suitable text include electronic text sources, such as text files, Portable Document Format (PDF) documents, rich text documents, etc. In some cases, pre-processing may be performed on the input text to remove unnecessary characters or formatting fields. The input text may be organized using one or more structural or organizational approaches, such as sections, paragraphs, pages, etc.

[0033] In some embodiments, the user device 104 and the computing device 102 may be communicatively connected via a network 110. The network 110 may be any suitable public or private network, including the Internet, a local area network, a virtual private network, or the like.

[0034] In some embodiments, the computing device 102 may include a classifier 112. The classifier 112 may be any suitable machine learning model trained using training data 114 to provide an output (e.g., an answer, generated text, etc.) in response to an input (e.g., a question submitted in the user interface 108 or other data). The training data 114 may include any suitable data (e.g., question / answer pairs with good / bad answers, examples of good and / or bad text generation, etc.) capable of training the classifier 112. In some cases, entities in the text may be matched using an ontology 117. The ontology 117 may be a domain-specific ontology (e.g., finance, law, business, medicine, science, etc.). The ontology 117 may include, among other features, a formal specification of various entities and their relationships. The ontology 117 may be utilized to identify synonymous words and / or phrases. In some embodiments, the ontology 117 may be predefined and / or the application 106 may construct at least a portion of the ontology 117 from an external source.

[0035] The MMDT generation module 120 may be configured to generate any suitable number of MMDTs from the text data 116 and the record data store 118. In some embodiments, these MMDTs may serve as training data for classifying future inputs. As an example, text in a police record (e.g., an example of text data) may be merged with numerically based evidence (e.g., call records, financial transaction data, location data obtained from a cell phone, etc.) to generate an MMDT related to data associated with a particular crime case. This MMDT may be associated with a label (e.g., blackmail) and used as labeled training data for the classifier 112 (e.g., an example of a machine learning model). The classifier 112 may be trained using various examples of such training data (e.g., some examples showing text / data records indicative of blackmail crimes, some examples showing other text / data records not indicative of blackmail crimes, etc.) along with any suitable supervised learning algorithm to configure the classifier 112 to classify subsequent inputs (e.g., text and data records received from the user device 104 and associated with the case) as indicative of a blackmail crime or as not indicative of a blackmail crime. In some embodiments, the generated MMDT can be used to answer questions about the incident for which the MMDT was generated. Operations for generating and utilizing an MMDT are discussed in further detail with respect to Figures 2-9.

[0036] FIG. 2 illustrates an example flow diagram of a method 200 performed by the computing device 102 (eg, by the MMDT generation module 120) of FIG. 1 in accordance with at least one embodiment.

[0037] Method 200 may begin at 201, where a corpus of documents may be obtained. In some embodiments, the corpus of documents may relate to a particular topic. As an example, the corpus of documents may include any suitable text from police records relating to a particular crime. The documents may have any suitable length and / or format.

[0038] At 202, a discourse tree may be generated for each text in the corpus based at least in part on rhetorical structure theory. Techniques for generating these discourse trees are discussed below in connection with FIG.

[0039] 3 is a diagram illustrating a text encoding 300 of a discourse tree (referred to as a "discourse tree 300") generated from an instance of text, according to at least one embodiment. By way of example only, discourse tree 300 may be generated from the following text: "I avoid flu shots because I am allergic to eggs. Most flu shots produced today use an egg-based manufacturing process that leaves trace amounts of egg protein behind. I do not need a flu shot since I got it last year. I believe it is not necessary for me, because the vaccine is not 100% effective. Also, I never get the flu, so I do not need a vaccine." A discourse tree 300 may be generated from the above input text based at least in part on rhetorical structure theory.

[0040] Rhetorical Structure Theory and Discourse Trees Linguistics is the scientific study of language. For example, linguistics can include the structure of a sentence (syntax), e.g. subject-verb-object, the meaning of a sentence (semantics), e.g. "dog bites man" vs. "man bites dog", and also what speakers do in conversation, i.e. discourse analysis or the analysis of language beyond the scope of the sentence.

[0041] The theoretical foundation of discourse (Rhetorical Structure Theory (RST)) can be attributed to Mann, William and Thompson, Sandra, "Rhetorical structure theory: A Theory of Text organization" (Text-Interdisciplinary Journal for the Study of Discourse) 8(3):243-281: 1988). RST helped to enable the analysis of discourse similar to how syntax and semantics of programming language theory helped to enable modern software compilers. More specifically, RST envisages structural blocks at at least two levels. The two levels include a first level such as kernelity and rhetorical relations, and a second level of structure or schema. Discourse parsers or other computer software can parse (parse) text into discourse trees. These parsers / software can be configured to identify various rhetorical relations between EDUs in the discourse tree.

[0042] Rhetorical structure theory models the logical organization of a text (the structure used by the writer) depending on the relationships between parts of the text. RST simulates text coherence by forming a hierarchical, connected structure of the text through a discourse tree. Rhetorical relations are divided into equivalence and subclasses. These relations are maintained across two or more text spans, thus achieving coherence. These text spans are called elementary discourse units (EDUs). Clauses in a sentence and sentences in a text are logically connected by the author. The meaning of a given sentence is related to the meaning of the previous sentence and the next sentence. This logical relationship between clauses is called the coherence structure of the text. RST is one of several theories of discourse based on a tree-like discourse structure, the discourse tree (DT). The leaves of the DT correspond to EDUs (successive atomic text spans). Adjacent EDUs are connected by coherence relations (e.g., attributes, sequences) that form higher-level discourse units. These units are further subordinated to this relational link. EDUs linked by a relationship are further differentiated based on their relative importance. The core is the core part of the relationship, while the satellites are the periphery. As mentioned above, to determine the correct request-answer pair, both topic and rhetorical match are analyzed. When a speaker answers a question with a phrase or sentence, the speaker's response must address the topic of this question. If the question is formulated implicitly, via the seed text of the message, an appropriate response is expected that not only stays on topic but also matches the generalized epistemic state about this seed.

[0043] Rhetorical Relationships As mentioned above, the aspects described herein use discourse trees (and / or communicative discourse trees) that include rhetorical relations. Rhetorical relations can be described in a variety of ways. Some rhetorical relations are described below. However, this list is not intended to be exhaustive.

[0044] [Table 1]

[0045] Some empirical studies have assumed that the majority of texts are constructed using core-satellite relationships, whereas other relationships do not involve a finite selection of nuclei. Examples of such relationships are given below.

[0046] [Table 2]

[0047] Returning to FIG. 3 , discourse tree 300 may be generated based on syntactic analysis (e.g., using a discourse parser and / or software) of the input text. Discourse tree 300 may include any suitable combination of the above rhetorical relations (also referred to as “rhetorical relations”). For example, discourse tree 300 may include rhetorical relations 302-322. Discourse tree 300 may further include several text units (e.g., basic discourse units (EDUs) 324-346). Each rhetorical relation represents a rhetorical relationship between two or more EDUs. As an example, rhetorical relation 314 may represent a rhetorical relationship (e.g., “explanation”) between EDU 334 and EDU 336.

[0048] Any suitable method may be utilized for constructing the discourse tree 300. One exemplary method for constructing the discourse tree 300 may include the following acts.

[0049] (1) Divide the discourse text into multiple units by (a) and (b) below. (a) Unit size may vary depending on the purpose of the analysis.

[0050] (b) Typically the unit is a clause. (2) Examine each unit and each of its neighboring units: Do relationships hold between them (e.g., identified at least in part based on a set of predefined rules)? (3) If the relationship is maintained, mark the relationship.

[0051] (4) If a relationship does not hold, the unit may be on the boundary of a higher level relationship. Look at the relationships that hold between larger units (spans).

[0052] (5) Continue until all units in the text have been understood. In some embodiments, the MMDT generation module 120 may generate a discourse tree for each text in the corpus. There may be discourse trees generated at various degrees of granularity. For example, a DT may be generated for each sentence, for each paragraph, for each document, etc. Each discourse tree may include nodes, where each non-terminal node (or edge) represents a rhetorical relationship between two of the fragments (e.g., sentence fragments), and each terminal node of the discourse tree is associated with one of the fragments.

[0053] Returning to Figure 2, once a DT has been generated for the texts of the corpus (e.g., police reports and / or textual information related to crime cases) (according to any suitable granularity), method 200 may proceed to 203, where a Communicative Discourse Tree (CDT) may be generated for each text in the corpus. Figure 4 describes a technique for generating a CDT from text in more detail.

[0054] 4 is a flow diagram of a method 400 for generating a communicative discourse tree according to at least one embodiment. In some embodiments, utilizing a communicative discourse tree to generate queries enables improved search engine results.

[0055] At block 401, process 400 includes accessing sentences (e.g., sentences from the corpus obtained at 201 of FIG. 2) that include fragments. At least one fragment includes a verb and one or more words, each word including a role of the word in the fragment, and each fragment is a basic discourse unit.

[0056] In some embodiments, the MMDT generation module 120 of FIG. 1 may identify that a sentence includes several fragments, each of which may include a verb, although a fragment may not include a verb.

[0057] At block 402, the process 400 includes generating a discourse tree that represents a rhetorical relationship between the sentence fragments. For example, the MMDT generation module 120 may generate one or more discourse trees as discussed in connection with Figure 3. Each discourse tree may include nodes, where each non-terminal node represents a rhetorical relationship between two of the sentence fragments, and each terminal node of the discourse tree's nodes is associated with one of the sentence fragments.

[0058] At block 403, the process 400 includes accessing a plurality of verb signatures. For example, the MMDT generation module 120 may access a list of verbs (e.g., from VerbNet, a list of predefined verbs, etc.). Each verb may match or be related to a verb of the fragment. For example, if the fragment included the verb "deny," the MMDT generation module 120 may access a list of verb signatures related to the verb "deny."

[0059] Each verb signature may include the verb of the fragment and one or more of the thematic roles. For example, the signature may include one or more of a noun phrase (NP), a noun (N), a communicative action (V), a verb phrase (VP), or an adverb (ADV). The thematic role describes the relationship between the verb and the associated words. For example, "the teacher amused the children" has a different signature than "small children amuse quickly." For the first fragment, the verb "deny," the application 102 accesses a list of frames or verb signatures for verbs that match "deny." The list is "NP V NP to be NP," "NP V that S," and "NP V NP."

[0060] Each verb signature includes a thematic role. The thematic role refers to the role of the verb in the sentence fragment. In some embodiments, the MMDT generation module 120 determines the thematic role in each verb signature. Exemplary thematic roles include "actor," "agent," "asset," "attribute," "beneficiary," "cause," "location destination source," "destination," "source," "location," "experiencer," "extent," "instrument," "material and product," "material," "product," "patient," "predicate," "recipient," "stimulus," "theme," "time," or "topic."

[0061] At block 404, the process 400 includes determining, for each verb signature of the verb signatures, a number of thematic roles of the respective signatures that match the roles of the words in the fragment. For example, if the fragment of the raw text sentence contains the verb "deny", the MMDT generation module 120 may determine that the verb "deny" has only three roles: "agent", "verb", and "theme".

[0062] At block 405, the process 400 includes selecting a particular verb signature from the verb signatures based on the particular verb signature with the highest number of matches. For example, the fragment "the rebels deny... that they control the territory" may be matched with the verb signature "deny" "NP V NP", where "control" is matched with control (rebel, territory). The verb signatures are nested, resulting in a nested signature of "deny (rebel, control (rebel, territory))". Each selected verb signature is associated with a corresponding fragment to complete the communicative discourse tree.

[0063] Returning to 204 in Figure 2, once a Communicative Discourse Tree (CDT) has been generated for a corpus of text in the manner described above in connection with Figures 3 and 4, the MMDT generation module 120 can construct an Augmented Discourse Tree (EDT) for the text corpus. Figures 5 and 6 describe in more detail a technique for generating an EDT for a text corpus.

[0064] FIG. 5 illustrates an example of an extended discourse tree, according to at least one embodiment. FIG. 5 illustrates an extended discourse tree 500. As shown, extended discourse tree 500 includes groups 500, 520, 530, 540, and 550. Each group includes a document (e.g., from a text corpus) and a discourse tree generated from the document. For example, group 510 includes discourse tree 511 and document 512, group 520 includes discourse tree 521 and document 522, etc. The discourse tree illustrated in FIG. 5 may be an example of a DT generated as described above in FIG. 2 at 202, and / or the discourse tree of FIG. 5 may be an example of a CDT generated as described above in FIG. 2 at 203.

[0065] In addition to links within a particular discourse tree, such as discourse trees 511, 521, 531, 541, and 551, the extended discourse tree 500 includes inter-discourse tree links 561-564 and associated inter-document links 571-574. As will be further described with respect to FIG. 6, the MMDT generation module 120 may construct discourse trees 511-515. Discourse tree 511 represents document 512, discourse tree 521 represents document 522, and so on. The extended discourse tree 500 may be constructed by building a discourse tree (e.g., DT and / or CDT) for each paragraph or document (e.g., using process 400 for each tree).

[0066] Inter-discourse tree link 561 connects discourse trees 511 and 521, inter-discourse tree link 562 connects discourse trees 521 and 531, inter-discourse tree link 563 connects discourse trees 511 and 541, and inter-discourse tree link 564 connects discourse trees 521 and 551. Based on the inter-discourse tree links 561-564, the MMDT generation module 120 creates inter-document links 571, 572, 573, and 574 corresponding to the inter-discourse tree links 561, 562, 563, and 564, respectively. The inter-document links 571-574 can be used to navigate documents 512, 522, 532, 542, and 552.

[0067] The MMDT generation module 120 determines one or more entities in a first discourse tree of the discourse trees 511-515. Examples of entities include places, things, people, or businesses. The MMDT generation module 120 then identifies identical entities that exist in other discourse trees. Based on the determined entities, the MMDT generation module 120 determines rhetorical relationships between each matching entity. These determinations utilize the same discourse rules as those used to generate the discourse tree.

[0068] For example, if an entity "San Francisco" appears in document 512, such as "San Francisco is in California," and document 522 further describes "San Francisco has a moderate climate but can be quite windy," the MMDT generation module 120 may determine that the rhetorical relationship between the entities "San Francisco" is one of "elaboration," and mark links 561 and 571 as "elaboration." In some embodiments, the MMDT generation module 120 may utilize a discourse parser to parse a combination of EDUs (e.g., "San Francisco is in California" from document 512 and "San Francisco has a moderate climate but can be quite windy" from document 522) to identify that one is an elaboration of the other. In some embodiments, this relationship may be unidirectional, or multiple rhetorical relationships may be used to represent a bidirectional relationship (e.g., to indicate that each of the two EDUs is an elaboration of the other). Continuing with the above example, the MMDT generation module 120 determines links 562-564 and corresponding links 572-574 based on the determined rhetorical relations. The MMDT generation module 120 combines the discourse trees of the paragraphs of the document to form an extended discourse tree 500.

[0069] By using the links in the extended discourse tree 500, the MMDT generation module 120 can navigate between paragraphs of the same document or between documents, such as documents 512 and 522. For example, if a user is interested in more information about a particular topic, the MMDT generation module 120 can navigate via a nucleus-to-satellite elaboration rhetorical relation in a paragraph or an elaboration rhetorical relation hyperlink to a document that provides more specific information about the topic.

[0070] Conversely, if the user determines that the suggested topic is not necessarily the one required, the user can return to a higher level view of the document (e.g., from satellite to core, or from narrow document to broad document).The MMDT generation module 120 can then navigate the elaboration relationships in reverse order, i.e., from satellite to core, in paragraphs or across documents.Similarly, the MMDT generation module 120 can facilitate other navigation options, such as relying on contrast or conditional rhetorical relationships to explore controversial topics.

[0071] To build rhetorical links between text fragments in different paragraphs or documents, the MMDT generation module 120 may identify relationships between entities from each text fragment of the original paragraph by using fictional text fragments or temporary paragraphs, and perform coreference analysis and discourse syntactic analysis on the paragraphs. In some embodiments, the MMDT generation module 120 may utilize an ontology (e.g., ontology 117 in FIG. 1 ) to identify relationships between entities based on relationships provided in the ontology.

[0072] 6 is a flow chart of an example process 600 for creating an extended discourse tree (e.g., extended discourse tree 500 of FIG. 5) according to at least one embodiment. The input of process 600 is a set of documents, and the output is an extended discourse tree encoded as a regular discourse tree with document identification labels for each node. For illustrative purposes, process 600 is described with respect to two documents, e.g., documents 110a-110b (e.g., the example corpus text obtained in 201 of FIG. 2), although process 600 can use any number of documents.

[0073] At block 601, process 600 includes accessing a first document and a second document. Examples of documents include texts, books, news articles, and other electronic text documents. In an example provided in the law enforcement field, the document may be any suitable text document corresponding to an alleged crime case. The document may include a police report, a statement by a witness / victim / alleged perpetrator, etc.

[0074] In one embodiment, the MMDT generation module 120 performs document analysis, including the generation of a document tree that represents the sentence and phrase structures of the document. Rhetorical relations associated with inter-document links can determine various navigation scenarios. By default, elaboration can be used. The MMDT generation module 120 can provide links to other documents related by attribute relations if the user is interested in questions such as "why" or "how". If the user expresses an objection to the document originally presented, or if the user requests a document that provides a contrast to the current document, the MMDT generation module 120 can provide links to documents related by contrast relations.

[0075] In yet another aspect, the MMDT generation module 120 obtains a first and a second document. At block 602, the process 600 includes creating a first discourse tree for a first paragraph of the first document. The MMDT generation module 120 accesses a paragraph from the first document. Each sentence of the paragraph includes a fragment or a basic discourse unit. At least one fragment includes a verb. Each word in the fragment includes a role, e.g., a function, of the word in the fragment. The MMDT generation module 120 generates a discourse tree representing the rhetorical relationship between the fragments according to the technique described above in connection with FIG. 3. The discourse tree includes a plurality of nodes, each non-terminal node representing a rhetorical relationship between two fragments, and each terminal node is associated with one of the fragments. The MMDT generation module 120 continues in this manner to build a set of discourse trees for each paragraph in the first document. Although the process 600 is described with respect to paragraphs as units of text, other sizes of text (eg, sentences, pages, chapters, etc.) can also be used.

[0076] At block 603, process 600 includes creating a second discourse tree for the second paragraph of the second document. At block 603, process 600 performs substantially similar steps for the second document as those performed for the first document at block 602. If process 600 creates extended discourse trees for more than two documents, process 600 performs the functions described at block 602 for the multiple documents. Process 600 may iterate over all pairs of discourse trees in the set of discourse trees, where each discourse tree corresponds to a document. A pair of discourse trees may be represented by:

[0077] DT i and D.T. j ∈DTA At block 604, the process 600 includes determining entities and corresponding first basic discourse units from the first discourse tree. Various methods can be used, such as keyword processing (searching for one of a list of predefined keywords in sentences of the first document), using a trained machine learning model, or searching internet resources. The MMDT generation module 120 generates the discourse tree DT. i and D.T. j Identify all noun phrases and named entities in

[0078] In one example, the MMDT generation module 120 extracts noun phrases from a discourse tree. The MMDT generation module 120 then uses a trained machine learning model to classify the noun phrases as (i) an entity or (ii) not an entity.

[0079] At block 605, the process 600 includes determining a second basic discourse unit in the second discourse tree that matches the first basic discourse unit. More specifically, the MMDT generation module 120 calculates the overlap and generates a DT. i and D.T. j A common entity between i,j The MMDT generation module 120 identifies E i,j The MMDT generation module 120 then establishes relationships between occurrences of entities in E i,j For each occurrence of an entity pair in i,j ) is formed.

[0080] In block 606, the process 600 includes, in response to determining a rhetorical relationship between the first basic discourse unit and the second basic discourse unit, creating an extended discourse tree by linking the first discourse tree and the second discourse tree through the rhetorical relationship. More specifically, the MMDT generation module 120 generates an extended discourse tree by, for example, generating an extended discourse tree by using an EDU (E i) and EDU(E j ), construct the DT, and classify the rhetorical relations for each rhetorical link by using the recognized relation labels for each rhetorical link.

[0081] In one aspect, the MMDT generation module 120 combines the first basic discourse unit and the second basic discourse unit into a temporary paragraph, and the discourse navigation application 102 then determines a rhetorical relationship between the first basic discourse unit and the second basic discourse unit in the temporary paragraph by applying discourse syntax analysis to the temporary paragraph.

[0082] In yet another aspect, in response to not determining a rhetorical relationship, the MMDT generation module 120 creates a default rhetorical relationship of type elaboration between the first basic discourse unit and the second basic discourse unit to link the first discourse tree and the second discourse tree.

[0083] In one embodiment, the MMDT generation module 120 performs automatic construction and categorization of links between text spans across documents, where the following approaches can be used: lexical distance, lexical chaining, information extraction, and linguistic template matching. Lexical distance can use cosine similarity across pairs of sentences, and lexical chaining can be more robust, leveraging synonymy and hypernymy.

[0084] The extended discourse tree can form relationships between two or more documents at various levels of granularity. For example, relationships between basic discourse units can be determined as described with respect to process 600. Additionally, the extended discourse tree can represent relationships between words, sentences, paragraphs, sections of documents, or entire documents. As shown, each individual graph is composed of smaller subgraphs of each individual document. Links are shown that represent logical connections between topics within a single document.

[0085] FIG. 7 illustrates relationships between text units of documents at various levels of granularity according to one embodiment. FIG. 7 illustrates discourse trees 701, 702, and 703, each corresponding to a separate document. FIG. 7 also illustrates various inter-document links, such as word links 710 linking words in documents 702 and 703, paragraph / sentence links 711 linking paragraphs or sentences in documents 701 and 702, phrase links 712 linking phrases in documents 701 and 703, and inter-document links 713 linking documents 701 and 703. The MMDT generation module 120 can navigate between documents 701-703 using links 710-713.

[0086] An extended discourse tree, such as the extended discourse tree created by process 700, can be used to navigate a document or other body of text. Extended discourse trees enable a variety of applications, such as autonomous agents, improved search and navigation, question-answer coordination, etc. In some embodiments, an EDT, such as the EDT created by process 700, can be utilized, at least in part, as training data for training a classifier (e.g., classifier 112 of FIG. 1) to classify an input (e.g., any suitable input including text and / or numeric-based data).

[0087] Returning to FIG. 2, at 205, associated data records may be obtained. In the present example where police records are utilized, the associated data records may include any suitable data records. By way of example only, the data records in this example may include call logs, location data, financial transactions (e.g., bank account statements), web page visits, images, etc. In general, the data records may be obtained from the same source or from different sources. These sources may be different from the source of the text data.

[0088] At 206, each data record may have at least a portion of the record transformed into a uniform format. For example, each data source may be transformed into a uniform, canonical format having normalized named representations such as time, date, location, person's name, phone number, account number (if available), etc. In some embodiments, the particular format and / or content transformed may depend on the context and a predefined set of rules. In some embodiments, this information may initially be provided in different formats or with different identifiers, but a set of rules and / or ontology 117 may be utilized to identify a uniform identifier (e.g., name), and the data in each record may be associated with the identifier (e.g., the originally included identifier and / or the uniform identifier).

[0089] At 207, for each elementary discourse unit (EDU) of an EDT (e.g., generated from a corpus of text), a number of candidate phrases that may be associated with ancillary data records may be identified. In some embodiments, identifying a particular candidate phrase (e.g., a particular EDU) may depend on the domain and a predefined protocol for identifying such candidate phrases.

[0090] At 208, for each candidate phrase, the MMDT generation module 120 may identify an entity of the candidate phrase. In some embodiments, the particular entity identified may depend on the context / domain and a set of predefined rules. In some embodiments, this information may initially be provided in different formats or with different identifiers, but a set of rules and / or an ontology 117 may be utilized to identify a unified identifier (e.g., a name), and the candidate phrase may be associated with the identifier (e.g., the originally included identifier and / or the unified identifier).

[0091] At 209, the MMDT generation module 120 may identify matching entities between the data records and the candidate phrases and / or between the data records. For example, the MMDT generation module 120 may iterate over each entity in each data record and compare those entities with entities associated with other data records. An association between two data records having matching entities may be maintained across any suitable number of matches found. In some embodiments, a list of matching data records may be maintained. As another example, the MMDT generation module 120 may iterate over each candidate phrase and compare the entity of the candidate phrase with entities associated with each data record. If a match is found between a candidate phrase and a data record, an association indicating the match may be maintained. In some embodiments, a list of matching EDU / data record pairs may be maintained.

[0092] At 210, the MMDT generation module 120 may perform operations for determining causal links between data records and / or between EDUs (candidate phrases) and data records. Typically, multiple data records are interconnected and represent events that are correlated (some of which cause other events). An algorithm for identifying causal links (e.g., when an event referenced in one data record causes an event referenced in another data record and / or between EDUs and data records). For example, a set of rules may be utilized to identify that if person A calls person B and person A transfers money to person B, the former event causes the latter event. In some embodiments, these rules may be based on the premise that there is no match between values ​​in the data records, i.e., if they share values, the earlier event causes the later event. Thus, a data record may be identified as the cause of an event described in another data record or text, and vice versa.

[0093] An algorithm (e.g., a causal link identification algorithm) for identifying causal links is provided. In some embodiments, two operators R(.) (reason) and C(.) (conclusion) as well as yet another negation - can be utilized. Two negation operators are required: ￢ (￢x indicates that x is false) for negating a propositional formula and - for negating R(.) and C(.). An argument is a formula of the form R(y):(-)C(x). An argument is a reason for concluding a claim. It has two main parts: premises (reason) and a conclusion. The functions R and C play the roles of providing reasons and concluding, respectively. An argument can be interpreted as follows: its conclusion follows as a corollary from the premises according to a given concept, and therefore holds. This concept refers to the nature of the link between them (e.g., the premises imply the conclusion), and is formally identified in the definition by a colon. However, the conclusion may be true but the function does not hold, and vice versa.

[0094] For example, R(y):C(x) corresponds to an output stating that "y is a reason for concluding x" and R(y):-C(x) corresponds to an output stating that "y is a reason for not concluding x". Processing nested arguments is important for discovering breakthrough answers because processing only the object-level layer or only the meta-level layer of the argument separately is insufficient. Nested arguments are central to the processing of texts and dialogues, and support must be provided for nested arguments and negations. Table 2 shows the different forms of arguments and negations that are made possible by a set of predefined definitions (x,y,z,t are propositional formulas to simplify matters). The table is not exclusive.

[0095] [Table 3]

[0096] The example arguments in Table 2 are related to the functioning of credit cards. By default, credit cards work (are ready to be used), especially if the account balance is positive. But there are exceptions: the bank may decline the transaction, whatever the reason. These examples show that internal and external reasons R and charges C may be able to be identified using argument mining techniques. Furthermore, by recursion, internal reasons and charges can be identified by argument mining techniques. Thus, nested structures seem to be more suitable as a target language for arguments when they occur in natural language dialogs and texts.

[0097] Table 3 includes templates that the MMDT generation module 120 can use to extract logical atoms from EDUs (e.g., candidate phrases) and convert rhetorical relations into RC operators to form logical expressions of arguments. To do so, semantic representations can be constructed for expressions of objects related to banking transactions ch(g). These semantic representations can be associated with EDUs. The determined structure of the discourse tree can be used to form RC expressions in L, which are subjected to argument analysis in downstream components.

[0098] [Table 4]

[0099] [Table 5]

[0100] [Table 6]

[0101] The MMDT generation module 120 can generate two causal chains (e.g., one from the EDUs / fragments of a discourse tree generated from a corpus of texts and the other from the text of a data record, the EDUs of the two discourse trees, the EDUs of the two different data records, etc.) according to the above and / or following rules to determine whether the result of the first chain is caused or implied by the latter or formal one. The set of arguments and their negation can be provided as a set of formulas, some of which are identified below. The result operator - is the minimal closure of a set of inference rules expanded with one meta-rule. These predefined rules can be utilized to determine whether a causal link (e.g., the rhetorical relation "cause") is identified between an EDU and a data record and / or between data records. Some of these rules are described below.

[0102] A metarule is any inference rule.

[0103]

number

[0104] This means that the The inference rule inversion process is performed whenever a negation occurs before the leftmost “R”, so in the general case, it is inference rule 1, where i,j∈{0,1}.

[0105]

number

[0106] The reasons are interchangeable. The following rules may show mutual support:

[0107]

number

[0108] Another rule collects different reasons for the same conclusion in a single argument.

[0109]

number

[0110] Cautious monotonicity means that the reasons of an argument can be extended with any premises that it justifies. A cut denotes a kind of minimality of the reasons of an argument.

[0111]

number

[0112] The next two rules represent nestings of R(.) and C(.): exportation shows how to simplify meta-arguments, and permutation shows that permutations of reasons are possible for some forms of meta-argument.

[0113]

number

[0114] If the minimal inference relations obey the above rules, then recursion, monotonicity and cut hold, which means that the operation of arguments by the inference rules is well-founded when using the consequence relations. Let Δ be a set of (negations of) arguments. Let α and β be arguments.

[0115] If α∈Δ then Δα (recursion) If Δβ, then Δ∪{α}β (monotonicity) If Δ∪{α}β and Δα, then Δβ (cut) At 212, the MMDT generation module 120 may perform any suitable operation to confirm the matches and / or causal links identified at 209 and / or 210. This may include iterating over each causal link, including those linking data records to each other and those linking data records to the EDT (e.g., EDUs / candidate phrases in the EDT). Starting from the EDT, the MMDT generation module 120 may generate nodes and edges for each data record involved in the confirmed entity matches and / or causal links to generate the MMDT. As an example, if an entity match is identified between two data records or between a data record and an EDU generated from a corpus of text, and a node does not already exist in the EDT for one (or both) data records, a new node may be generated (e.g., for each data record), and depending on the particular use case, the new node of one data record may be linked to the EDU generated from the corpus or the new node generated for the other data record. For example, if an EDU generated from a corpus contains an entity that matches an entity found in a data record (e.g., they both contain a term found in the ontology 117 of FIG. 1, they both contain a matching noun such as "San Francisco", etc.) and the node in the data record does not already exist, the MMDT generation module 120 may generate a new node for the EDT and the node corresponding to the EDU may be linked to this new node using an edge associated with a rhetorical relation (e.g., "elaboration"). In some embodiments, the MMDT generation module 120 may link entity matches via one rhetorical relation (e.g., "elaboration"), while causal links may be linked in the MMDT via a different rhetorical relation (e.g., "cause").

[0116] 8 is a diagram illustrating an example of a multimodal discourse tree 800 according to at least one embodiment. Figure 8 illustrates the expanded discourse tree 500 of Figure 5 with additional edges corresponding to data records 802, 804, and 806.

[0117] In addition to links between specific discourse trees, such as discourse trees 511, 521, 531, 541, and 551, the extended discourse tree 500 includes inter-discourse tree links 561-564 and associated inter-document links 571-574. As will be further described with respect to FIG. 6, the MMDT generation module 120 can construct discourse trees 511-515. Discourse tree 511 represents document 512, discourse tree 521 represents document 522, and so on. The extended discourse tree 500 can be constructed by building a discourse tree (e.g., DT and / or CDT) for each paragraph or document. The discourse trees 511-515 can be constructed from text of a corpus.

[0118] Inter-discourse tree link 561 connects discourse trees 511 and 521, inter-discourse tree link 562 connects discourse trees 521 and 531, inter-discourse tree link 563 connects discourse trees 511 and 541, and inter-discourse tree link 564 connects discourse trees 521 and 551. Based on the inter-discourse tree links 561-564, the MMDT generation module 120 creates inter-document links 571, 572, 573, and 574 corresponding to the inter-discourse tree links 561, 562, 563, and 564, respectively. The inter-document links 571-574 can be used to navigate documents 512, 522, 532, 542, and 552.

[0119] The MMDT generation module 120 determines one or more entities in a first discourse tree of the discourse trees 511-515. Examples of entities include places, things, people, or businesses. The MMDT generation module 120 then identifies identical entities that exist in other discourse trees. Based on the determined entities, the MMDT generation module 120 determines rhetorical relationships between each matching entity.

[0120] As described above, the MMDT generation module 120 may be configured to identify one or more entities within the EDUs and data records of a first discourse tree of the discourse trees 511-515. For example, the MMDT generation module 120 may identify entities referenced in the discourse tree 510 and the data record 802. Based on the determined entities, the MMDT generation module 120 may generate links 808 (e.g., edges).

[0121] In some embodiments, the MMDT generation module 120 may be configured to identify one or more causal links within and / or between EDUs and data records of a first one of the discourse trees 511-515. For example, using the algorithm described above in connection with FIG. 2, the MMDT generation module 120 may identify causal links between data records 802 and 804 and between the discourse tree 541 (or an EDU of the discourse tree 541) and the data record 806. The MMDT generation module 120 may generate causal links 810 and 812, respectively, based on these identifications.

[0122] By using the links in the extended discourse tree 500, the MMDT generation module 120 can navigate between paragraphs of the same document or between documents of a text corpus, and between data records and discourse trees. The MMDT 800 may not, at least initially, contain rhetorical relations corresponding to the links 808-812.

[0123] Returning to FIG. 2, at 214, the MMDT generation module 120 may perform operations to identify rhetorical relations between the phases / data record matches / links identified at 212 and the data records / data record matches / links. To identify rhetorical relations between text fragments, the MMDT generation module 120 may perform coherence analysis and discourse syntax analysis on the paragraphs using fictional text fragments or temporary paragraphs from each text fragment of the original paragraph to identify relationships between entities. In some embodiments, the MMDT generation module 120 may utilize an ontology (e.g., ontology 117 of FIG. 1) to identify relationships between entities based on relationships provided in the ontology. The identified rhetorical relations may be associated with the links 808-812 of FIG. 8.

[0124] At 216, the MMDT generation module 120 may perform operations to identify data records (e.g., data records 808-812) as corresponding to a nucleus or satellite of the EDT. Any suitable predefined set of rules may be utilized to identify the data records as corresponding to a nucleus or satellite of the EDT (e.g., of a candidate phrase / EDU with which the EDT is associated).

[0125] At 218, the generated MMDT may be converted to a normalized MMDT. This conversion may include generating EDUs for the data records and appending the generated EDUs to the EDT according to the decisions made at 216. A simple example is described below in which a data record previously linked to a pair of textual EDUs and associated with an elaboration rhetorical relationship may be added as an EDU. Rhetorical relationships (causes) may be inserted to strengthen the kernel according to a predefined set of rules. This process may be performed for each data record until it is included as an EDU in the now completed MMDT.

[0126] [Table 7]

[0127] FIG. 9 illustrates the relationships between text units of documents at various levels of granularity and between those text units and associated data records in an MMDT, according to at least one embodiment. FIG. 9 illustrates an MMDT 900 that includes discourse trees 901, 902, and 903 (e.g., example DTs or CDTs generated using the techniques described above in FIG. 3 and FIG. 4, respectively), each corresponding to a separate document. FIG. 9 also illustrates various inter-document links, such as word links 910 linking words in documents 902 and 903, paragraph / sentence links 911 linking paragraphs or sentences in documents 901 and 902, phrase links 912 linking phrases in documents 901 and 903, and inter-document links 913 linking documents 901 and 903. MMDT 900 also includes data records 905-909. The MMDT 900 may include any suitable number of causal links (e.g., causal links 910-912) and / or entity links (e.g., entity links 913-915) identified in the manner described above in connection with FIG. 2. In some embodiments, each of the data records 905-909 may be included as an EDU in the discourse trees 901-903 that link the data records 905-909. The MMDT generation module 120 may use any suitable links to navigate between the documents 901-903 and / or data records 905-909.

[0128] 10 is a flow diagram of a method 1000 for generating a multimodal discourse tree according to at least one embodiment. In some embodiments, the method 1000 may be performed by the computing device 102 of FIG. 1 (e.g., the application 106 of FIG. 1, the MMDT generation module 120 of FIG. 1). The operations of the method 1000 may be performed in any suitable order. In some embodiments, the method 1000 may include more operations or fewer operations than those shown in FIG. 10.

[0129] Method 1000 may begin at 1001, where a corpus of text and one or more data records separate from the corpus of text may be obtained (e.g., from text data 116 and record data store 118, respectively, of FIG. 1). The one or more data records may be obtained from any suitable source. In some embodiments, the source of the one or more data records may be different from the source of the corpus of text.

[0130] At 1002, an expanded discourse tree (e.g., EDT 500 of FIGS. 5 and 8) may be generated for a corpus of text. In some embodiments, the expanded discourse tree includes multiple discourse trees (e.g., each an example of DT 300 of FIG. 3 or CDT 400 of FIG. 4). In some embodiments, each discourse tree includes multiple nodes, each terminal node of the discourse tree corresponds to a fragment of text (and / or a data record), and each non-terminal node of the discourse tree indicates a rhetorical relationship between the nodes of the discourse tree. In some embodiments, the expanded discourse tree includes further links between the multiple discourse trees that indicate further rhetorical relationships between the nodes of the respective discourse trees.

[0131] At 1003, entity matches may be identified between (and / or between) the set of base discourse units of the multiple discourse trees and one or more data records. In some embodiments, the entity matches are identified by comparing a first entity identified from the base discourse units and a second entity identified from the data records.

[0132] At 1004, one or more causal links may be identified. For example, a causal link identification algorithm (e.g., as described in connection with FIG. 2) may be performed to identify one or more causal links between two of the one or more data records.

[0133] At 1005, for each identified entity match and each of the identified one or more causal links, a corresponding rhetorical relationship. As an example, for each entity match, a "detail" rhetorical relationship may be identified, and for each causal link, a "cause" rhetorical relationship may be identified.

[0134] At 1006, a respective node may be generated for the expanded discourse tree for each identified entity match and each identified causal link.

[0135] At 1007, a multimodal discourse tree (e.g., MMDTs 800 and 900 in Figures 8 and 9) may be created by linking the respective nodes generated for each entity match and each causal link to respective nodes in an expanded discourse tree based at least in part on the determined corresponding rhetorical relationships.

[0136] In some embodiments, generating the extended discourse comprises: i) generating a first discourse tree from a first text of the corpus of text, the first discourse tree corresponding to a first portion of the first text; generating the extended discourse further comprises: ii) generating a second discourse tree from a second text of the corpus of text, the second discourse tree corresponding to a second portion of the second text; generating the extended discourse further comprises: iii) in response to determining a particular rhetorical relationship between respective basic discourse units of the first discourse tree and the second discourse tree, linking the first discourse tree and the second discourse tree using the particular rhetorical relationship.

[0137] In some embodiments, the first discourse tree and the second discourse tree are communicative discourse trees that include respective verb signatures generated for each basic discourse unit of the first discourse tree and the second discourse tree.

[0138] In some embodiments, identifying the entity further comprises comparing the second entity identified from the data record with a third entity identified from a second data record, the entity referring to one of: (i) a person, (ii) a business, (iii) a location, (iv) a name of a document, or (v) a date or time.

[0139] In some embodiments, identifying the entity match further comprises identifying the entity from a predefined ontology.

[0140] In some embodiments, method 1000 may further include classifying a subsequent input based at least in part on the multimodal discourse tree, where classifying the subsequent input includes i) generating a training dataset including a plurality of multimodal discourse trees, where each multimodal discourse tree corresponds to a respective text corpus and a respective set of data records, where each multimodal discourse tree is associated with a label corresponding to a classification, where classifying the subsequent input further includes ii) training a machine learning model to classify an input based at least in part on the training dataset and a supervised learning algorithm, and iii) generating a corresponding multimodal discourse tree from the subsequent input, where the subsequent input includes a respective set of text and a respective set of data records, where classifying the subsequent input further includes iv) classifying the subsequent input based at least in part on providing the corresponding multimodal discourse generated from the subsequent input as an input to the machine learning model and receiving an output from the machine learning model indicative of a classification of the subsequent input.

[0141] In some embodiments, method 1000 further includes navigating the corpus of text using the multimodal discourse tree, where navigating the corpus of text includes accessing the multimodal discourse tree and determining from the multimodal discourse tree a first basic discourse unit responsive to a query from a user device, the first basic discourse unit corresponding to a first node of a first discourse tree of the multimodal discourse tree, and navigating the corpus of text further includes determining from the multimodal discourse tree a set of navigation options, the set of navigation options being based on (i) a first rhetorical relationship between the first node of the first discourse tree and a second node of the first discourse tree and (ii) a second rhetorical relationship between the first node and a third node of a second discourse tree of the multimodal discourse tree, or (iii) a third rhetorical relationship between the first node and a third node of a second discourse tree of the multimodal discourse tree. and a third rhetorical relationship between the first node of one discourse tree and a fourth node of the multimodal discourse tree associated with a corresponding data record, and navigating the corpus of text further includes presenting at least two of the first, second or third rhetorical relationships on the user device, and in response to receiving further user input including a selection of the first rhetorical relationship, the second rhetorical relationship or the third rhetorical relationship, i) presenting a second basic discourse unit corresponding to the second node based at least in part on determining the selection corresponding to the first rhetorical relationship, ii) presenting a third basic discourse unit corresponding to the third node based at least in part on determining the selection corresponding to the first rhetorical relationship, or iii) presenting at least a portion of the corresponding data record based at least in part on determining the selection corresponding to the third rhetorical relationship.

[0142] Exemplary Computing System 11 is a simplified diagram of a distributed system 1100 for implementing one of the above aspects. In the illustrated aspect, the distributed system 1100 includes one or more client computing devices 1102, 1104, 1106, and 1108 configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., via one or more networks 1110. A server 1112 can be communicatively coupled to the remote client computing devices 1102, 1104, 1106, and 1108 via the network 1110.

[0143] In various aspects, the server 1112 may be adapted to execute one or more services or software applications provided by one or more of the system's components. The services or software applications may include non-virtual and virtual environments. The virtual environments may include those used for virtual events, trade shows, simulators, classrooms, purchasing commodity transactions, and business activities, whether two-dimensional or three-dimensional (3D) representations, page-based logical environments, and the like. These services may be provided to users of the client computing devices 1102, 1104, 1106, and / or 1108 as web-based or cloud services, or under a Software as a Service (SaaS) model. Users operating the client computing devices 1102, 1104, 1106, and / or 1108 may then utilize one or more client applications to interact with the server 1112 to utilize the services provided by these components.

[0144] In the illustrated configuration, the software components 1118, 1120, and 1122 of the distributed system 1100 are shown implemented on the server 1112. Also, in other aspects, one or more of the components of the distributed system 1100 and / or the services provided by these components may be implemented by one or more of the client computing devices 1102, 1104, 1106, and / or 1108. A user operating a client computing device may then utilize one or more client applications to use the services provided by these components. These components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from the distributed system 1100. Thus, the illustrated embodiment is an example of a distributed system for implementing one embodiment of the system and is not intended to be limiting.

[0145] The client computing devices 1102, 1104, 1106 and / or 1108 may be handheld mobile devices (e.g., iPhone, mobile phone, iPad, computing tablet, Personal Digital Assistant (PDA)) or wearable devices (e.g., Google Glass head mounted display) running software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, and / or Internet, email, Short Message Service (SMS), BlackBerry, or other communication protocols available. The client computing devices may be general purpose personal computers, including, by way of example, personal and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may be workstation computers running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS. Alternatively, or additionally, the client computing devices 1102, 1104, 1106, and 1108 may be thin client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox gaming consoles with or without Kinect gesture input devices), and / or other electronic devices such as personal messaging devices capable of communicating over the network 1110.

[0146] Although the exemplary distributed system 1100 is shown having four client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the server 1112.

[0147] Network 1110 in distributed system 1100 may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of commercially available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk, etc. By way of example only, network 1110 may be a Local Area Network (LAN), such as one based on Ethernet, Token Ring, etc. Network 1110 may also be a Wide Area Network and the Internet. The network 1110 may include a virtual network, including, but not limited to, a Virtual Private Network (VPN), an intranet, an extranet, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics (IEEE) 802.9 suite of protocols, Bluetooth and / or other wireless protocols), and / or any combination thereof, and / or other networks.

[0148] The servers 1112 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 1112 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization. One or more flexible pools of logical storage may be virtualized to maintain virtual storage for the servers. The virtual network may be controlled by the servers 1112 using software-defined networking. In various aspects, the servers 1112 may be adapted to execute one or more services or software applications described in the disclosure above. For example, the servers 1112 may correspond to a server for executing the above-mentioned processes according to aspects of the present disclosure.

[0149] Server 1112 may run an operating system, including any of those listed above, as well as any commercially available server operating system. Server 1112 may also run any of a variety of additional server and / or middle tier applications, including a Hypertext Transport Protocol (HTTP) server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a JAVA server, a database server, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM International Business Machines, etc.

[0150] In some implementations, the server 1112 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of the client computing devices 1102, 1104, 1106, and 1108. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party information sources and continuous data streams, including real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like. The server 1112 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of the client computing devices 1102, 1104, 1106, and 1108.

[0151] The distributed system 1100 may also include one or more databases 1114 and 1116. The databases 1114 and 1116 may reside in a variety of locations. As an example, one or more of the databases 1114 and 1116 may reside on a non-transitory storage medium local to (and / or present on) the server 1112. Alternatively, the databases 1114 and 1116 may be remote from the server 1112 and communicate with the server 1112 via a network-based or dedicated connection. In one set of embodiments, the databases 1114 and 1116 may reside on a Storage-Area Network (SAN). Similarly, any necessary files to perform functions attributed to the server 1112 may be stored locally and / or remotely on the server 1112 as appropriate. In one set of embodiments, the databases 1114 and 1116 may include relational databases, such as those provided by Oracle Corporation, adapted to store, update, and retrieve data in response to SQL-formatted commands.

[0152] 12 is a simplified block diagram of one or more components of a system environment 1200 capable of delivering services provided by one or more components of an embodiment of the system as cloud services, according to one embodiment of the disclosure. In the embodiment shown, the system environment 1200 includes one or more client computing devices 1204, 1206, and 1208 that may be used by a user to interact with a cloud infrastructure system 1202 that provides cloud services. The client computing devices may be configured to run client applications, such as a web browser, a proprietary client application (e.g., Oracle Forms), or other applications, that may be used by a user of the client computing device to interact with the cloud infrastructure system 1202 to use services provided by the cloud infrastructure system 1202.

[0153] It should be understood that the cloud infrastructure system 1202 depicted in the figure may have other components than those shown. Additionally, the illustrated aspect is only one example of a cloud infrastructure system that may incorporate aspects of the present invention. In some embodiments, cloud infrastructure system 1202 may have more or fewer components than those depicted in the figure, may combine two or more components, or may have a different configuration or arrangement of components.

[0154] The client computing devices 1204 , 1206 , and 1208 may be devices similar to those described above for the client computing devices 1102 , 1104 , 1106 , and 1108 .

[0155] Although the exemplary system environment 1200 is shown with three client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the cloud infrastructure system 1202.

[0156] Network 1210 may facilitate communication and exchange of data between client computing devices 1204, 1206, and 1208 and cloud infrastructure system 1202. Each network may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of commercially available protocols, including those described above for network 1210.

[0157] Cloud infrastructure system 1202 may include one or more computers and / or servers, which may include those described above for server 1212.

[0158] In certain aspects, the services provided by the cloud infrastructure system may include a multitude of services available on demand to users of the cloud infrastructure system, such as online data storage and backup solutions, web-based email services, hosted office suites and document collaboration services, database processing, managed technical support services, and the like. The services provided by the cloud infrastructure system are dynamically scalable to meet the needs of its users. A specific instantiation of a service provided by the cloud infrastructure system is referred to herein as a "service instance." In general, any service available to users from a cloud service provider's system over a communications network such as the Internet is referred to as a "cloud service." Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from the customer's own on-premise servers and systems. For example, the cloud service provider's system may host an application, and a user may order and use the application on demand over a communications network such as the Internet.

[0159] In some examples, services in a computer network cloud infrastructure may include storage, hosted databases, hosted web servers, protected computer network access to software applications, or other services provided to users by cloud vendors or otherwise known in the art. For example, services may include password-protected access to remote storage on the cloud over the Internet. As another example, services may include web-services-based hosted relational databases and scripting language middleware engines for private use by networked developers. As another example, services may include access to email software applications hosted on the cloud vendor's website.

[0160] In certain aspects, cloud infrastructure system 1202 may include a suite of application, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. One example of such a cloud infrastructure system is the Oracle Public Cloud offered by the present assignee.

[0161] Large amounts of data, sometimes referred to as big data, can be hosted and / or manipulated by infrastructure systems at multiple levels and at different scales. Such data may include data sets so large and complex that they can be difficult to process using typical database management tools or traditional data processing applications. For example, terabytes of data may be difficult to store, retrieve, and process using personal computers or their rack-based counterparts. Data of such size may be difficult to work with using modern relational database management systems and desktop statistics and visualization packages. They may require massively parallel processing software running thousands of server computers, beyond the architecture of commonly used software tools, to capture, curate, manage, and process the data within acceptable elapsed times.

[0162] Analysts and researchers can store and process extremely large data sets to visualize, detect trends, and / or interact with large amounts of data. Dozens, hundreds, or thousands of processors linked in parallel can act on such data to display it or to simulate external forces on it or what it represents. These data sets may involve structured data, such as data organized in a database or according to a structured model, and / or unstructured data (e.g., emails, images, data blobs (binary large objects), web pages, complex event processing). By enhancing the ability of aspects to relatively quickly focus more (or fewer) computing resources on a target, cloud infrastructure systems can be better utilized to perform tasks on large data sets based on requests from businesses, government agencies, research organizations, private individuals, groups of like-minded individuals or organizations, or other entities.

[0163] In various aspects, cloud infrastructure system 1202 may be adapted to automatically provision, manage, and track customer subscriptions to services provided by cloud infrastructure system 1202. Cloud infrastructure system 1202 may provide cloud services through various deployment models. For example, services may be provided under a public cloud model, where cloud infrastructure system 1202 is owned by an organization that sells cloud services (e.g., owned by Oracle Corporation) and the services are available to the general public or various industry enterprises. As another example, services may be provided under a private cloud model, where cloud infrastructure system 1202 operates solely for a single organization and may provide services to one or more entities within that organization. Cloud services may also be provided under a community cloud model, where cloud infrastructure system 1202 and the services provided by cloud infrastructure system 1202 are shared by several organizations within an associated community. Cloud services may also be provided under a hybrid cloud model, which is a combination of two or more different models.

[0164] In some embodiments, the services provided by cloud infrastructure system 1202 may include one or more services provided under the Software as a Service (SaaS) category, the Platform as a Service (PaaS) category, the Infrastructure as a Service (IaaS) category, or other categories of services including hybrid services. A customer may order one or more services provided by cloud infrastructure system 1202 via a subscription order. Cloud infrastructure system 1202 then performs processing to provide the services in the customer's subscription order.

[0165] In some embodiments, the services provided by the cloud infrastructure system 1202 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, application services may be provided by the cloud infrastructure system through a SaaS platform. The SaaS platform may be configured to provide cloud services that fall into the SaaS category. For example, the SaaS platform may provide functionality for building and delivering a suite of on-demand applications on a unified development and deployment platform. The SaaS platform may manage and control the underlying software and infrastructure for providing the SaaS services. By utilizing the services provided by the SaaS platform, customers can utilize applications that run on the cloud infrastructure system. Customers can obtain application services without the customer having to purchase separate licenses and support. A variety of different SaaS services may be provided. Examples include, but are not limited to, services that provide solutions for sales performance management, enterprise integration, and business flexibility for large organizations.

[0166] In some embodiments, platform services may be provided by the cloud infrastructure system through a PaaS platform. The PaaS platform may be configured to provide cloud services that fall under the PaaS category. Examples of platform services may include, but are not limited to, services that allow an organization (such as Oracle) to consolidate existing applications on a shared common architecture, and the ability to build new applications that leverage shared services provided by the platform. The PaaS platform may manage and control the underlying software and infrastructure to provide the PaaS services. Customers may obtain the PaaS services provided by the cloud infrastructure system without the customer having to purchase separate licenses and support. Examples of platform services include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), etc.

[0167] By utilizing the services provided by the PaaS platform, customers can utilize programming languages ​​and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, the platform services provided by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion middleware services), and Java cloud services. In one aspect, the database cloud services may support a shared service deployment model that allows organizations to pool database resources and provide database-as-a-service to customers in the form of a database cloud. The middleware cloud services may provide customers with a platform for developing and deploying various business applications in the cloud infrastructure system, and the Java cloud services may provide customers with a platform for deploying Java applications in the cloud infrastructure system.

[0168] A variety of different infrastructure services may be provided by the IaaS platform in a cloud infrastructure system. The infrastructure services facilitate management and control of basic computing resources such as storage, network, and other underlying computing resources for customers who use the services provided by the SaaS and PaaS platforms.

[0169] Also, in certain aspects, cloud infrastructure system 1202 may include infrastructure resources 1230 for providing resources used to provide various services to customers of the cloud infrastructure system. In one aspect, infrastructure resources 1230 may include a pre-integrated optimized combination of hardware, such as servers, storage, and networking resources, for running the services offered by the PaaS and SaaS platforms.

[0170] In some embodiments, resources in cloud infrastructure system 1202 may be shared by multiple users and dynamically reallocated on a demand basis. Resources may also be allocated to users in different time periods. For example, cloud infrastructure system 1202 may maximize resource utilization by allowing a first set of users in a first time period to utilize resources of the cloud infrastructure system for a defined period of time and allowing reallocation of the same resources to another set of users located in a different time period.

[0171] In certain aspects, several internal shared services 1232 may be provided that are shared by various components or modules of, and services provided by, cloud infrastructure system 1202. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0172] In certain aspects, cloud infrastructure system 1202 may provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one aspect, cloud management functionality may include functionality for provisioning, managing, tracking, etc., customer subscriptions received by cloud infrastructure system 1202.

[0173] In one aspect, as shown in the figure, cloud management functionality may be provided by one or more modules, such as an order management module 1220, an order orchestration module 1222, an order provisioning module 1211, an order management and monitoring module 1210, and an identity management module 1228. These modules may include or be provided using one or more computers and / or servers, which may be general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0174] In an example operation 1234, a customer using a client device, such as client computing devices 1204, 1206, or 1208, may interact with cloud infrastructure system 1202 by requesting one or more services offered by cloud infrastructure system 1202 and placing an order for a subscription to one or more services provided by cloud infrastructure system 1202. In certain aspects, a customer may access cloud user interfaces (UIs), namely cloud UI 1212, cloud UI 1214, and / or cloud UI 1216, and place a subscription order via these UIs. Order information received by cloud infrastructure system 1202 in response to a customer placing an order may include information identifying the customer and the one or more services offered by cloud infrastructure system 1202 to which the customer intends to subscribe.

[0175] After an order is placed by a customer, the order information is received via the cloud UI 1210, 1214 and / or 1216.

[0176] At operation 1236, the order is stored in an order database 1218. The order database 1218 may be one of several databases operated by the cloud infrastructure system 1202 and in conjunction with other system elements.

[0177] At operation 1238, the order information is forwarded to the order management module 1220. In some examples, the order management module 1220 can be configured to perform billing and accounting functions related to the order, such as confirming the order and booking the order upon confirmation.

[0178] At operation 1240, information regarding the order is communicated to the order orchestration module 1222. The order orchestration module 1222 may utilize the order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some examples, the order orchestration module 1222 may orchestrate the provisioning of resources to support the services subscribed to using the services of the order provisioning module 1211.

[0179] In certain aspects, the order orchestration module 1222 enables management of business processes associated with each order and applies business logic to determine whether the order should proceed to provisioning. In operation 1242, upon receiving an order for a new subscription, the order orchestration module 1222 sends a request to the order provisioning module 1211 to allocate resources and configure those resources required to fulfill the subscription order. The order provisioning module 1211 enables the allocation of resources for services ordered by a customer. The order provisioning module 1211 provides a level of abstraction between the cloud services provided by the cloud infrastructure system 1202 and the physical implementation layer used to provision resources to provide the requested services. Thus, the order orchestration module 1222 can be decoupled from implementation details such as whether services and resources are actually provisioned on the fly or pre-provisioned and only allocated / allocated when requested.

[0180] In operation 1244 , once the services and resources have been provisioned, notification of the rendered services may be sent to the customer on the client computing devices 1204 , 1206 and / or 1208 by the order provisioning module 1211 of the cloud infrastructure system 1202 .

[0181] At operation 1246, the customer's subscription order may be managed and tracked by the order management and monitoring module 1210. In some examples, the order management and monitoring module 1210 may be configured to collect usage statistics for the services in the subscription order, such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system up and down time.

[0182] In certain aspects, cloud infrastructure system 1202 may include identity management module 1228. Identity management module 1228 may be configured to provide identity services, such as access management and authorization services in cloud infrastructure system 1202. In some embodiments, identity management module 1228 may control information about customers who wish to utilize services provided by cloud infrastructure system 1202. Such information may include information authenticating the identities of such customers and information describing which actions those customers are authorized to perform on various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). Identity management module 1228 may also include management of descriptive information about each customer and how and by whom this descriptive information may be accessed and modified.

[0183] 13 illustrates an exemplary computing subsystem 1300 in which various aspects can be implemented. The computing subsystem 1300 can be used to implement any of the computing subsystems described above. As shown in the figure, the computing subsystem 1300 includes a processing unit 1304 that communicates with several peripheral subsystems via a bus subsystem 1302. These peripheral subsystems can include a processing acceleration unit 1306, an I / O subsystem 1308, a storage subsystem 1318, and a communication subsystem 1311. The storage subsystem 1318 includes a tangible computer-readable storage medium 1309 and a system memory 1310.

[0184] The bus subsystem 1302 provides a mechanism for allowing the various components and subsystems of the computing subsystem 1300 to communicate with each other as intended. Although the bus subsystem 1302 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1302 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured to the IEEE P1186.1 standard.

[0185] The processing unit 1304, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computing subsystem 1300. The processing unit 1304 may include one or more processors. These processors may include single-core or multi-core processors. In certain aspects, the processing unit 1304 may be implemented as one or more independent processing units 1332 and / or 1334 with single-core or multi-core processors included in each processing unit. In other aspects, the processing unit 1304 may be implemented as a quad-core processing unit formed by incorporating two dual-core processors on a single chip.

[0186] In various aspects, the processing unit 1304 may execute various programs in response to program code and may maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the processing unit 1304 and / or in the storage subsystem 1318. Through suitable programming, the processing unit 1304 may provide the various functions described above. The computing subsystem 1300 may also additionally include a processing acceleration unit 1306, which may include a digital signal processor (DSP), special purpose processor, etc.

[0187] The I / O subsystem 1308 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touch pads or touch screens, which are integrated with displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices, along with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include motion sensing and / or gesture recognizers, such as Microsoft Kinect motion sensors, which allow a user to control and interact with an input device, such as a Microsoft Xbox 360 game controller, through a natural user interface using gestures and spoken commands. User interface input devices may also include eye gesture recognizers, such as the Google Glass blink detector, which detects eye movements from a user (e.g., "blinking" while taking pictures and / or making menu selections) and translates the eye gestures as input to an input device (e.g., Google Glass). The user interface input devices may also include voice recognition sensing devices that allow a user to interact with a voice recognition system (eg, the Siri® navigator) via voice commands.

[0188] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye gaze detection devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound equipment. User interface input devices may also include audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0189] The user interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a flat panel device such as one using a Cathode Ray Tube (CRT), Liquid Crystal Display (LCD) or plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computing subsystem 1300 to a user or to another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0190] Computing subsystem 1300 may include a storage subsystem 1318 that comprises software elements currently shown as located in system memory 1310. System memory 1310 may store program instructions loadable and executable on processing unit 1304, as well as data generated during the execution of these programs.

[0191] Depending on the configuration and type of computing subsystem 1300, the system memory 1310 may be volatile (such as Random Access Memory (RAM)) and / or non-volatile (such as Read-Only Memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to the processing unit 1304 and / or that are currently being operated on and executed by the processing unit 1304. In some implementations, the system memory 1310 may include multiple different types of memory, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). In some implementations, a Basic Input / Output System (BIOS), containing the basic routines that help to transfer information between elements within the computing subsystem 1300, such as during start-up, may typically be stored in ROM. By way of example and not limitation, system memory 1310 also illustrates application programs 1312, program data 1314, and an operating system 1316, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like.As an example, operating system 1316 may include various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems, various commercially available UNIX or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome OS, etc.), and / or mobile operating systems such as iOS, Windows Phone, Android OS, BlackBerry 10 OS, and Palm OS operating systems.

[0192] The storage subsystem 1318 may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some aspects. Software (programs, code modules, instructions) that, when executed by a processor, provide the above-described functionality may be stored in the storage subsystem 1318. These software modules or instructions may be executed by the processing unit 1304. The storage subsystem 1318 may also provide a repository for storing data used in accordance with the present invention.

[0193] The storage subsystem 1318 may also include a computer readable storage medium reader 1320 that may be further connected to a computer readable storage medium 1309. Together and optionally in combination with the system memory 1310, the computer readable storage medium 1309 may comprehensively represent remote, local, fixed and / or removable storage devices, as well as storage media for temporarily and / or permanently containing, storing, transmitting and retrieving computer readable information.

[0194] The computer readable storage medium 1309 containing the code or portions of code may include any suitable medium known or used in the art. Such media includes storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing and / or transmitting information. This may include tangible non-transitory computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable media. Also, where specified, this may include intangible transitory computer readable media, such as data signals, data transmissions, or any other media usable to transmit the desired information and accessible by the computing subsystem 1300.

[0195] By way of example, computer readable storage medium 1309 may include hard disk drives that read from or write to non-removable, non-volatile magnetic media, magnetic disk drives that read from or write to removable, non-volatile magnetic disks, and optical disk drives that read from or write to removable, non-volatile optical disks, such as CD ROMs, DVDs and Blu-ray disks or other optical media. Computer readable storage medium 1309 may include, but is not limited to, Zip drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. Additionally, computer-readable storage media 1309 may include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media may provide computer-readable instructions, data structures, program modules, and other data to computing subsystem 1300.

[0196] The communications subsystem 1311 provides an interface with other computing subsystems and networks. The communications subsystem 1311 serves as an interface for receiving data from other systems and transmitting data from the computing subsystem 1300 to other systems. For example, the communications subsystem 1311 may enable the computing subsystem 1300 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1311 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technology such as 3G, 4G, or EDGE (enhanced data rates for global evolution)), a WiFi (IEEE 902.9 family standard, or other mobile communications technology, or any combination thereof), a Global Positioning System (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1311 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0197] Also, in some embodiments, the communications subsystem 1311 may receive incoming communications in the form of structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, etc. on behalf of one or more users that may be using the computing subsystem 1300.

[0198] As an example, the communications subsystem 1311 may be configured to receive data feeds 1326, such as web feeds, such as Twitter feeds, Facebook updates, Rich Site Summary (RSS) feeds, in real time from users of social media networks and / or other communications services, and / or receive real-time updates from one or more third party sources.

[0199] Additionally, the communications subsystem 1311 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1328 of real-time events and / or event updates 1330 that may be continuous or may be unbounded with essentially no clear ends. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0200] The communications subsystem 1311 may also be configured to output structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computing subsystem 1300.

[0201] The computing subsystem 1300 may be one of a variety of types, including a handheld mobile device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack or other data processing system.

[0202] Due to the ever-changing nature of computers and networks, the description of the computing subsystem 1300 shown in the figure is intended only as a specific example. Many other configurations are possible having more or fewer components than the system shown in the figure. For example, customized hardware may be used and / or specific elements may be implemented in hardware, firmware, software (including applets), or a combination. Additionally, connections to other computing devices, such as network input / output devices, may be utilized. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other means and / or methods for implementing various aspects. Various aspects of the present disclosure may be implemented using a computer program product that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0203] In the foregoing specification, the aspects of the invention have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. The various features and aspects of the invention described above may be used individually or together. Moreover, the aspects may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are therefore to be regarded as illustrative rather than restrictive.

Claims

1. 1. A computer-implemented method for generating a multimodal discourse tree, comprising: obtaining a corpus of text and one or more data records separate from said corpus of text; generating an extended discourse tree for the corpus of text, the extended discourse tree including a plurality of discourse trees, each discourse tree including a plurality of nodes, each terminal node of the discourse tree corresponding to a fragment of text, each non-terminal node of the discourse tree indicating a rhetorical relationship between the nodes of the discourse trees, the extended discourse tree including further links between the plurality of discourse trees indicating further rhetorical relationships between the nodes of the respective discourse trees, the computer-implemented method further comprising: identifying an entity match between a set of basic discourse units of the plurality of discourse trees and the one or more data records, the entity match being identified by comparing a first entity identified from the basic discourse units and a second entity identified from the data records, the computer-implemented method further comprising: identifying one or more causal links between two of said one or more data records; determining a corresponding rhetorical relationship for each identified entity match and each of the one or more identified causal links; generating a respective node for each identified entity match and each identified causal link for the expanded discourse tree; and creating the multimodal discourse tree by linking the respective nodes generated for each entity match and each causal link to respective nodes of the expanded discourse tree based at least in part on the determined corresponding rhetorical relationships.

2. The step of generating an extended discourse tree comprises: generating a first discourse tree from a first text of the corpus of text, the first discourse tree corresponding to a first portion of the first text, the step of generating an extended discourse tree further comprising: generating a second discourse tree from a second text of the corpus of text, the second discourse tree corresponding to a second portion of the second text, the step of generating an extended discourse tree further comprising:

2. The computer-implemented method of claim 1, further comprising: in response to determining a particular rhetorical relationship between basic discourse units of the first discourse tree and the second discourse tree, linking the first discourse tree and the second discourse tree using the particular rhetorical relationship.

3. 3. The computer-implemented method of claim 2, wherein the first discourse tree and the second discourse tree are communicative discourse trees that include respective verb signatures generated for each basic discourse unit of the first discourse tree and the second discourse tree.

4. 3. The computer-implemented method of claim 2, wherein identifying the entity match further comprises comparing the second entity identified from the data record with a third entity identified from a second data record, where an entity refers to one of: (i) a person, (ii) a business, (iii) a location, (iv) a name of a document, (v) a date or time, (vi) a transaction, or (vii) an activity.

5. The computer-implemented method of claim 1 , wherein the entity matching further comprises identifying the entity from a predefined ontology.

6. The method further includes classifying a subsequent input based at least in part on the multimodal discourse tree, the step of classifying the subsequent input comprising: generating a training data set including a plurality of multimodal discourse trees, each multimodal discourse tree corresponding to a respective text corpus and a respective set of data records, each multimodal discourse tree being associated with a label corresponding to a classification, said step of classifying said subsequent input further comprising: training a machine learning model to classify an input based at least in part on the training dataset and a supervised learning algorithm; and generating a corresponding multimodal discourse tree from the subsequent input, the subsequent input including a corresponding text corpus and a corresponding set of data records, and classifying the subsequent input further comprising:

2. The computer-implemented method of claim 1, comprising: classifying the subsequent input based at least in part on providing the corresponding multimodal discourse tree generated from the subsequent input as an input to the machine learning model, and receiving an output from the machine learning model indicative of a classification label for the subsequent input.

7. The method further includes using the multimodal discourse tree to navigate the corpus of text, the navigating the corpus of text comprising: accessing the multimodal discourse tree; and determining from the multimodal discourse tree a first basic discourse unit responsive to a query from a user device, the first basic discourse unit corresponding to a first node of a first discourse tree of the multimodal discourse tree, the navigating through the corpus of text further comprising: determining a set of navigation options from the multimodal discourse tree, the set of navigation options including at least two of: (i) a first rhetorical relationship between the first node of the first discourse tree and a second node of the first discourse tree, and (ii) a second rhetorical relationship between the first node and a third node of a second discourse tree of the multimodal discourse tree, or (iii) a third rhetorical relationship between the first node of the first discourse tree and a fourth node of the multimodal discourse tree associated with a corresponding data record, and the step of navigating the corpus of text further includes: presenting at least two of the first, second or third rhetorical relationships at the user device; in response to receiving further user input comprising a selection of the first rhetorical relationship, the second rhetorical relationship, or the third rhetorical relationship; presenting a second basic discourse unit corresponding to the second node based at least in part on determining the selection corresponding to the first rhetorical relationship; presenting a third basic discourse unit corresponding to the third node based at least in part on determining the selection corresponding to the first rhetorical relationship; or and presenting at least a portion of the corresponding data records based at least in part on determining the selection that corresponds to the third rhetorical relationship.

8. 1. A computing device comprising: one or more processors; and one or more memories storing computer-executable instructions for generating a multimodal discourse tree, the instructions, when executed by the one or more processors, causing the computing device to: Obtaining a corpus of text and one or more data records separate from the corpus of text; and generating an extended discourse tree for the corpus of text, the extended discourse tree including a plurality of discourse trees, each discourse tree including a plurality of nodes, each terminal node of the discourse tree corresponding to a fragment of text, each non-terminal node of the discourse tree indicating a rhetorical relationship between the nodes of the discourse trees, the extended discourse tree including further links between the plurality of discourse trees indicating further rhetorical relationships between the nodes of the respective discourse trees. The instructions, when executed by the one or more processors, further cause the computing device to: identifying an entity match between a set of basic discourse units of the plurality of discourse trees and the one or more data records, the entity match being identified by comparing a first entity identified from the basic discourse units and a second entity identified from the data records, the instructions when executed by the one or more processors further causing the computing device to: identifying one or more causal links between two data records of the one or more data records; determining a corresponding rhetorical relationship for each identified entity match and each of the one or more identified causal links; generating a respective node for each identified entity match and each identified causal link for said expanded discourse tree; and creating the multimodal discourse tree by linking the respective nodes generated for each entity match and each causal link to respective nodes of the expanded discourse tree based at least in part on the determined corresponding rhetorical relationships.

9. Performing operations to generate the extended discourse tree further includes causing the computing device to: generating a first discourse tree from a first text of the corpus of text, the first discourse tree corresponding to a first portion of the first text, and performing the operations to generate the extended discourse tree further comprises: generating a second discourse tree from a second text of the corpus of text, the second discourse tree corresponding to a second portion of the second text, and performing the operations to generate the extended discourse tree further comprises:

9. The computing device of claim 8, further comprising: in response to determining a particular rhetorical relationship between basic discourse units of the first discourse tree and the second discourse tree, the computing device is configured to generate links between the first discourse tree and the second discourse tree using the rhetorical relationship.

10. 10. The computing device of claim 9, wherein the first discourse tree and the second discourse tree are communicative discourse trees that include respective verb signatures generated for each basic discourse unit of the first discourse tree and the second discourse tree.

11. 10. The computing device of claim 8, wherein identifying the entity match further causes the computing device to compare the second entity identified from the data record to a third entity identified from a second data record, wherein an entity refers to one of: (i) a person, (ii) a business, (iii) a location, (iv) a name of a document, or (v) a date or time.

12. The computing device of claim 8 , wherein the entity matching further comprises identifying the entity from a predefined ontology.

13. The computing device classifies a subsequent input based at least in part on the multimodal discourse tree, where classifying the subsequent input includes providing to the computing device: generating a training data set including a plurality of multimodal discourse trees, each multimodal discourse tree corresponding to a respective text corpus and a respective set of data records, each multimodal discourse tree associated with a label corresponding to a classification, and classifying the subsequent input further comprises causing the computing device to: training a machine learning model to classify an input based at least in part on the training dataset and a supervised learning algorithm; and generating a corresponding multimodal discourse tree from the subsequent input, the subsequent input including a corresponding text corpus and a corresponding set of data records, wherein classifying the subsequent input comprises causing the computing device to:

10. The computing device of claim 8, further comprising: providing the corresponding multimodal discourse tree generated from the subsequent input as an input to the machine learning model to classify the subsequent input based at least in part on receiving an output from the machine learning model indicative of a classification label for the subsequent input.

14. The computing device navigates the corpus of text using the multimodal discourse tree, and navigating the corpus of text further comprises: accessing the multimodal discourse tree; and determining from the multimodal discourse tree a first basic discourse unit responsive to a query from a user device, the first basic discourse unit corresponding to a first node of a first discourse tree of the multimodal discourse tree, and navigating the corpus of text further comprises: determining a set of navigation options from the multimodal discourse tree, the set of navigation options including at least two of: (i) a first rhetorical relationship between the first node of the first discourse tree and a second node of the first discourse tree, and (ii) a second rhetorical relationship between the first node and a third node of a second discourse tree of the multimodal discourse tree, or (iii) a third rhetorical relationship between the first node of the first discourse tree and a fourth node of the multimodal discourse tree associated with a corresponding data record, and navigating the corpus of text further includes: presenting at least two of the first, second or third rhetorical relationships at the user device; in response to receiving further user input comprising a selection of the first rhetorical relationship, the second rhetorical relationship, or the third rhetorical relationship; presenting a second basic discourse unit corresponding to the second node based at least in part on determining the selection corresponding to the first rhetorical relationship; presenting a third basic discourse unit corresponding to the third node based at least in part on determining the selection corresponding to the first rhetorical relationship; or and presenting at least a portion of the corresponding data records based at least in part on determining the selection that corresponds to the third rhetorical relationship.

15. A computer program for causing one or more processors to execute the method according to any one of claims 1 to 7.

Images