System and method for applying pragmatic principles for interaction with visual analytics - Patents.com

A natural language interface with pragmatics principles enhances data visualization tools by allowing users to interact with complex datasets through regular queries, improving navigation and understanding of data through dynamic visualization updates.

JP7579908B2Active Publication Date: 2024-11-08TABLEAU SOFTWARE INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023076652
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2017-11-06
Filing Date
2023-05-08
Publication Date
2024-11-08
Estimated Expiration
2038-05-03

AI Technical Summary

Technical Problem

Existing data visualization tools struggle with complex datasets, requiring expert modeling or simple queries, and often provide limited interactive visualizations that are difficult to navigate and understand.

Method used

Implementing a natural language interface that applies pragmatics principles to interpret user queries, allowing for interactive query dialogs and dynamic visualization updates through natural language commands, including handling incomplete utterances, referencing entities, and resolving ambiguity.

Benefits of technology

Enables users to efficiently explore and analyze complex datasets using regular language queries, providing dynamic and targeted visual responses, and improving interaction with data visualization applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579908000002
    Figure 0007579908000002
  • Figure 0007579908000003
    Figure 0007579908000003
  • Figure 0007579908000004
    Figure 0007579908000004
Patent Text Reader

Abstract

To provide a method for applying pragmatics principles for interaction with visual analytics, an electronic device, and a storage medium.SOLUTION: A method includes: using natural language processing for visual analysis of a dataset; displaying a data visualization based on a dataset retrieved from a database using a first set of one or more database queries; receiving user input (e.g., keyboard or voice) to specify a natural language command related to the displayed data visualization; based on the displayed data visualization, extracting one or more independent analytic phrases from the natural language command; calculating a set of one or more conversation centers associated with the natural language command based on the set of analytic phrases; calculating a set of analytical functions associated with the set of conversation centers, thereby creating a set of one or more functional phrases; and updating the data visualization based on the set of one or more functional phrases.SELECTED DRAWING: Figure 13A
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001]

[0001] The disclosed implementations relate generally to data visualization, and more specifically to systems, methods, and user interfaces that enable users to interact with and explore datasets using a natural language interface. [Background technology]

[0002]

[0002] Data visualization applications allow users to visually understand data sets, including distributions, trends, outliers, and other factors important to making business decisions. Some data sets are very large or complex, containing many data fields. Various tools, such as dashboards with multiple data visualizations, can be used to help understand and analyze the data. However, within complex user interfaces, some features can be difficult to use or discover. Most systems return only very basic interactive visualizations in response to queries, while others require expert modeling to create effective queries. Other systems require simple closed-ended questions and can only return a single text answer or a static visualization. Summary of the Invention [Problem to be solved by the invention]

[0003]

[0003] Thus, there is a need for tools that allow users to effectively utilize the functionality provided by data visualization applications. One solution to this problem is to provide a natural language interface as part of the data visualization application (e.g., within the user interface for the data visualization application) for an interactive query dialog that provides graphical answers to natural language queries. The natural language interface allows users to access complex functionality through ordinary questions and commands. Questions and insights often arise from previous questions or patterns in the data that humans see. By modeling the interaction as a conversation, the natural language interface can apply pragmatic principles to improve the interaction with the visual analytics. Through various techniques for inferring the grammatical and lexical structure of utterances and their context, the natural language interface supports various pragmatic forms of natural language interaction with the visual analytics. These practical forms include understanding incomplete utterances, referencing entities within the utterances and visualization properties, supporting long and complex utterances, identifying synonyms and related concepts, and "repairing" responses to previous utterances. Additionally, the natural language interface provides appropriate visualization responses within existing visualizations or by creating new visualizations as needed, and resolves ambiguities through targeted text feedback and ambiguity widgets. In this manner, the natural language interface enables users to efficiently explore data displayed within a data visualization application (e.g., in a data visualization). [Means for solving the problem]

[0004]

[0004] According to some implementations, the method is performed on an electronic device having a display. For example, the electronic device can be a smartphone, a tablet, a notebook computer, or a desktop computer. The device displays a data visualization based on a data set retrieved from a database using a first set of one or more database queries. A user specifies a first natural language command associated with the displayed data visualization. Based on the displayed data visualization, the device extracts a first set of one or more independent analytical phrases from the first natural language command. The device then calculates a first set of one or more conversational centers associated with the first natural language command based on the first set of one or more analytical phrases. The device then calculates a first set of analytical functions associated with the first set of one or more conversational centers, thereby creating a first set of one or more function clauses. The device then updates the data visualization based on the first set of one or more function clauses.

[0005]

[0005] In some implementations, the device receives a second natural language command associated with the updated data visualization. According to some implementations, after receiving the second natural language command, the device extracts a second set of one or more independent analytical phrases from the second natural language command and calculates a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analytical phrases. The device then derives a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules. The device calculates a second set of one or more analytical functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function phrases. The device then updates the data visualization based on the second set of one or more function phrases.

[0006]

[0006] In some implementations, each conversation center of the first set of one or more conversation centers, the temporary set of one or more conversation centers, and the second set of one or more conversation centers includes a value of a variable (such as a data attribute or a visualization characteristic). In such implementations, the device uses the transition rule by performing a series of operations, including determining whether the first variable is included in the first set of one or more conversation centers, determining whether the first variable is included in the temporary set of one or more conversation centers, determining a transition rule for each of the one or more transition rules to apply based on whether the first variable is included in the first set of one or more conversation centers and / or the temporary set of one or more conversation centers, and applying each transition rule.

[0007]

[0007] In some implementations, the one or more transition rules used by the device include a CONTINUE rule for including each conversation center in the first set of one or more conversation centers in the second set of one or more conversation centers and for adding one or more conversation centers from the temporary set of one or more conversation centers to the second set of one or more conversation centers.

[0008]

[0008] In some such implementations, the device applies each transition rule by performing a series of operations including, in accordance with a determination that (i) the first variable is included in a temporary set of one or more conversation centers, and (ii) the first variable is not included in the first set of one or more conversation centers, the device applying a CONTINUE rule such that the first variable is included in a second set of one or more conversation centers.

[0009]

[0009] In some implementations, the one or more transition rules used by the device include a RETAIN rule for retaining each conversation center in the first set of one or more conversation centers in the second set of one or more conversation centers without adding a conversation center from the temporary set of one or more conversation centers to the second set of one or more conversation centers.

[0010]

[0010] In some such implementations, the device applies each transition rule by performing a series of operations including applying a RETAIN rule to include each conversation center of the first set of one or more conversation centers in the second set of one or more conversation centers in accordance with a determination that (i) the first variable is included in the first set of one or more conversation centers and (ii) the first variable is not included in the temporary set of one or more conversation centers.

[0011]

[0011] In some implementations, the one or more transition rules used by the device include a SHIFT rule for including each conversation center of a first set of one or more conversation centers in a second set of one or more conversation centers and replacing one or more conversation centers of the second set of one or more conversation centers with a conversation center of the temporary set of one or more conversation centers.

[0012]

[0012] In some such implementations, the device applies each transition rule by performing a series of operations including: (i) determining whether a first value of the first variable in the first set of one or more conversation centers differs from a second value of the first variable in the temporary set of one or more conversation centers in accordance with a determination that the first variable is included in the first set of one or more conversation centers; and, in accordance with a determination that the first value differs from the second value, including each conversation center in the first set of one or more conversation centers in a second set of one or more conversation centers and applying a SHIFT rule to replace the value of the first variable in the second set of one or more conversation centers with the second value.

[0013]

[0013] In some such implementations, the device further determines whether a widget corresponding to the first variable has been deleted by a user, and, in accordance with a determination that the widget has been deleted, applies a SHIFT rule to include each conversation center in the first set of one or more conversation centers in a second set of one or more conversation centers, and to replace the value of the first variable in the second set of one or more conversation centers with a new value (e.g., a maximum value, or a superset value) that includes the first value.

[0014]

[0014] In some implementations, the device creates a first set of one or more queries based on a first set of one or more function clauses, re-queries the database using the first set of one or more queries, thereby retrieving a second data set, and then displays an updated data visualization using the second data set. In some implementations, the device creates a second set of one or more queries based on a second set of one or more function clauses, re-queries the database using the second set of one or more queries, thereby retrieving a third data set, and then displays an updated data visualization using the third data set. In some cases, the re-querying of the database is performed locally on the computing device using data cached or stored on the computing device. For example, if one or more filters are specified in the natural language command, the re-querying is typically performed locally. In some implementations, the device further creates and displays a new data visualization using the second data set or the third data set.

[0015]

[0015] In some implementations, the device further determines whether the user has selected a dataset different from the first dataset or whether the user has reset the visualization of the data, and if so, resets each of the first set of one or more conversation centers, the temporary set of one or more conversation centers, and the second set of one or more conversation centers to an empty set that does not include a conversation center.

[0016]

[0016] Typically, the electronic device includes one or more processors, a memory, a display, and one or more programs stored in the memory. The programs are configured for execution by the one or more processors and configured to perform any of the methods described herein. The one or more programs include instructions for displaying a data visualization based on a first data set retrieved from a database using a first set of one or more queries. The one or more programs also include instructions for receiving a first user input for specifying a first natural language command associated with the data visualization. The one or more programs also include instructions for extracting a first set of one or more independent analytical phrases from the first natural language command. The one or more programs also include instructions for calculating a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The one or more programs also include instructions for calculating a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses, and updating the data visualization based on the first set of one or more function clauses.

[0017]

[0017] In some implementations, the one or more programs include instructions for receiving a second user input to specify a second natural language command associated with the updated data visualization. The one or more programs also include instructions for extracting a second set of one or more independent analytical phrases from the second natural language command. The one or more programs also include instructions for calculating a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analytical phrases. The one or more programs also include instructions for deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules. The one or more programs also include instructions for calculating a second set of one or more analytical functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function phrases, and updating the data visualization based on the second set of one or more function phrases.

[0018]

[0018] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured for execution by a computing device having one or more processors, a memory, and a display. The one or more programs are configured to perform any of the methods described herein. The one or more programs include instructions for displaying a data visualization based on a first data set retrieved from a database using a first set of one or more queries. The one or more programs also include instructions for receiving a first user input for specifying a first natural language command associated with the data visualization. The one or more programs also include instructions for extracting a first set of one or more independent analytical phrases from the first natural language command. The one or more programs also include instructions for calculating a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The one or more programs also include instructions for calculating a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses, and updating the data visualization based on the first set of one or more function clauses.

[0019]

[0019] In some implementations, the one or more programs include instructions for receiving a second user input to specify a second natural language command associated with the updated data visualization. The one or more programs also include instructions for extracting a second set of one or more independent analytical phrases from the second natural language command. The one or more programs also include instructions for calculating a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analytical phrases. The one or more programs also include instructions for deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules. The one or more programs also include instructions for calculating a second set of one or more analytical functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function phrases, and updating the data visualization based on the second set of one or more function phrases.

[0020]

[0020] In another aspect, according to some implementations, the device displays a data visualization based on a data set retrieved from a database using a first set of one or more database queries. The user specifies a first natural language command associated with the displayed data visualization. Based on the displayed data visualization, the device extracts a first set of one or more independent analytical phrases from the first natural language command. The device then calculates a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The device then calculates a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function phrases. The device then updates the data visualization based on the first set of one or more function phrases. The user specifies a second natural language command associated with the updated data visualization. After receiving the second natural language command, the device extracts a second set of one or more independent analytical phrases from the second natural language command and calculates a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analytical phrases. The device then calculates cohesion between the first set of one or more analytic phrases and the second set of one or more analytic phrases. The device then derives a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers based on the cohesion. The device calculates a second set of one or more analytic functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function phrases. The device then updates a visualization of the data based on the second set of one or more function phrases.

[0021]

[0021] In some implementations, the device calculates cohesion and derives the second set of one or more conversation centers by performing a series of operations including identifying a phrase structure of a second set of one or more analysis phrases, identifying one or more pragmatic forms based on the phrase structure, and deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers based on the one or more pragmatic forms.

[0022] In some such implementations, the apparatus identifies the phrase structure by performing a series of operations including applying a probabilistic grammar to parse the second natural language command, thereby obtaining a parsed output, and decomposing the parsed output into corresponding categorical attributes and data attributes. In some such implementations, parsing the second natural language command further includes inferring the syntactic structure by using a part-of-speech API (e.g., a part-of-speech (PS) tagger) provided by the natural language toolkit.

[0023]

[0023] In some implementations, the device identifies one or more pragmatics by performing a series of operations including determining whether the second natural language command is an incomplete utterance (sometimes referred to as an ellipsis) by determining whether one or more linguistic elements are not present in the phrase structure. In some such implementations, the device derives the second set of one or more conversation centers by performing a series of operations including determining a first subset of conversation centers of a first set of one or more conversation centers according to a determination that the second natural language command is an incomplete utterance, where the first subset of conversation centers corresponding to the one or more linguistic elements are not present in the phrase structure, and calculating a second set of one or more conversation centers by combining the temporary set of one or more conversation centers with the first subset of conversation centers.

[0024]

[0024] In some implementations, the device identifies one or more forms of pragmatics by performing a series of operations including determining whether the second natural language command is a referring expression by determining whether one or more anaphoric references are present in the phrase structure, and the device derives a second set of one or more conversation centers by performing another series of operations including searching a first set of one or more conversation centers to find a first subset of conversation centers corresponding to phrase chunks of the second natural language command that include the first anaphoric reference of the one or more anaphoric references in accordance with the determination that the second natural language command is a referring expression, and calculating a second set of one or more conversation centers based on the temporary set of one or more conversation centers and the first subset of conversation centers.

[0025]

[0025] In some such implementations, the device further determines whether the first anaphoric reference involves a verb of the second natural language command, and if so, searches a first set of one or more conversation centers to find a first action conversation center that references the action verb (e.g., "filter"), and calculates a second set of one or more conversation centers based on the temporary set of one or more conversation centers, the first subset of conversation centers, and the first action conversation center.

[0026]

[0026] In some such implementations, the device determines whether the first anaphoric reference is a strict reference that indicates some object of the environment, usually by pointing out, and if so, calculates a second set of one or more conversation centers based on the temporary set of one or more conversation centers and the characteristics of the object. The strict reference is usually enabled through multimodal interaction (e.g., by using a mouse in addition to speech or text).

[0027]

[0027] In some such implementations, the device determines whether the first anaphoric reference indicates a visualization characteristic of a visualization of the updated data, and if so, calculates a second set of one or more conversation centers based on the temporary set of one or more conversation centers and data related to the visualization characteristic.

[0028]

[0028] In some implementations, the device identifies one or more word forms by performing a series of operations including determining whether the second natural language command is a repair utterance by determining whether the phrase structure corresponds to one or more predefined repair utterances (e.g., to repair potential ambiguity of the first natural language command or how the results are presented to the user). For example, the user utters "except the condo" or "change the condo to a townhome". In such implementations, if the device determines that the second natural language command is a repair utterance, the device calculates a second set of one or more conversation centers based on the temporary set of one or more conversation centers, and updates one or more data attributes in the second set of one or more conversation centers based on the one or more predefined repair utterances and phrase structure.

[0029]

[0029] In some such implementations, the device determines whether the phrase structure corresponds to a repair utterance for changing a default behavior associated with displaying the data visualization (e.g., highlighting to select a response such as "no filter, instead"), and if so, the device changes the default behavior with respect to the display.

[0030]

[0030] In some implementations, the device identifies the one or more forms of pragmatics by performing a series of operations including determining whether the second natural language command is a conjunctive expression by (i) determining an explicit or implicit presence of a conjunction in the phrase structure, and (ii) determining whether the temporary set of one or more conversation centers includes each conversation center of the first set of one or more conversation centers. In such implementations, the device derives the second set of one or more conversation centers by performing a series of operations including calculating a second set of one or more conversation centers based on the temporary set of one or more conversation centers according to a determination that the second natural language command is a conjunctive expression.

[0031]

[0031] In some such implementations, the device determines whether the second natural language command has more than one conjunction, and calculates a second set of one or more analytic functions by linearizing the second natural language command according to the determination that the second natural language command has more than one conjunction. In some such implementations, the device linearizes the second natural language command by performing a series of operations including: generating a parse tree for the second natural language command; traversing the parse tree in a post-order manner to extract a first analytic phrase and a second analytic phrase, where the first analytic phrase and the second analytic phrase are adjacent nodes in the parse tree; calculating first analytic functions and second analytic functions corresponding to the first analytic phrase and the second analytic phrase, respectively; and combining the first analytic function with the second analytic function by applying one or more logical operators based on one or more characteristics of the first analytic function and the second analytic function, where the one or more characteristics include an attribute type, an operator type, and a value.

[0032]

[0032] In some such implementations, the first analytical function includes a first attribute (sometimes referred to herein as a variable and including a visualization characteristic), a first operator, and a first value, and the second analytical function includes a second attribute (sometimes referred to herein as a variable and including a visualization characteristic), a second operator, and a second value.

[0033]

[0033] In some such implementations, combining the first analytic function and the second analytic function includes determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute, determining whether the first attribute and the second attribute are identical and whether the first attribute and the second attribute are identical, and applying a union operator to combine the first analytic function and the second analytic function in accordance with the determination that both are categorical attributes.

[0034]

[0034] In some such implementations, combining the first analytic function and the second analytic function includes determining whether the first attribute is a categorical attribute or an ordinal attribute, determining whether the second attribute is a categorical attribute or an ordinal attribute, determining whether the first attribute and the second attribute are identical, and applying a cross operator to combine the first analytic function and the second analytic function in accordance with a determination that the first attribute and the second attribute are not identical.

[0035]

[0035] In some such implementations, combining the first analytic function and the second analytic function includes determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute, determining that the first attribute and the second attribute are identical, determining operator types of the first operator and the second operator in accordance with a determination that the first attribute and the second attribute are identical and both are ordinal attributes, and applying a union operator to combine the first analytic function and the second analytic function in accordance with a determination that the first operator and the second operator are both equality operators.

[0036]

[0036] In some such implementations, combining the first analytic function and the second analytic function includes determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute, determining that the first attribute and the second attribute are identical, determining that the first attribute and the second attribute are identical, determining operator types of the first operator and the second operator in accordance with a determination that the first attribute and the second attribute are identical and are both ordinal attributes, determining whether the first value is less than the second value in accordance with a determination that the first operator is a "relatively smaller" operator and the second operator is a "relatively larger" operator, and applying a union operator to combine the first analytic function and the second analytic function in accordance with a determination that the first value is less than the second value.

[0037]

[0037] In some such implementations, combining the first analytic function and the second analytic function includes determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute, determining that the first attribute and the second attribute are identical, determining that the first attribute and the second attribute are identical, determining operator types of the first operator and the second operator in accordance with a determination that the first attribute and the second attribute are identical and are both ordinal attributes, determining whether the first value is less than the second value in accordance with a determination that the first operator is a "relatively greater" operator and the second operator is a "relatively small" operator, and applying a cross operator to combine the first analytic function and the second analytic function in accordance with a determination that the first value is less than the second value.

[0038]

[0038] In some implementations, the apparatus further calculates semantic associations between a second set of the one or more extracted analytic phrases and one or more attributes of the data included in the updated data visualization, and calculates analytic functions associated with the second set of the one or more analytic phrases, thereby creating a second set of one or more function phrases based on the semantically related attributes of the data. In contrast to grammatical cohesion or inter-context cohesion, lexical cohesion looks for cohesion within a context.

[0039]

[0039] In some such implementations, the apparatus calculates the semantic relevance by performing a series of operations including training a first neural network model on a large corpus of text, thereby learning word embeddings; using a second neural network model to calculate a first word vector for a first word of a first phrase of a second set of one or more analysis phrases, the first word vector mapping the first word to the word embeddings; using the second neural network model to calculate a second word vector for a first data attribute of the one or more data attributes, the second word vector mapping the first data attribute to the word embeddings; and calculating the relevance between the first word vector and the second word vector using a similarity metric.

[0040] In some such implementations, the first neural network model includes a Word2vec™ model. In some such implementations, the second neural network model includes a recurrent neural network model.

[0041]

[0041] In some such implementations, the similarity metric is based on at least (i) the Woo-Palmer distance between the meanings of words associated with the first word vector and the second word vector, (ii) a weighting factor, and (iii) a pairwise cosine distance between the first word vector and the second word vector.

[0042]

[0042] In some such implementations, the device calculates the analysis functions by performing a series of operations including obtaining definitions of words of a second set of one or more analysis phrases from a publicly available dictionary, determining whether the word definitions include one or more predefined definitions while using a part-of-speech API provided by the natural language toolkit, and mapping the one or more predefined adjectives to one or more analysis functions in accordance with a determination that the word definition includes one or more predefined adjectives.

[0043]

[0043] Typically, the electronic device includes one or more processors, a memory, a display, and one or more programs stored in the memory. The programs are configured for execution by the one or more processors and configured to perform any of the methods described herein. The one or more programs include instructions for displaying a data visualization based on a first data set retrieved from a database using a first set of one or more queries. The one or more programs also include instructions for receiving a first user input for specifying a first natural language command associated with the data visualization. The one or more programs also include instructions for extracting a first set of one or more independent analytical phrases from the first natural language command. The one or more programs also include instructions for calculating a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The one or more programs also include instructions for calculating a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses. The one or more programs also include instructions for updating the data visualization based on the first set of one or more function clauses. The one or more programs also include instructions for receiving a second user input to specify a second natural language command associated with the visualization of the updated data. The one or more programs also include instructions for extracting a second set of one or more independent analytical phrases from the second natural language command. The one or more programs also include instructions for calculating a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analytical phrases. The one or more programs also include instructions for calculating cohesion between the first set of one or more analytical phrases and the second set of one or more analytical phrases.The one or more programs also include instructions for deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers based on the cohesion. The one or more programs also include instructions for calculating a second set of one or more analytic functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function clauses, and updating a visualization of the data based on the second set of one or more function clauses.

[0044]

[0044] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured for execution by a computing device having one or more processors, a memory, and a display. The one or more programs are configured to perform any of the methods described herein. The one or more programs include instructions for displaying a data visualization based on a first data set retrieved from a database using a first set of one or more queries. The one or more programs also include instructions for receiving a first user input for specifying a first natural language command associated with the data visualization. The one or more programs also include instructions for extracting a first set of one or more independent analytical phrases from the first natural language command. The one or more programs also include instructions for calculating a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The one or more programs also include instructions for calculating a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses. The one or more programs also include instructions for updating the data visualization based on the first set of one or more function clauses. The one or more programs also include instructions for receiving a second user input to specify a second natural language command associated with the visualization of the updated data. The one or more programs also include instructions for extracting a second set of one or more independent analytical phrases from the second natural language command. The one or more programs also include instructions for calculating a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analytical phrases. The one or more programs also include instructions for calculating cohesion between the first set of one or more analytical phrases and the second set of one or more analytical phrases.The one or more programs also include instructions for deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers based on the cohesion. The one or more programs also include instructions for calculating a second set of one or more analytic functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function clauses, and updating a visualization of the data based on the second set of one or more function clauses.

[0045]

[0045] In another aspect, according to some implementations, the method is performed on an electronic device having a display. For example, the electronic device can be a smartphone, a tablet, a notebook computer, or a desktop computer. The device displays a data visualization based on a data set retrieved from a database using a first set of one or more database queries. A user specifies a first natural language command associated with the displayed data visualization. Based on the displayed data visualization, the device extracts a first set of one or more independent analytical phrases from the first natural language command. The device then calculates a first set of one or more conversational centers associated with the first natural language command based on the first set of one or more analytical phrases. The device then calculates a first set of analytical functions associated with the first set of one or more conversational centers, thereby creating a first set of one or more function clauses. The device then updates the data visualization based on the first set of one or more function clauses. The user specifies a second natural language command associated with the updated data visualization. After receiving the second natural language command, the device extracts a second set of one or more independent analysis phrases from the second natural language command and calculates a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analysis phrases. The device then derives a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules. The device then updates a visualization of the data based on the second set of one or more conversation centers.

[0046]

[0046] In some implementations, the device determines one or more data attributes corresponding to a second set of one or more conversation centers, scans the displayed data visualizations to identify one or more of the displayed data visualizations that include a mark of data whose characteristics correspond to a first data attribute of the one or more data attributes, and highlights the mark of data whose characteristics correspond to the first data attribute. In some such implementations, the device further filters results from the displayed data visualization that include a mark of data whose characteristics do not correspond to the one or more data attributes. Furthermore, in some such implementations, the device receives user input (e.g., via natural language commands such as "exclude," "remove," and "filter only") that determines whether to filter or highlight the mark of data.

[0047] In some implementations, the visualization characteristics include one or more of a color, a size, and a shape. In some implementations, the visualization characteristics correspond to a visual encoding of the mark of the data. In some implementations, the visual encoding is one or more of a color, a size, and a shape.

[0048]

[0048] In some implementations, the device determines whether any of the displayed data visualizations include a mark of data whose characteristics correspond to the first data attribute, and if so, generates details of a new data visualization using the first data attribute (e.g., aggregation type) and displays the new data visualization. In some such implementations, displaying the new data visualization further includes determining a chart type based on the details, and generating and displaying the chart. Further, in some such implementations, the chart is arranged using a two-dimensional grid-based layout algorithm that is automatically coordinated with other data visualizations (sometimes referred to herein as views).

[0049]

[0049] In some implementations, the device further performs a set of operations including: calculating a second set of one or more analytical functions associated with a second set of one or more conversation centers, thereby creating a second set of one or more function clauses; selecting a first function clause from the second set of one or more function clauses, the first function clause including a parameterized data selection criterion; selecting an initial range for a value of a parameter of the parameterized data selection criterion; displaying an editable user interface control (e.g., a widget) corresponding to the parameterized data selection criterion, the user interface control displaying a current value of the parameter; and ordering the displayed set of one or more editable user interface controls based on a query order of the second natural language command, the query order being inferred while extracting the second set of one or more analytical clauses from the second natural language command. In some such implementations, the user interface control allows for adjustment of the first function clause. Further, in some such implementations, the user interface control displays a slider that allows a user to adjust the first function clause. In some such implementations, ordering the displayed set of one or more editable user interface controls further includes using a library that facilitates compact arrangement of small word-scale visualizations within the text. In some such implementations, the library is Sparklificator™.

[0050]

[0050] In some implementations, the device executes a series of operations aimed at automatically correcting some user errors. The series of operations further includes: determining a first token of the second natural language command that does not correspond to any of the analytic phrases of the second set of one or more analytic phrases (e.g., due to a parsing failure); searching for a correctly spelled term corresponding to the first token using a search library by comparing the first token with one or more features of the first dataset; and replacing the first token of the second natural language command with the correctly spelled term to obtain a third natural language command; and extracting the second set of one or more analytic phrases from the third natural language command. In some such implementations, the one or more features include data attributes, cell values, and related keywords of the first dataset. In some such implementations, the search library is a fuzzy string library such as Fuse.js™.

[0051]

[0051] In some such implementations, the apparatus further performs a set of operations including determining whether there is a correctly spelled term corresponding to the first token, and, in accordance with a determination that there is no correctly spelled term corresponding to the first token, parsing the second natural language command to obtain a parse tree, pruning the parse tree to remove a portion of the tree that corresponds to the first token, and extracting a second set of one or more analytic phrases based on the pruned parse tree.

[0052] In some implementations, the device further generates textual feedback indicating that the first token was not recognized and therefore removed from the second natural language command - a situation that typically occurs when an utterance is only partially understood. In some such implementations, the device displays the first token.

[0053]

[0053] In some implementations, the device further generates text feedback indicating that the correctly spelled term is replaced with the first token of the second natural language command. This is typically the situation when the utterance was not successfully understood but the device presents an alternative query. Furthermore, in some such implementations, the device displays and highlights the correctly spelled term.

[0054]

[0054] Typically, the electronic device includes one or more processors, a memory, a display, and one or more programs stored in the memory. The programs are configured for execution by the one or more processors and configured to perform any of the methods described herein. The one or more programs include instructions for displaying a data visualization based on a first data set retrieved from a database using a first set of one or more queries. The one or more programs also include instructions for receiving a first user input for specifying a first natural language command associated with the data visualization. The one or more programs also include instructions for extracting a first set of one or more independent analytical phrases from the first natural language command. The one or more programs also include instructions for calculating a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The one or more programs also include instructions for calculating a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses. The one or more programs also include instructions for updating the data visualization based on the first set of one or more function clauses. The one or more programs also include instructions for receiving a second user input to specify a second natural language command associated with the visualization of the updated data. The one or more programs also include instructions for extracting a second set of one or more independent analysis phrases from the second natural language command. The one or more programs also include instructions for calculating a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analysis phrases. The one or more programs also include instructions for deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules.The one or more programs also include instructions for updating the data visualization based on a second set of one or more conversation centers, where the updating includes determining one or more data attributes corresponding to the second set of one or more conversation centers, scanning the displayed data visualizations to identify one or more of the displayed data visualizations that include a mark of data whose characteristics correspond to a first data attribute of the one or more data attributes, and highlighting the mark of data whose characteristics correspond to the first data attribute.

[0055]

[0055] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured for execution by a computing device having one or more processors, a memory, and a display. The one or more programs are configured to perform any of the methods described herein. The one or more programs include instructions for displaying a data visualization based on a first data set retrieved from a database using a first set of one or more queries. The one or more programs also include instructions for receiving a first user input for specifying a first natural language command associated with the data visualization. The one or more programs also include instructions for extracting a first set of one or more independent analytical phrases from the first natural language command. The one or more programs also include instructions for calculating a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analytical phrases. The one or more programs also include instructions for calculating a first set of analytical functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses. The one or more programs also include instructions for updating the data visualization based on the first set of one or more function clauses. The one or more programs also include instructions for receiving a second user input to specify a second natural language command associated with the visualization of the updated data. The one or more programs also include instructions for extracting a second set of one or more independent analysis phrases from the second natural language command. The one or more programs also include instructions for calculating a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analysis phrases. The one or more programs also include instructions for deriving a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules.The one or more programs also include instructions for updating the data visualization based on a second set of one or more conversation centers, where the updating includes determining one or more data attributes corresponding to the second set of one or more conversation centers, scanning the displayed data visualizations to identify one or more of the displayed data visualizations that include a mark of data whose characteristics correspond to a first data attribute of the one or more data attributes, and highlighting the mark of data whose characteristics correspond to the first data attribute.

[0056]

[0056] Thus, disclosed are methods, systems, and graphical user interfaces that enable a user to efficiently explore data displayed within a data visualization application.

[0057]

[0057] Both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

[0058]

[0058] To better understand the systems, methods, and graphical user interfaces described above, as well as additional systems, methods, and graphical user interfaces for providing visualization analysis of data, reference should be made to the following implementations in conjunction with the subsequent drawings, in which like reference numerals indicate corresponding parts throughout. [Brief description of the drawings]

[0059] [Figure 1]

[0059] A graphical user interface used in some implementations is shown. [Diagram 2]

[0060] FIG. 1 is a block diagram of a computing device according to some implementations. [Figure 3A]

[0061] 1 is a process flow diagram illustrating a process for using natural language applying pragmatics principles for visual analysis of a dataset, according to some implementations. [Figure 3B] A state machine diagram illustrating the states of a conversation center and the transitions between states when certain transition rules are triggered, according to some implementations. [Figure 4A]

[0062] FIG. 1 illustrates the use of different transition rules when processing a conversation to be analyzed, according to some implementations. [Figure 4B] FIG. 1 illustrates the use of different transition rules when processing a conversation to be analyzed, according to some implementations. [Figure 4C] FIG. 4A illustrates a graphical user interface (related to FIGS. 4A-4B) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 4D] FIG. 4A illustrates a graphical user interface (related to FIGS. 4A-4B) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 4E] FIG. 4A illustrates a graphical user interface (related to FIGS. 4A-4B) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 4F] FIG. 4A illustrates a graphical user interface (related to FIGS. 4A-4B) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 4G] FIG. 4A illustrates a graphical user interface (related to FIGS. 4A-4B) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Diagram 5]

[0063] FIG. 1 illustrates a general framework for the application of pragmatic principles for visual analysis according to some implementations. [Figure 6A]

[0064] FIG. 1 illustrates the application of pragmatic principles to incomplete utterances (sometimes referred to herein as ellipsis) according to some implementations. [Figure 6B] FIG. 6B illustrates a graphical user interface (related to FIG. 6A ) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 6C] FIG. 6B illustrates a graphical user interface (related to FIG. 6A ) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 6D] FIG. 6B illustrates a graphical user interface (related to FIG. 6A ) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 7A]

[0065] A diagram illustrating the application of pragmatic principles to utterances involving referring expressions (sometimes referred to herein as anaphoric references) according to some implementations. [Figure 7B] FIG. 7B illustrates a graphical user interface (related to FIG. 7A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 7C] FIG. 7B illustrates a graphical user interface (related to FIG. 7A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 7D] FIG. 7B illustrates a graphical user interface (related to FIG. 7A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 7E] FIG. 7B illustrates a graphical user interface (related to FIG. 7A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 7F] FIG. 7B illustrates a graphical user interface (related to FIG. 7A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 8A]

[0066] FIG. 1 illustrates the application of pragmatic principles to utterances with conjunctions according to some implementations. [Figure 8B] 8A shows a graphical user interface (with reference to FIG. 8A ) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 8C] 1 illustrates how the system iteratively connects analytic functions of adjacent nodes of a parse tree through linearization, according to some implementations. [Figure 9A]

[0067] A diagram illustrating the application of pragmatic principles to process lexical cohesion, according to some implementations. [Figure 9B] FIG. 9B illustrates a graphical user interface (related to FIG. 9A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 9C] FIG. 9B illustrates a graphical user interface (related to FIG. 9A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 9D] FIG. 9B illustrates a graphical user interface (related to FIG. 9A) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 10A]

[0068] FIG. 1 illustrates the application of pragmatic principles for repair utterances, according to some implementations. [Figure 10B]FIG. 10B illustrates a graphical user interface (related to FIG. 10A ) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 10C] FIG. 10B illustrates a graphical user interface (related to FIG. 10A ) for interactive data analysis using natural language processing in data visualization applications according to some implementations. [Figure 11A]

[0069] FIG. 1 illustrates the application of pragmatic principles to process responses and feedback according to some implementations. [Figure 11B] FIG. 11B is a data visualization diagram further illustrating the methodology shown in FIG. 11A. [Figure 11C] A diagram illustrating how a dashboard is progressively built based on input speech according to some implementations. [Figure 12A]

[0070] 1 illustrates a collection of widgets generated to handle ambiguity in user queries, according to some implementations. [Figure 12B] 13 illustrates examples of feedback for various situations according to some implementations. [Figure 13A]

[0071] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13B] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13C] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13D] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13E]1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13F] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13G] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13H] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13I] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 13J] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset applying principles of pragmatics, according to some implementations. [Figure 14A]

[0072] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14B] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14C] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14D] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14E] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14F] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14G] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14H] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14I] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14J] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14K] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14L] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14M] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14N]1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14O] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14P] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14Q] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 14R] 1 provides a flowchart of a process for using natural language for visual analysis of a dataset that applies principles of pragmatics, including processing of various forms of pragmatics, according to some implementations. [Figure 15A]

[0073] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15B] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15C] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15D]Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15E] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15F] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15G] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. [Figure 15H] Provides a flowchart of a process using natural language for visual analysis of a dataset that applies pragmatic principles, including for processing responses and feedback, and for handling ambiguity in a user's query, according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0060]

[0074] Reference will now be made to implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without the need for these specific details.

[0061]

[0075] FIG. 1 illustrates a graphical user interface 100 for interactive data analysis. The user interface 100 includes a data tab 114 and an analysis tab 116, according to some implementations. When the data tab 114 is selected, the user interface 100 displays a schema information area 110, also referred to as a data pane. The schema information area 110 provides named data elements (e.g., field names) that can be selected and used to build a visualization of the data. In some implementations, the list of field names is divided into groups of dimensions (e.g., categorical data) and groups of measurements (e.g., quantities). Some implementations also include a list of parameters. When the analysis tab 116 is selected, the user interface displays a list of analysis functions instead of data elements (not shown).

[0062]

[0076] The graphical user interface 100 also includes a data visualization area 112. The data visualization area 112 includes multiple shelf areas, such as a column shelf area 120 and a row shelf area 122, which are also referred to as the column shelf 120 and the row shelf 122. As shown here, the data visualization area 112 also has a large space for displaying visual graphics (also referred to herein as data visualizations). The space is initially empty of visual graphics because no data elements have been selected yet. In some implementations, the data visualization area 112 has multiple layers, called sheets.

[0063]

[0077] In some implementations, the graphical user interface 100 also includes a natural language processing area 124. The natural language processing area 124 includes an input bar (also referred to herein as a command bar) for receiving natural language commands. A user can interact with the input bar to provide a command. For example, a user can type a command into the input bar to provide a command. Additionally, a user may indirectly interact with the input bar by speaking into a microphone (e.g., voice input device 220) to provide a command. In some implementations, data elements are initially associated with the columns shelf 120 and rows shelf 122 (e.g., using a drag-and-drop operation from the schema information area 110 to the columns shelf 120 and / or rows shelf 122). After the initial association, a user can use natural language commands (e.g., in the natural language processing area 124) to further explore the visualization of the displayed data. In some instances, a user uses the natural language processing area 124 to create an initial association, which results in one or more data elements being placed on the columns shelf 120 and rows shelf 122. For example, a user may provide a command to create a relationship between data element X and data element Y. In response to receiving the command, the column shelf 120 and the row shelf 122 may be prepopulated with data elements (e.g., the column shelf 120 may be prepopulated with data element X and the row shelf 122 may be prepopulated with data element Y, or vice versa).

[0064]

[0078] FIG. 2 is a block diagram illustrating a computing device 200 capable of displaying a graphical user interface 100 according to some implementations. Various examples of computing devices 200 include desktop computers, laptop computers, tablet computers, and other computing devices having a display and a processor capable of executing a data visualization application 230. The computing device 200 typically includes one or more processing units (processors or cores) 202, one or more network or other communication interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components. The communication bus 208 optionally includes circuitry (also called a chipset) that interconnects and controls communication between the system components. The computing device 200 includes a user interface 210. The user interface 210 typically includes a display device 212. In some implementations, the computing device 200 includes input devices such as a keyboard, a mouse, and / or other input buttons 216. Alternatively, or in addition, in some implementations, the display device 212 includes a touch-sensitive surface 214, in which case the display device 212 is a touch-sensitive display. In some implementations, the touch-sensitive surface 214 is configured to detect various swipe gestures (e.g., vertical and / or horizontal continuous gestures) and / or other gestures (e.g., single / double taps). In computing devices with a touch-sensitive display 214, a physical keyboard is optional (e.g., a soft keyboard may be displayed when keyboard input is required). The user interface 210 also includes an audio output device 218, such as a speaker, or an audio output connection connected to a speaker, earphones, or headphones. Additionally, some computing devices 200 use a microphone and voice recognition to complement or replace the keyboard.Optionally, computing device 200 includes an audio input device 220 (eg, a microphone) for capturing audio (eg, speech from a user).

[0065]

[0079] Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. It may also include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 206 includes one or more storage devices located remotely from processor 202. Memory 206, or a non-volatile memory device within memory 206, includes a non-transitory computer-readable storage medium. In some implementations, memory 206 or a computer-readable storage medium of memory 206 stores the following programs, modules, and data structures, or a subset or superset thereof: An operating system 222, which contains the procedures for handling various basic system services and performing hardware-dependent tasks. A communications module 224, which is used to connect the computing device 200 to other computers and devices via one or more communications network interfaces 204 (wired or wireless), such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. · A web browser 226 (or other application capable of displaying web pages), which allows a user to communicate with remote computers or devices over the network. An audio input module 228 (e.g., a microphone module) for processing audio captured by the audio input device 220. The captured audio may be transmitted to a remote server and / or processed by an application (e.g., a data visualization application 230) running on the computing device 200. a data visualization application 230 for generating visualizations of data and associated features. The application 230 includes a graphical user interface 232 (e.g., the graphical user interface 100 shown in FIG. 1 ) through which a user constructs a visual graphic. For example, the user selects one or more data sources 240 (which may be stored on the computing device 200 or stored remotely), selects data fields from the data sources, and defines the visual graphic using the selected fields; and Zero or more databases or data sources 240 (e.g., a first data source 240-1 and a second data source 240-2) used by the data visualization application 230. In some implementations, the data sources are stored as spreadsheet files, CSV files, text files, JSON files, XML files, flat files, or stored in a relational database.

[0066]

[0080] In some implementations, the data visualization application 230 includes a data visualization generation module 234 that receives user input (e.g., visual details 236) and generates a corresponding visual graphic. The data visualization application 230 then displays the generated visual graphic in the user interface 232. In some implementations, the data visualization application 230 runs as a standalone application (e.g., a desktop application). In some implementations, the data visualization application 230 runs within another application (e.g., a server-based application) using a web page provided by the web browser 226 or a web server.

[0067]

[0081] In some implementations, information provided by the user (e.g., user input) is stored as visual details 236. In some implementations, visual details 236 include previous natural language commands received from the user or characteristics specified by the user through natural language commands.

[0068]

[0082] In some implementations, the data visualization application 230 includes a language processing module 238 for processing (e.g., interpreting) commands provided by a user of the computing device. In some implementations, the commands are natural language commands (e.g., captured by the voice input device 220). In some implementations, the language processing module 238 includes sub-modules, such as an auto-complete module, a pragmatics module, and an ambiguity module, each of which is described in further detail below.

[0069]

[0083] In some implementations, the memory 206 stores the metrics and / or scores determined by the language processing module 238. Additionally, the memory 206 may store thresholds and other criteria that are compared to the metrics and / or scores determined by the language processing module 238. For example, the language processing module 238 may determine a relevance metric (described in more detail below) for the analyzed phrases of the received command. The language processing module 238 may then compare the relevance metric to a threshold stored in the memory 206.

[0070]

[0084] The above-identified executable modules, applications, or sequences of procedures may be stored in one or more of the aforementioned memory devices and correspond to sequences of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise reconfigured in various implementations. In some implementations, memory 206 stores a subset of the above-identified modules and data structures. Additionally, memory 206 may store additional modules or data structures not described above.

[0071]

[0085] 2 illustrates a computing device 200, which is intended as a functional illustration of various features that may be present, rather than as a structural schematic of the implementations described herein. In practice, and as will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated.

[0072]

[0086] Figure 3A is a process flow diagram showing a process or framework for using natural language that applies the principles of pragmatics for visual analysis of a dataset. The framework based on the conversational dialogue model extends the centering approach adopted in pragmatic theory to support the transitional states between sentences that continue, hold, and shift the context of data attributes during playback. This framework supports an interface for the "visual analysis cycle", i.e., the fluid and iterative exploration and refinement in visual analysis. The dialogue with visual analysis is most effective when the user can focus on answering questions about the data rather than on how to operate the interface to the analysis tool. Pragmatics is particularly important for the visual analysis flow where questions and insights often arise from previous questions and patterns in the data that people see.

[0073]

[0087] A series of utterances that show coherence form a conversation. Coherence is a semantic property of a conversation based on the interpretation of each individual utterance with respect to the interpretation of other utterances. As described above, to correctly interpret a set of utterances, the process framework utilizes and extends a model commonly used in the discourse structure called conversational centering according to some implementations. In this model, utterances are divided into constituent discourse segments, and the relationships that can hold between two segments are embedded. The center refers to an entity that helps link that utterance to other utterances in the conversation. Consider the discourse segment DS of utterances U1,..., U m . Each utterance U n (1 ≦ n < m) in DS is assigned a series of forward centers C f (U n , DS) that indicate the current focus of the conversation. For each utterance other than the first utterance of the segment, a set of backward centers C b (U n , DS) is assigned. The set of backward centers for the new utterance U n+1 is C b (U n+1 , DS), which is Un The forward center of f (U n , DS). In the context of visual analysis conversation, the forward and backward centers include data attributes and values, visual characteristics, and analytical actions (e.g., filter, highlight).

[0074]

[0088] Each discourse segment exhibits both global coherence, i.e., the global context of the entire conversation, which typically indicates the topic or subject of the conversation, and local coherence, i.e., the coherence between utterances within that conversation. f (U n , D.S.) and C. b (U n We show that we infer a sequence of utterances in a local context through transition states of continuation, retention, and substitution between utterances (DS), DS, and DS. The framework extends this conversational centering theory for visually analyzing conversations by introducing a set of rules for each of these local coherence components, subject to several implementations.

[0075]

[0089] Utterance U n Given, a system that implements this framework has a forward-looking center C f (U n The system responds by executing a set of analytical functions derived from the set of variables X (which can be attributes or visualization features), operators op, and values ​​v (usually constants). For example, if the user says "measles in the UK", the system creates two functions: F_CAT(Disease, ==, Measles) and F_CAT(Country, ==, UK). When the user enters a new utterance U n+1 When you provide U, the system will first select U without considering any prior context. n+1 From temporary center set C temp (U n+1 , DS). The system then applies a set of rules to create a b (U n+1, D.S.) and C. temp (U n+1 Based on some set operations between C f (U n+1 , DS), and then, according to some implementations, the forward-looking center is used to respond to the user's utterances.

[0076]

[0090] FIG. 3A illustrates two utterances U according to some implementations. n (302) and U n+1 We show this process for (312). The system selects U without considering any prior context. n From temporary center set C temp (U n )(306) is created (304). n Upon receiving the utterance, the system n-1 Based on this, a set of backward conversation centers C b (U n )(301) or U n If is the first utterance, then C b (U n ) to the empty set. The system then applies (308) one or more transition rules (described below with reference to FIG. 3B) to determine the rear-facing center C b (U n )(301) and Temporary Conversation Center C temp (U n )(306) to the set of forward-looking centers C f (U n )(310) is derived. Positive Conversation Center C f (U n )(310) also corresponds to the following utterance U n+1 Backwards Conversation Center C for (312) b (U n+ 1) The process described so far is n+1 The process is repeated for U without considering any prior context. n From temporary center set C temp (U n) (316) (314). Next, the system calculates the rear-facing center C b (U n+1 )(310) and Temporary Conversation Center C temp (U n+1 )(316) to the set of forward-looking centers C f (U n+1 The system applies one or more transition rules (318) to derive a visualization of the one or more data (319). The system may also use a forward-looking conversation center C to update the visualization of the one or more data according to some implementations. f (U n+1 )(319). The system also uses the following utterance U n+2 Backwards Conversation Center C b (U n+2 ) as the set of forward-looking centers C f (U n+1 ) (319). In some implementations, when the user navigates to a different dataset or resets the visualization, the system updates the global coherence of the analyzed conversations and clears all previous states (e.g., forward conversation centers, backward conversation centers, temporary conversation centers, etc.).

[0077]

[0091] 3B is a state machine diagram illustrating states of the conversation center and transitions between states, according to some implementations. State 322 is the backward conversation center C b and Temporary Conversation Center C temp Each of states 324, 326, and 328 encapsulates a forward-looking conversation center C f The conversation center represents the different states of speech. n+1 (not shown). Below is a description of each transition rule, i.e., when each transition occurs and how the end state is calculated.

[0078]

[0092] According to some implementations, the transition continuation 323 transfers the context to the backward center C b From forward facing center C f That is, backward conversation center C bEach conversation center is a positive conversation center C f (324). A given utterance U in discourse segment DS n+1 If we use set notation for , then as a result of this transition, we get, along with other entities: C b (U n+1 , DS)∈C f (U n+1 , D.S.

[0079]

[0093] In some implementations, this transition occurs when variable X is temp (U n+1 ) but C b (U n+1 , DS). In this case, the system performs the following sum operation: C f (U n+1 , DS)=C b (U n+1 , DS) ∪ C temp (U n+1 , D.S.

[0080]

[0094] According to some implementations, the transition hold 325 may be performed to hold the forward looking center C without adding additional entities to the forward looking center. f (326) Rear-facing center C b Keeping the context from (322), i.e. C f (U n+1 , DS)=C b (U n+1 , D.S.

[0081]

[0095] According to some implementations, this transition hold 325 occurs when the variable X is b (U n+1 , DS) but C temp (U n+1 Triggers if the specified device is not in the specified range (DS).

[0082]

[0096] In some implementations, using a shift transition 327, the context shifts from the backward conversation center 322 to the forward conversation center 328. C f (U n+1 , DS) ≠ C b (U n+1 , D.S.

[0083]

[0097] In some implementations, the shift transition 327 is performed when the variable X is C b (U n+1 , D.S.) and C. temp (U n+1 , DS) but the corresponding values ​​are different. In this case, the system b (U n+1 , DS) to C temp (U n+1 , DS). As shown in FIG. 3B, this replacement can be expressed using the formula: C f (U n+1 , DS)=C b (U n+1 , DS)-X Cb +X Ctemp

[0084]

[0098] In some implementations, a shift transition 327 also occurs when a filter constraint is removed, for example removing the measles widget changes the disease variable from measles to all diseases.

[0085]

[0099] FIG. 4A illustrates the use of different transition rules when processing a conversation to be analyzed according to some implementations. Assume that the system implements the framework described above with reference to FIG. 3A and FIG. 3B. When the system receives an utterance 400 (e.g., “measles in the UK”), the system calculates (402) forward conversation centers (404) corresponding to the utterance and initializes backward conversation centers according to some implementations. The conversation centers of the example utterance are “measles” and “UK”. Based on the calculated conversation centers, the system applies (406) filters to update the visualization, for example applying categorical and spatial filters to show measles in the UK in visualization 408. FIG. 4D illustrates a zoomed-in version of visualization 408. As shown in FIG. 4D, the utterance “measles in the UK” leads to the system showing a visualization of “disease outbreaks around the world” with the system focused on (DISEASE==measles (442) and COUNTRY==UK (444)). Returning to FIG. 4A, assuming the user responds to the visualization 408 with a new utterance 410 (e.g., "Show me the orange spike"), the system calculates (412) the forward conversation center by applying the continuation rule (323, described above with reference to FIG. 3B). This is because the temporary conversation center (not shown) for the example utterance includes the visualization characteristics (types of variables), namely "orange spike" corresponding to the shape variable (having a value of "surge") and the color variable (having a value of "orange"), which are not present in the backward conversation center (404). Furthermore, since the utterance 410 does not reference other variables (DISEASE or COUNTRY) in the backward conversation center (404), the system does not apply the hold or shift rules. The forward conversation center (414) for the utterance 410 is thus calculated to be {measles, UK, orange spike}. Based on these forward looking centers, the system updates the corresponding graphs in the data visualization, for example, by highlighting orange spikes in the row for measles in the graph of visualization 418. FIG. 4E shows a larger version of visualization 418.As this example shows, in response to the utterance “Show me the orange spike,” the system, according to some implementations, highlights the spike and annotates the spike with the words “4.4 cases of measles in 2013” ​​(446).

[0086]

[0100] FIG. 4B is another diagram further illustrating the application of different transition rules, for example, to utterances. This example follows a user's response to the visualization 438 of FIG. 4A. Referring to FIG. 4B, when the system receives an utterance 420 (e.g., "mumps over there"), according to some implementations, the system calculates (422) a forward conversation center corresponding to the utterance. The backward conversation center (424) (e.g., {measles, UK, orange surge}) corresponds to the forward conversation center (414) calculated in the previous step (in response to utterance 410) shown in FIG. 4A. The temporary conversation center (not shown) of utterance 420 (e.g., "mumps over there") does not include the conversation center derived for the previous utterance (e.g., "orange surge"). This triggers the system to apply the retention rule (described above with reference to 325 of FIG. 3B). For example, the temporary conversation center includes a variable called DISEASE, but the value of that variable has been changed from measles to mumps. This causes the system to apply the shift rules (described above with reference to 327 in FIG. 3B). As a result, the system computes forward conversation centers (424) (e.g., {mumps, UK}). Based on these forward conversation centers, the system responds by applying the necessary filters (e.g., keeping the spatial filter for UK, updating the categorical filter to mumps) and updating the visualization (e.g., removing the highlighted spike in the measles row) (426), as shown in visualization 428. FIG. 4F shows a zoomed-in version of visualization 428. As shown in FIG. 4F, the utterance "mumps over there" results in the system showing a visualization of "Disease outbreaks around the world" with a focus on (DISEASE==mumps (448), and COUNTRY==UK (450)).

[0087]

[0101] To continue the example, again referring to FIG. 4B, suppose the user asks a question with another utterance (430) (e.g., “Measles epidemic in Malawi, Congo, and Angola”), the system calculates (432) the forward center corresponding to this utterance by applying the shift rule described above. In this illustrated example, the reason for applying the shift rule is that the value of the variable DISEASE is now different, i.e., measles and epidemic. Similarly, the geographic region has changed from the UK to Malawi, Congo, and Angola. Based on applying the shift rule, the system calculates (434) the forward conversation center (in this example, the centers are {measles, epidemic, Malawi, Congo, Angola}). Based on the forward conversation center (434), the system further responds to the utterance by applying (436) appropriate filters (e.g., a categorical filter for measles epidemic, and a new spatial filter for Malawi, Congo, and Angola instead of the UK). Thereby, a visualization 438 is generated (e.g., showing the measles epidemic in Malawi, Congo, and Angola). Figure 4G shows a zoomed-in version of visualization 438. As shown in Figure 4G, the utterance "Measles epidemic in Malawi, Congo, and Angola" results in the system showing a visualization of "Disease outbreaks around the world" with focus (DISEASE == Measles (452), IMPACT == Epidemic (454), COUNTRY == Malawi (456), COUNTRY == Congo (458), COUNTRY == Angola (460)).

[0088]

[0102] 4C illustrates updates to the visualizations of data in response to user utterances described above with reference to FIGURES 4A and 4B, according to some implementations. Visualizations 408, 418, 428, and 438 were described above with reference to enlarged views 4D, 4E, 4F, and 4G, respectively.

[0089]

[0103] FIG. 5 is a diagram showing a general framework for the application of the principles of pragmatics for visual analysis according to some implementations. This framework utilizes and extends a model commonly used in a discourse structure called conversation centering according to some implementations. In this model, an utterance is divided into constitutive discourse segments, and the relationships that can hold between two segments are embedded. A center refers to an entity that helps link that utterance to other utterances in the conversation. Consider the discourse segments DS of utterances U1, …, U m For the discourse segment DS of. Each utterance U n (1 ≦ n < m) is assigned a series of forward centers C f (U n , DS) that indicate the current focus of the conversation. For each utterance other than the first utterance of the segment, a set of backward centers C b (U n , DS) is assigned. The set of backward centers for the new utterance U n+1 is C b (U n+1 , DS), which is equal to the forward center of U n (i.e., C f (U n , DS)). In the context of a conversation for visual analysis, the forward and backward centers include data attributes and values, visual characteristics, as well as analysis actions (e.g., filters, highlights).

[0090]

[0104] Given an utterance U n , a system implementing this framework responds by executing a series of analysis functions derived from the forward center C f (U n , DS). The analysis function F(X, op, v) consists of a variable X (which can be an attribute or a visualization characteristic), an operator op, and a value v (usually a constant). For example, if the user says "measles in the UK", the system creates two functions, namely F_CAT(disease, ==, measles) and F_CAT(country, ==, UK). When the user makes a new utterance U n+1When you provide U, the system will first select U without considering any prior context. n+1 From temporary center set C temp (U n+1 , DS). The system then applies a set of rules to create a b (U n+1 , D.S.) and C. temp (U n+1 Based on some set operations between C f (U n+1 , DS), and then, according to some implementations, the forward-looking center is used to respond to the user's utterances.

[0091]

[0105] FIG. 5 illustrates two utterances U in some implementations. n (500) and U n+1 We show this process for (520). The system calculates U without considering any prior context. n+1 From temporary center set C temp (U n+1 )(528) is calculated (526). Utterance U n+1 Upon receiving the utterance, the system n Based on this, a set of backward conversation centers C b (U n+1 )(504) or U n If is the first utterance, then C b (U n+1 )(504) was initialized to the empty set.

[0092]

[0106] Conversation centering assumes that utterances exhibit connections between them. The way in which these utterances link together to form a conversation is cohesion. Cohesion arises as a result of the combination of both lexical and grammatical structures in the constituent phrases. Identifying the phrase structure is therefore a logical starting point for decomposing the utterance into one or more analytical functions that are applied to the visualization. The structure of the phrase includes both lexical and grammatical structures. In FIG. 5, a system implementing this framework, in step 522, analyzes utterances U n+1Calculate the phrase structure of (520). Typically, a parser is used to calculate the phrase structure. The parser accepts an input sentence (sometimes called a query or natural language command) and applies a set of grammar rules specific to a particular natural language, such as English, to split the input sentence into a set of tokens (language elements). In some implementations, the grammar rules can be modified to fit the environment. In some implementations, a probabilistic grammar is applied to provide a structural description of the input query. The probabilistic grammar helps resolve ambiguities in parsing the sentence. Probability distributions (over the grammar production rules) can be estimated, for example, from a corpus of hand-analyzed sentences. Some implementations infer additional syntactic structure by using a part-of-speech (POS) tagger that assigns a part of speech, such as noun, verb, adjective, etc., to each word (sometimes called a token). Some implementations decompose the parsed output into corresponding categorical and ordered data attributes. As indicated by the dashed line connecting blocks 500 and 510, in some implementations, the system generates a grammar structure for the utterance U. n The phrase structure of (500) is also calculated (510).

[0093]

[0107] Using the phrase structure, the system proceeds to determine (530) the type of pragmatic form (examples of which are described below with reference to Figures 6A-10C) and any other phrase structure related information (e.g., linguistic elements that are present in one utterance but not the other). Then, according to some implementations, based on the pragmatic form and phrase structure information (532), the system generates a temporary conversation center (528) and a backward conversation center (533). b (504) is used to make the utterance U n+1 (520) Positive Conversation Center C f (536) is derived (534). As shown in FIG. 5, n+1 Even before receiving (520), the system may apply (506) an initial set of operations (e.g., applying filters) to the existing visualization to generate a backward conversation center C b(504) based on one or more initial visualizations (508). In some implementations, the forward-looking conversation center C f Based on (536), the system applies (538) a second set of operations (e.g., applying a filter) to update an existing visualization or generate a new visualization (540).

[0094]

[0108] FIG. 6A illustrates the application of pragmatic principles to incomplete utterances (sometimes called ellipses) according to some implementations. Ellipses are syntactically incomplete sentence fragments that leave out one or more linguistic elements. These utterances can be better understood in a previously established context. FIG. 6A illustrates how the incomplete utterance "townhome" is understood within the context of the previous utterance "house under 1M in Ballard." When the system receives the utterance U, which in this example is the utterance "house under 1M in Ballard," n Upon receiving (600), the system calculates the forward center in step 602 using the methodology described above with reference to FIG. 5 according to some implementations. n+1 6C , the utterance "Homes under $1M in Ballard" results in the system showing a visualization of "Past Home Sales - Seattle" centered on (Ballard (644) with LAST_SALE_PRICE less than 1.0M (642)). Referring back to FIG. 6A , in some implementations, the system also calculates the set of conversation centers to be {Houses, Ballard, and 1M} at step 610 using one or more techniques described above with reference to step 510 of FIG. 5 . n The phrase structure (610) of the sentence is calculated.

[0095]

[0109] In some implementations, the system receives the utterance U, which in this example is the utterance “townhome.” n+1 Upon receiving (620), the system n+1 The system then calculates (626) a temporary conversation center for (620). In this example, the system calculates the conversation center (628) to be the set {Townhome}. Additionally, the system may, according to some implementations, calculate a temporary conversation center for the utterance U using the techniques described above with reference to step 522 (FIG. 5). n+1 The phrase structure of (620) is calculated (622).

[0096]

[0110] As previously discussed, ellipsis excludes one or more linguistic elements. According to some implementations, using the phrase structure (612 and 624), the system parses the utterance U n+1 The utterance U corresponding to the missing linguistic element in (620) n Determine the subset of conversational centers in (600). In this example, the linguistic elements, namely a noun phrase indicating location following a prepositional phrase (corresponding to "ballad") and a noun phrase indicating price value following another prepositional phrase (corresponding to "1M", or more precisely, "under 1M"), are included in the utterance U n+1 (620) is missing, but the utterance U n Since the utterance U exists in (600), the system computes the subset to be the set {Ballad, 1M}. n (600) Phrase "house" and utterance U n+1 The phrase "townhome" in (620) corresponds to a similar linguistic element (e.g., both phrases are noun phrases and indicate a type of housing).

[0097]

[0111] In step 634, according to some implementations, the system combines the temporary set of conversation centers, in this example the set {Townhomes}, with a subset of conversation centers (632), which then generates the speech U n+1A set of forward conversation centers (638) for the utterance "townhome" is reached. According to some implementations, the system determines the type of filter to apply to the data set based on the calculated set of forward conversation centers (636) and applies the appropriate filter in step 638 to display the appropriate data visualization (640). In this example, the conversation centers "Ballard" and "1M" are retained from the backward conversation centers (604), so the system retains the numeric filter (corresponding to 1M) and the spatial filter (corresponding to Ballard). Also, because the value of the conversation center (corresponding to the home_type variable) was changed from townhome to house, the system applies a categorical filter to home_type to display townhome (rather than house). FIG. 6D shows a zoomed-in version of visualization 640. As shown in FIG. 6D, the utterance "townhome" results in the system showing a visualization of "Past Home Sales - Seattle", retaining the LAST_SALE_PRICE widget or filter (642) and spatial filter 644 from the previous visualization 608, and replacing HOME_TYPE with townhouse (646).

[0098]

[0112] 6B illustrates updates to the visualization of data in response to the user's utterances described above with reference to FIG. 6A, according to some implementations. Visualizations 608 and 640 were described above with reference to enlarged views 6C and 6D, respectively.

[0099]

[0113] FIG. 7A illustrates the application of pragmatic principles for utterances with referring expressions (sometimes referred to herein as anaphoric references) according to some implementations. Referring expressions help unify texts, create economy, and prevent unnecessary repetition. References are conversational forms, which are not semantically interpreted in themselves, but make a reference to something else for its interpretation. When the interpretation is within the text, this is known as anaphoric reference. In visual analysis interactions, references involve data attributes and analysis functions. FIG. 7A illustrates how an utterance of "previous year" is understood in the context of the previous utterance "prices in 2015". When the system considers an utterance U, which in this example is the utterance "prices in 2015", the utterance n Upon receiving (700), the system calculates the forward center in step 702 using the methodology described above with reference to FIG. 5 according to some implementations (step 502), and n+1 7A , the system initializes backward looking centers (for {price, 2015}). For the example utterance, the system computes a set of conversation centers as {price, 2015}. In some implementations, the system applies a filter (706) to the dataset based on the set of conversation centers (704) and displays a visualization (708) of the data. In this example, the system applies a time filter showing home prices in 2015. FIG. 7C shows a zoomed-in version of visualization 708. As shown in FIG. 7C , the utterance "price in 2015" results in the system showing a visualization of "Historical Home Sales in 2015 - Seattle." Although not shown, for example, due to a previous visualization in response to a previous utterance, the system has checked prices in Seattle. Referring back to FIG. 7A , according to some implementations, the system may filter (706) the dataset based on the set of conversation centers, which in this example is "previous year." n+1 Upon receiving (720), the system generates an utterance U using the techniques described above with reference to step 522 (FIG. 5). n+1 Calculate the phrase structure of (720) (722).

[0100]

[0114] As mentioned above, referring to an expression using anaphoric reference refers to something else in the text. Based on the phrase structure (724), the system may, according to some implementations, generate a grammatical representation of the utterance U n+1 The system identifies (726) an anaphor for (720). In this example, the system identifies the anaphoric word "before" (728). According to some implementations, using the identified anaphoric word, the system then identifies (734) a phrase chunk that contains the reference in order to identify the entity to which the reference refers. In the illustrated example, the system identifies a phrase chunk "year" that corresponds to the anaphoric word "before." According to some implementations, based on the identified anaphoric word and phrase chunk, in step 730, the system searches the back center to find such entities and replaces the anaphoric reference with these entities. Additionally, in some implementations, as in this example, the system detects and applies an appropriate function to the value of the entity. In the illustrated example, the system also detects that the user is referring to a "before" year, and therefore the value of 2015 is decremented by one before arriving at the correct value of the year variable. According to some implementations, the system calculates the "before" date using a time function (e.g., DATECALC). The system arrives at a set of forward conversation centers (736), which in this example is the set {price, 2014}. Based on this set, the system takes the necessary steps to update the visualization at step 738, according to some implementations. In this example, the system keeps the year reference and updates the time filter to 2014 to display the visualization at 740. FIG. 7D shows a zoomed-in version of visualization 740. As shown in FIG. 7D, the utterance "previous year" results in the system showing a visualization of "Historical Home Sales - Seattle" for the year 2014, the year before 2015 (from the previous visualization).

[0101]

[0115] 7B-7D show graphical user interfaces (related to FIG. 7A ) for interactive data analysis using natural language processing in a data visualization application according to some implementations. FIG. 7B shows updates to the data visualization in response to a user utterance described above with reference to FIG. 7A , according to some implementations. Visualizations 708 and 740 were described above with reference to enlarged views 7C and 7D, respectively.

[0102]

[0116] Although not shown in FIG. 7A, in some implementations, the system repeats the steps of recognizing multiple anaphoric references in a single expression. Additionally, in some implementations, the system identifies many types of anaphoric references in a given utterance, such as "it," "them," "them," "ones," "previous," and "next." As another illustrative example, consider the utterance "Show me Fremont, Queen Anne, and Ballard," followed by "Condos in those neighborhoods." In this example, they refer to several values ​​of the attribute in the attribute neighborhood (i.e., Fremont, Queen Anne, and Ballard), as indicated by the word neighborhood.

[0103]

[0117] In some implementations, the reference indicates a value of an attribute of the data. In some implementations, the reference indicates an action that needs to be performed by the system. For example, consider the utterance "eliminate Ballard" followed by "to Fremont," a word that is not immediately followed by any noun, but is immediately preceded by the verb word "do." In such a case, the system determines the action or actions stated in the previous utterance, which in this example is the action "eliminate."

[0104]

[0118] In some implementations, the system supports references that are outside of the text and within the context of the visualization. In some such implementations, the forward-looking center C f , Rear facing center C bIn contrast to text in a visualization, indirect referencing refers to a context within the visualization. In some implementations, this form of indirect referencing involves a strict reference, which points to some object in the environment, usually by pointing to it. In some such implementations, the system supports strict referencing by allowing multimodal interaction (mouse+voice / text). FIG. 7E shows an example visualization in response to a strict reference. In some implementations, this form of indirect referencing involves visualization property references, which use properties in the visualization, such as mark properties, text in labels, axes and titles. FIG. 7F shows an example visualization in response to a reference to a visualization property.

[0105]

[0119] FIG. 8A illustrates the application of pragmatic principles to utterances with conjunctions according to some implementations. Conjunctions in an utterance convey a set of relationships between phrase fragments called conjunctions. In conversations, users tend to repeatedly create compound queries by adding multiple conjunctions. Explicit use of conjunctions or linking words such as "and", "or", "also" between phrases is often avoided. FIG. 8A illustrates how the utterance "Houses in Ballard that were under 600k last summer" is understood in the context of the previous utterance "Houses in Ballard". When the system interprets the utterance U, which in this example is the utterance "Houses in Ballard", n Upon receiving (800), the system calculates the forward center in step 802 using the methodology described above with reference to FIG. 5 according to some implementations (step 502), and n+1 For the example utterance, the system initializes backward-looking centers (for {home, Ballard}). For the example utterance, the system computes the set of conversation centers to be {house, Ballard}. In some implementations, the system applies a filter (806) to the dataset based on the set of conversation centers (804) and displays a visualization (808) of the data. In this example, the system applies categorical and spatial filters that show houses in Ballard.

[0106]

[0120] According to some implementations, the utterance U, which in this example is "a house in Ballard that was under 600k last summer," n+1 When (810) is received by the system, the system converts the utterance U into a quantified form using the techniques described above with reference to step 522 (FIG. 5). n+1 The system computes (812) a phrase structure for (810). As previously mentioned, a compound query consists of multiple (possibly implicit) conjunctions between its constituent phrases. Based on the phrase structure (814), the system computes (815) a phrase structure for the utterance U, according to some implementations. n+1 The system identifies (816) conjunctions of (810). In this example, the system identifies a number of queries (818), namely, "house," "in Ballard," "under 600k," and "last summer." Based on these queries (818) and the set of backward conversation centers (804), the system computes (820) a set of forward conversation centers (822) according to some implementations. For example, the system may compute (820) a set of forward conversation centers for the utterance U n Select a matching context from (800) and n+1 Add new conversation centers derived from the conjunction from (810). The set of forward conversation centers (822) in this example is the set {Houses, Ballard, <600k, Last Summer}. Based on this set, the system takes the necessary steps to refine the current visualization in step 824, according to some implementations. In this example, the system applies a numeric filter on house prices and a time filter showing past summers to show the visualization in 826. FIG. 8B shows a zoomed-in version of visualization 826. As shown in FIG. 8B, the utterance "Houses in Ballard that were under 600k last summer" results in the system showing a visualization of "Past Home Sales - Seattle in Ballard (828)" with last summer, LAST_SALE_PRICE (830), and under 600k (832). In this example, the system further breaks down the period (the "Last Summer" period) based on the previous visualization as the period 2015-6 to 2015-8-31 (834), according to some implementations.

[0107]

[0121] FIG. 8C illustrates how the system iteratively connects analytic functions of adjacent nodes in the parse tree by linearization, according to some implementations. Finding implicit data coherence between conjunctions is sometimes a challenging task. In the example illustrated in FIG. 8B, all conjunctions refer to the same entity, “Houses in Ballard.” However, conjunctions may map to different entities. An example utterance is “Houses in Ballard in condos under 600k in South Lake Union.” The system determines whether individual conjunctions are resolved to the same entity or different entities, according to some implementations. In some such implementations, the system uses rule-based techniques to take a potentially long utterance with potentially implicit conjunctions and convert the utterance into a collection of analytic functions chained together by logical operators. The system then executes these analytic functions in response to the user's utterance, according to some implementations.

[0108]

[0122] In some implementations, the system breaks down multiple conjunctions in a compound utterance to invoke one or more corresponding analysis functions through a process of linearization. In some such implementations, an analysis function F(X, op, v) consists of variables X (e.g., attributes), operators op, and values ​​v. Each attribute is categorical or ordered. Ordered data types are further classified into ordinal and quantitative. The linearization process combines analysis functions using logical operators (i.e., ∧, ∨), taking into account the types of attributes and operators, as described below.

[0109]

[0123] Applying the ∨ operator: If two or more adjacent conjunctions share an attribute and the data type of the attribute is categorical, according to some implementations, the system connects these conjunctions with ∨. Similarly, if the shared attribute is ordered and the function operator is ==, the system applies ∨, according to some implementations. In such cases, applying ∧ would not match any items in the data table, so ∨ is logically more suitable as a selection. For example, if the utterance is "Show me condos and townhomes," according to some implementations, the system generates the following combination analysis function: (F_CAT(House Type, ==, Condominium) ∨ F_CAT(House Type, ==, Townhome)). In this example, both "condominium" and "townhome" belong to the same categorical attribute, i.e., house type. At the same time, both "condominium" and "townhome" cannot be a specific house (item), so applying the ∨ operator is logically more suitable than applying the ∧ operator. Similarly, if the user says "a house with 2 and 3 bedrooms," the system generates (F_ORDINAL(beds, == 2) ∨ F_ORDINAL(beds, == 3)) according to some implementations. If the attribute types are ordered and the condition X< v1 and X> v2 The ∨ operator is also suitable if the expression v1 < v2 For example, if the utterance is "before 2013 and after 2014", then according to some implementations the ∨ operator will be used between the two conjunctions. Again, applying the ∧ operator will not match any items in the data table.

[0110]

[0124] Applying the ∧ operator: If the attribute type is ordered and the condition X> v1 and X< v2 The ∧ operator is suitable when v1 < v2For example, "a house over 400k and under 700k" becomes (F NUMERIC(price, >, 4000000) ∧ F NUMERIC(price, <, 700000)). "2-4 beds" becomes (F ORDINAL(beds, >=, 2) ∧ F NUMERIC(beds, <=, 4)). Note that applying the ∨ operator will match all items in the data table. In some implementations, the ∧ operator is also applied when there are no common attributes between two conjunctions. For example, the utterance "a house with 2 beds and priced under 600k" becomes (F ORDINAL(beds, ==, 2) ∧ F NUMERIC(price, <=, 600000)).

[0111]

[0125] To generate a representation of the analysis functions for the entire utterance, the system traverses the corresponding parse tree for the utterance generated by a parser (e.g., the parser described above with reference to Figure 5) in post-order and applies the above two rules iteratively on phrases as shown in Figure 8C. In the example shown in Figure 8C, the system takes the utterance "townhomes under 1M, condos under 600K" as input and iteratively applies the above rules to generate a set of analysis functions.

[0112]

[0126] FIG. 9A illustrates the application of pragmatic principles to process lexical cohesion according to some implementations. The three previous types of pragmatics - ellipsis, reference, and conjunction - provide grammatical cohesion in a conversation. In addition to these grammatical building blocks, users often find ways to express concepts through related word meanings, i.e. meanings in conversation, nouns called lexical cohesion. These word meanings can be as simple as spellings, stems, plural variations (e.g., "singular interest" vs. "plural interest"), synonyms (e.g., "country" vs. "nation"), related or co-occurring nouns (e.g., "violence" vs. "crime"). Often, word meanings are related to each other in a semantic context.

[0113]

[0127] Figure 9A shows how the utterance "cheapest" is understood in the context of the previous utterance "the most expensive house in Queen Anne." When the system receives the utterance U n Upon receiving (900), the system calculates the forward center in step 902 using the methodology described above with reference to FIG. 5 according to some implementations (step 502), and n+1 9B shows a zoomed-in version of visualization 908. As shown in FIG. 9B, the utterance "Most Expensive Homes in Queen Anne" results in the system showing a visualization of "Historical Home Sales - Seattle" that includes the top 10% of LAST_SALE_PRICE (928) in Queen Anne (930) according to some implementations.

[0114]

[0128] According to some implementations, the utterance U, which in this example is "the cheapest one", n+1 When (910) is received by the system, the system converts the utterance U into a quantified form using the techniques described above with reference to step 522 (FIG. 5). n+1 The system calculates (912) a phrase structure for the utterance (914). As mentioned above, a user's utterance sometimes has word meanings that are better understood in the context of previous utterances. Based on the phrase structure (914), the system calculates a phrase structure for the utterance U according to some implementations. n+1A lexical cohesion candidate for (910) is identified (916). In this example, the system identifies a candidate called "cheapest" (918) for cohesion analysis. Based on the identified cohesion candidate or candidates (918) and the set of backward conversation centers (904), the system computes (920) a set of forward conversation centers (922) according to some implementations. In the example shown, the system computes (920) a set of forward conversation centers for the utterance U according to some implementations. n Continuing with the remainder of the context from (904), we compute semantically related data attributes (e.g., sale_price) that correspond to the lexical cohesion candidates (e.g., “most expensive” or “cheapest”), substituting the relevant numerical attributes.

[0115]

[0129] In some implementations, the system uses a Word2vec™ model that includes trained vector representations of large text corpora to identify the meaning of words in the attributes by computing word vectors using a recurrent neural network. i and data attribute d j Semantic relationship between S rel is the maximum score calculated as follows:

number

[0116]

[0130] The Word2vec™ model is used here only as an example. Many other neural network models can be used to identify word meanings, such as Stanford University's GloVe™. Some libraries, such as GenSim™ and Deeplearning4j™, bring a selection of different word embedding models in a single package.

[0117]

[0131] In some implementations, the system not only calculates the semantic associations between the terms and the data attributes, but also calculates the type of analytical function associated with each term. For example, the system performs an additional step for queries such as "show me the cheapest houses near Ballard" or "where are the mansions in South Lake Union". The system considers the corresponding dictionary definitions as additional features for these word vectors, and according to some implementations, uses a POS tagger to check whether the definitions contain quantitative adjectives such as "less than", "over", "low", "high". The system then maps appropriate analytical functions to these adjectives according to some implementations. Figure 9B shows an example where the term "most expensive" is mapped to the top N (sale price). Figure 9C shows another example where the term "cheapest" is mapped to the bottom N (sale price). Figure 9D similarly shows a visualization (940) where the term "most deadly" is mapped to the top N values ​​of the attribute "number of deaths" (942).

[0118]

[0132] Referring back to FIG. 9A, for the running example, the system computes a set of forward conversation centers (922), such that {cheapest, homes, Queen Anne}, according to some implementations. Based on this set, the system performs the steps necessary to refine the current visualization in step 924 to generate an updated visualization (926), according to some implementations. In this example, the system refines the current visualization by mapping "cheapest" to bottom-N(sale_price) and applying a numeric filter to the home price. FIG. 9C shows a zoomed-in version of visualization 926. As shown in FIG. 9C, the utterance "cheapest" results in the system showing a visualization of "past home sales-seattle" (926) for the bottom 10% of LAST_SALE_PRICE (932) for Queen Anne (930), according to some implementations.

[0119]

[0133] FIG. 10A illustrates the application of pragmatic principles to repair utterances, according to some implementations. During the course of a conversation, it is common for users to correct or clarify previous utterances. In some implementations, the system supports the use of follow-up repair utterances to correct or "repair" potentially ambiguous utterances or to change the default behavior of how results are presented to the user. For example, to update the system's default behavior, such as highlighting for selection, the user can use utterances such as "No, filter instead." As another example, to update attributes of data, the user can use utterances such as "Exclude condos" or "Change condos to townhomes," as shown in FIG. 7.

[0120]

[0134] Figure 10A shows how the utterance "except condos" is understood in the context of the previous utterance "Houses on Green Lake." When the system receives an utterance U, which in this example is the utterance "Houses on Green Lake," nUpon receiving (1000), the system calculates the forward center in step 1002 using the methodology described above with reference to FIG. 5 according to some implementations (step 502), and n+1 10B shows a zoomed-in version of the visualization 1008. For the example utterance, the system computes the set of conversation centers to be {Houses, Green Lake}. The system generates or updates (1006) a data visualization (1008) based on the computed set of conversation centers (1004) according to some implementations. In some implementations, the system applies a filter (1004) to the data set based on the set of conversation centers and displays (1008) the data visualization. In this example, the system applies a categorical filter and a spatial filter that show houses in Green Lake. FIG. 10B shows a zoomed-in version of the visualization 1008. As shown in FIG. 10B, the utterance "Houses in Green Lake" results in the system showing a visualization of "Historical Home Sales - Seattle" in Green Lake (1028).

[0121]

[0135] According to some implementations, the utterance U, which in this example is "except for condominiums," n+1 When (1010) is received by the system, the system converts the utterance U into a syllabary using the techniques described above with reference to step 522 (FIG. 5). n+1 The system calculates (1012) a phrase structure for (1010). As described above, a repair utterance modifies or clarifies a previous utterance. Based on the phrase structure (1014), the system may, according to some implementations, compile an utterance U n+1The system identifies the previous utterance (1010) as a repair utterance (1016) and then identifies the associated repair term (1018) in the utterance. In this example, the system identifies the repair term "except" in the utterance "except condo". Based on the one or more repair utterances and the set of repair terms (1018), the system computes (1020) a set of forward conversation centers (1022) according to some implementations. For example, the system identifies conversation centers and / or data attributes of the previous utterance that are associated with the identified one or more repair terms according to some implementations. In some implementations, as shown in FIG. 10A, the system repairs or disambiguates the conversation centers based on the repair conditions (1018). The set of forward conversation centers (1022) in this example is the set {Houses without condos, Green Lake}. Based on this set, the system takes the necessary steps (1024) to update the results from the previous visualization according to some implementations. In this example, the system excludes Condos to show the visualization at 1026. Figure 10C shows a zoomed in version of visualization 1026. As shown in Figure 10C, the utterance "Exclude Condos" results in the system showing a visualization of "Historical Home Sales - Seattle" in Green Lake (1028), excluding HOME_TYPE==Condo / Coop (1030).

[0122]

[0136] FIG. 11A illustrates the application of pragmatic principles to process responses and feedback according to some implementations. To provide context, a general framework for applying pragmatic principles for visual analytics is first described herein. This framework, according to some implementations, utilizes and extends a commonly used model for discourse structure called conversational centering, in which an utterance is divided into constituent discourse segments and embeds relationships that may hold between two segments. A center denotes an entity that helps link the utterance to other utterances in the discourse. U1, ..., U2, ... ... m Consider a discourse segment DS. Each utterance U of DS nFor (1 ≦ n < m), a series of forward centers C indicating the current focus of the conversation f (U n , DS) is assigned. For each utterance other than the first utterance of the segment, a set of backward centers C b (U n , DS) is assigned. The set of backward centers for a new utterance U n+1 is C b (U n+1 , DS), which is equal to the forward center of U n (i.e., C f (U n , DS)). In the context of visually analyzing a conversation, the forward and backward centers include data attributes and values, visual characteristics, and analysis actions (e.g., filters, highlights).

[0123]

[0137] Given an utterance U n , a system implementing this framework responds by executing a series of analysis functions derived from the forward center C f (U n , DS). The analysis function F(X, op, v) consists of a variable X (which may be an attribute or a visualization characteristic), an operator op, and a value v (usually a constant). For example, if the user says "measles in the UK", the system creates two functions, namely F_CAT(disease, ==, measles) and F_CAT(country, ==, UK). When the user provides a new utterance U n+1 , the system first creates a set of temporary centers C n+1 (U temp , DS) from U n+1 without considering any previous context. Next, the system applies a series of rules to create the forward center of C b (U n+1 , DS) based on some set operations between C temp (U n+1 , DS) and C f (U n+1 , DS). Next, according to some implementations, the forward center is used to respond to the user's utterance.

[0124]

[0138] To support the conversation, the visualizations displayed by the system provide cohesive and appropriate responses to various utterances. In some cases, the system responds by modifying the visual encoding of existing visualizations, while in other cases, the system creates new diagrams to better support visual analysis conversations. In addition to appropriate visualization responses, the system helps the user understand how the system interpreted the utterance by generating appropriate feedback and allows the user to revise the interpretation through some interface controls, if necessary. In traditional dashboards, the user interacts by selecting visualized items or attributes, which are highlighted to provide immediate visual feedback. At the same time, other diagrams are updated by highlighting or excluding items. However, in natural language interfaces, instead of making explicit mouse / keyboard selections, the user is referring to various attributes or values, making the challenge of determining how each view in the dashboard should respond to the utterance non-trivial. Another complication arises when the system must support multiple visualizations.

[0125]

[0139] 11A illustrates a methodology for generating a response according to some implementations. To determine how a view (V) in a dashboard should respond to an utterance, the system according to some implementations proceeds as follows: The system first generates a response to a previous utterance U n+1 Based on the conversation center of , and a set of temporary conversation centers computed using only the current context / utterance, n+1 A set of forward conversation centers corresponding to (1100)C f (U n+1) in the updated dashboard. The system creates (1102) a list of all data attributes (1104). The system then determines (1106) whether any of the existing visualizations encodes each attribute in the list of data attributes (1104), for example by invoking a visualization manager. The system then determines (1118) if the visualization (sometimes referred to herein as a view) directly encodes each attribute as its dimension (e.g., as a visualization feature) (e.g., without using any aggregation function such as count or average). If the attribute is found to be encoded by an existing visualization V (i.e., the condition checked in 1118 is true / yes), the system highlights (1122) marks associated with the criteria (1124) corresponding to the respective attribute in the updated dashboard. On the other hand, if the system determines that the selected visualization (denoted as visualization V) does not directly encode the respective attribute, the system filters (1120) results that do not match the criteria (1124) in the updated dashboard, according to some implementations. This is typically the case when a secondary chart applies further data transformations to the result set (e.g., using line or bar graphs). In some such implementations, the system further highlights one or more results that match the criteria (1124) corresponding to the respective attribute in the updated dashboard.

[0126]

[0140] Figure 11B is an example visualization (1170) further illustrating the methodology shown in Figure 11A, according to some implementations. The system highlights items that match the criterion "measles in the UK" in a map graph (1172). The system also highlights a series (1176) in a line graph (1174) and a bar (1180) in a bar graph (1178) that represents "measles". However, the bar graph of impact (1182) does not include a list of attributes (e.g., the list {X1, X2, ... X m} (1104)), it is not possible to highlight any marks. Thus, the system filters out results that do not match the criterion "measles in the UK" and updates the chart accordingly. Note that the user can change the default behavior by explicitly expressing a choice between filtering or highlighting (e.g., "filter out", "remove", "filter only").

[0127]

[0141] During the flow of visual analysis, there may be situations where existing visualizations cannot meet the evolving information needs of the user. This scenario occurs, for example, when a particular data attribute cannot be effectively encoded in an existing visualization (e.g., time values ​​in a map), justifying the need to create a new visualization in response. According to some implementations, taking inspiration from work on connecting visualizations with linguistic details, the system supports the creation of different kinds of visualizations (e.g., bar charts, line charts, map charts, and scatter plots).

[0128]

[0142] 11C illustrates how the dashboard is built incrementally based on input utterances. The system generates visualization 1140 in response to the utterance "average price by neighborhood." When the user next provides the utterance "average price over time," the system responds by generating a line graph (1152) in visualization (1150) that shows the average price over time. Now, if the user next provides the utterance "by house type," the system enhances the line graph (1152) with lines (1162) that correspond to different house types in visualization (1160).

[0129]

[0143] Referring back to FIG. 11A, the underlying algorithm for creating or modifying an existing visualization works as follows: First, the system determines whether the creation of a new visualization or modification of an existing visualization is required (1106). The system determines whether the creation of a new visualization or modification of an existing visualization is required (1107). f (U n+1) (1100) and searches for any current visualizations that encode these data characteristics. If there is no match with the existing visualization details, as shown as an arrow "no" next to decision block 1106, the system generates a corresponding new detail consisting of the attributes and aggregation type. In FIG. 11A, this is illustrated by step 1108, which uses an algorithm to determine the type of chart (e.g., bar chart, map chart, or scatter plot) according to some implementations. In some such implementations, the system employs an automatic presentation algorithm to determine the type of chart generated based on this detail. In some such implementations, the system uses a simplified version of the automatic presentation algorithm described in Show me: Automatic presentation for visual analysis, by J. Mackinlay, P. Hanrahan, and C. Stole, which is incorporated herein by reference. Once the chart type (1110) is determined, the system generates the chart type (1112) and obtains the generated chart (1114). The system then deploys the new chart (1114) according to some implementations. In some such implementations, the system uses a two-dimensional grid-based layout algorithm that automatically coordinates the display of the new chart (1114) with other visualization views. The updated dashboard (1124) responds to subsequent utterances through actions such as highlighting or filtering.

[0130]

[0144] Additionally, although not shown in FIG. 11A, note that according to some implementations, the system repeats at least steps 1106, 1118, 1120, 1122, 1108, 1112, and 1116 for each data attribute in the list of data attributes (1104).

[0131]

[0145] FIG. 12A illustrates a collection of widgets generated to handle ambiguity in a user's query, according to some implementations. A challenge for natural language understanding systems that support conversational dialogs is to determine the intent of an utterance. In some implementations, the system automatically resolves various forms of syntactic, lexical, and semantic ambiguity. These resolutions are represented in the form of widgets and feedback to help the user understand the system's intent and the origin of how the utterance was interpreted. By manipulating these widgets and viewing feedback of which results are displayed in the visualization, the user can, for example, instantiate a follow-up repair utterance to override or clarify the system decision that was made.

[0132]

[0146] In some implementations, the system identifies one or more widgets from an analytic function derived from the utterance. In some such implementations, the system organizes and presents the widgets in an intuitive manner so that the user can understand how the system interprets their utterance and then use these widgets to modify the interpretation. To this end, the system takes the original utterance and arranges the widgets in the same order as the terms in the corresponding query. In some such implementations, the system achieves this by using a library such as Sparklificator™ that facilitates the placement of small, word-scale visualizations within text in a compact manner. Additionally, some implementations provide the user with a collection of interfaces that include the ability to manipulate and / or delete widgets, modify queries, and break down ambiguous queries.

[0133]

[0147] FIG. 12A shows how the system presents widgets for the utterance "Condos near Ballard under 1.2M" according to some implementations. In this example, the first term "condo" is resolved into a widget that represents the criteria "HOME_TYPE equals condo / co-op" (1202). A second widget then conveys the ambiguous distance represented by "near Ballard" (1204). Because "under 1.2M" does not explicitly mention any attribute, the system determines whether the value 1200000 is within the minimum and maximum values ​​of any numeric attribute of the data. If such an attribute exists (in this case LAST_SALE_PRICE), the system conveys it to the user (via widget 1206) and then allows the user to change the attribute using a drop-down menu (1208).

[0134]

[0148] In addition to handling ambiguity, in some implementations, the system also provides feedback and meaningful hints to correct the text when the system fails to fully understand the query. For example, if the system cannot successfully parse a given utterance, the system first attempts to automatically correct misspelled words by comparing tokens with attributes, cell values, and relevant keywords in the current dataset using fuzzy string matching. When a user creates a partially recognized query, the system sorts out unrecognized terms from the corresponding parse tree and then displays results based on the understood tokens. FIG. 12B illustrates different example situations and the corresponding feedback generated by the system, according to some implementations.

[0135]

[0149] 13A-13J provide a flow diagram illustrating a method 1300 of using natural language for visual analysis of a data set applying principles of pragmatics (1302). The steps of the method 1300 may be performed by a computer (e.g., computing device 200). In some implementations, the computer includes a display, one or more processors, and a memory (1304). FIGS. 13A-13J correspond to instructions stored in a computer memory or computer-readable storage medium (e.g., memory 206 of computing device 200). The memory stores one or more programs configured for execution by one or more processors (e.g., processor 202) (1306). For example, the operations of the method 1300 are performed, at least in part, by the data visualization generation module 234 and / or the language processing module 238.

[0136]

[0150] In some implementations, the computer displays (1308) a data visualization based on the dataset retrieved from the database using a first set of one or more queries. For example, with reference to FIG. 1, a user can associate one or more data fields from the schema information area 110 with one or more shelves (e.g., column shelf 120 and row shelf 122 in FIG. 1) of the data visualization area 112. In response to receiving a user association, in some implementations, the computer retrieves data for the data fields from the dataset using a set of one or more queries, and then displays a data visualization (e.g., data visualization 408) in the data visualization area 112 corresponding to the received user input. Displaying a data visualization is discussed in more detail above with reference to FIG. 1.

[0137]

[0151] The computer receives (1310) a first user input specifying a first natural language command associated with the displayed data visualization. In some implementations, the user input is received as text input from a user in a data entry area of ​​the display proximate to the displayed data visualization (e.g., via the keyboard 216 or via the touch-sensitive display 214). In some implementations, the user input is received as a voice command using a microphone (e.g., voice input device 220) coupled to the computer. For example, with reference to FIG. 4A, the displayed data visualization 408 relates to measles in the UK. Receiving input (e.g., commands / queries) from a user is discussed in further detail above with reference to FIG. 1.

[0138]

[0152] Based on the displayed data visualization, the computer extracts (1312) a first set of one or more independent analytical phrases from the first natural language command. For example, referring to FIG. 4A, the first natural language command received by the computer reads "measles in the UK". The data visualization displayed prior to receiving the first natural language command is about disease outbreaks around the world. In some implementations, the computer extracts "measles" and "in the UK" from the first natural language command because these analytical phrases are relevant to the displayed data visualization. When phrases directly reference data fields in the displayed data visualization, the extraction (1312) is straightforward. Collect all phrases that directly reference data fields. In some implementations, the computer extracts or removes stop words, filler words, or any predefined set of words from the input query and extracts (1312) all other phrases from the first natural language command as they may be relevant to the displayed data visualization. Some implementations use this approach when phrases in the natural language command somewhat indirectly reference data fields in the displayed visualization.

[0139]

[0153] According to some implementations, the language processing module 238 calculates (1314) a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analysis phrases. The framework based on the conversational dialogue model is described above with reference to FIG. 3A, FIG. 5, and FIG. 11. A center refers to an entity that helps link the utterance (sometimes referred to herein as a natural language command) to other utterances in a discourse (a sequence of utterances). The conversation center includes data attributes and values, visual characteristics, and analysis actions. Computing the conversation center based on the analysis phrase includes mapping the analysis phrase to one or more conversation centers after necessary transformations and analyses. For the example utterance "measles in the UK", the language processing module 238 processes the phrase "measles" and analyzes the phrase to infer that it refers to the data attribute DISEASE as shown in FIG. 4D above.

[0140]

[0154] Subsequently, according to some implementations, the language processing module 238 calculates (1316) a first set of analysis functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function phrases. As described above with reference to FIG. 3A, FIG. 5, and FIG. 11, according to some implementations, each analysis function consists of a variable, an operator, and a value. In some implementations, for the example utterance "measles in the UK", the language processing module 238 creates two functions, namely F_CAT(disease,==,measles) and F_CAT(country,==,UK). In some implementations, as another example, for the utterance "condominium under 600K", the language processing module 238 creates two functions, F_CAT(house type,==condominium) and F_NUMERIC(price,<,600000). In both of these examples, the language processing module 238 searches for one or more data attributes associated with the displayed data visualization that correspond to the first set of one or more conversation centers to identify a first set of data attributes, according to some implementations. The language processing module 238 also identifies a first set of operators (e.g., operator ==, operator <) and a first set of values ​​that correspond to the first set of data attributes by consulting the first set of one or more conversation centers, according to some implementations. Using the first set of variables (attributes), the first set of corresponding operators, and the first set of values, the language processing module 238 constructs a first set of one or more analysis functions, thereby creating a first set of one or more function clauses.

[0141]

[0155] In some implementations, the computer updates (1318) the data visualization based on the first set of one or more function clauses calculated in step 1316. As shown in FIG. 13I, in some implementations, the computer re-queries (1372) the database using a second set of one or more queries based on the first set of one or more function clauses, thereby retrieving a second data set. In some cases, the re-querying of the database is performed locally on the computing device using data cached or stored on the computing device. For example, if one or more filters are specified in the natural language command, the re-query is typically performed locally. In some such implementations, the computer updates (1374) the data visualization based on the second data set. In some implementations, the computer further creates and displays (1376) a new data visualization using the second data set (e.g., without updating the existing data visualization or visualizations).

[0142]

[0156] Referring back now to FIG. 13B, the computer receives (1320) a second user input to specify a second natural language command associated with the displayed data visualization. In some implementations, the user input is received as text input from the user in a data entry area of ​​the display proximate to the displayed data visualization (e.g., via the keyboard 216 or via the touch-sensitive display 214). In some implementations, the user input is received as a voice command using a microphone (e.g., voice input device 220) coupled to the computer. For example, with reference to FIG. 4A, the displayed data visualization 408 is related to measles in the UK when the computer receives a second user input of "show me the orange spikes". Receiving input (e.g., commands / queries) from a user is discussed in more detail above with reference to FIG. 1.

[0143]

[0157] Based on the displayed data visualization, the computer extracts (1322) a second set of one or more independent analytical phrases from the second natural language command. For example, referring to FIG. 4A, the second natural language command (410) received by the computer reads, "Show me the orange surge." In some implementations, in this example, the computer extracts "orange surge" from the second natural language command, because these analytical phrases are related to the displayed data visualization (related to measles in the UK and have orange surge, visualization properties). If the phrases directly reference data fields in the displayed data visualization, the extraction (1322) is straightforward. Collect all phrases that directly reference data fields. In some implementations, the computer extracts or removes stop words, filler words, or any predefined set of words from the input query and extracts (1322) all other phrases from the second natural language command as they may be related to the displayed data visualization. Some implementations use this approach when phrases in a natural language command refer somewhat indirectly to data fields in the displayed visualization.

[0144]

[0158] According to some implementations, the language processing module calculates (1324) a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analysis phrases.

[0145]

[0159] The language processing module derives a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules according to some implementations. In some such implementations (1332), each conversation center of the first set of one or more conversation centers, the temporary set of one or more conversation centers, and the second set of one or more conversation centers includes a value of a variable (such as a data attribute or visualization characteristic). In some such implementations, the language processing module uses the transition rules by performing a series of operations (as shown in FIG. 13C), including determining whether the first variable is included in the first set of one or more conversation centers (1334), determining whether the first variable is included in the temporary set of one or more conversation centers (1336), determining a transition rule for each of the one or more transition rules to apply based on whether the first variable is included in the first set of one or more conversation centers and / or the temporary set of one or more conversation centers (1338), and applying each transition rule (1339).

[0146]

[0160] 13D, the one or more transition rules used by the language processing module 238 include a CONTINUE rule for including each conversation center of the first set of one or more conversation centers in the second set of one or more conversation centers and adding one or more conversation centers from the temporary set of one or more conversation centers to the second set of one or more conversation centers (1340). In some such implementations, applying (1342) each transition rule includes applying (1344) a CONTINUE rule to include the first variable in the second set of one or more conversation centers according to a determination that (i) the first variable is included in the temporary set of one or more conversation centers and (ii) the first variable is not included in the first set of one or more conversation centers.

[0147]

[0161] 13E, the one or more transition rules used by the language processing module 238 include a RETAIN rule for retaining each conversation center of the first set of one or more conversation centers in the second set of one or more conversation centers without adding a conversation center from the temporary set of one or more conversation centers to the second set of one or more conversation centers (1346). In some such implementations, applying the respective transition rules (1348) includes applying the RETAIN rule to include each conversation center of the first set of one or more conversation centers in the second set of one or more conversation centers in accordance with a determination that (i) the first variable is included in the first set of one or more conversation centers and (ii) the first variable is not included in the temporary set of one or more conversation centers (1350).

[0148]

[0162] In some implementations, as shown in FIG. 13F , the one or more transition rules used by the language processing module 238 include a SHIFT rule for including each conversation center of the first set of one or more conversation centers in a second set of one or more conversation centers and exchanging the one or more conversation centers of the second set of one or more conversation centers with a conversation center of the temporary set of one or more conversation centers (1352). In some such implementations, applying each transition rule (1354) includes the language processing module 228 performing a set of operations to determine whether a value of the first variable of the first set of one or more conversation centers differs from a value of the first variable of the temporary set of one or more conversation centers (1358) in accordance with a determination that (i) the first variable is included in the temporary set of one or more conversation centers and (ii) the first variable is not included in the first set of one or more conversation centers, and applying a SHIFT rule (1360) to include each conversation center of the first set of one or more conversation centers in a second set of one or more conversation centers and exchange the first variable of the second set of one or more conversation centers with the value of the first variable of the temporary set of one or more conversation centers in accordance with a determination that the values ​​of the first variable are different. In some such implementations, applying each transition rule (1354) may include determining whether a widget corresponding to the first variable has been deleted by the user (1362), as shown in FIG. 13G, and, in accordance with a determination that the widget has been deleted, applying a SHIFT rule (1364) to include each conversation center in the first set of one or more conversation centers in a second set of one or more conversation centers and to replace the value of the first variable in the second set of one or more conversation centers with a new value (e.g., a maximum value, or a superset value) that includes the value of the first variable in the first set of one or more conversation centers.

[0149]

[0163] 13B, the language processing module 238 calculates (1328) a second set of one or more analysis functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function phrases. The language processing module 238 performs this step in a manner similar to step 1316 described above, using the second set of one or more conversation centers calculated in step 1326.

[0150]

[0164] The computer updates the data visualization based on the second set of one or more function clauses (1330) according to some implementations. In some such implementations, as shown in FIG. 13J, the computer re-queries the database using the second set of one or more queries based on the first set of one or more function clauses (1378), thereby retrieving a third dataset (1380), which updates the data visualization based on the third dataset. Further, in some such implementations, the computer creates and displays a new data visualization using the third dataset (e.g., without updating the existing data visualization or visualizations) (1382).

[0151]

[0165] In some implementations, as shown in FIG. 13H , the computer further determines whether the user has selected a dataset different from the first dataset (1366), determines whether the user has reset the data visualization (1368), and resets each of the first set of one or more conversation centers, the temporary set of one or more conversation centers, and the second set of one or more conversation centers to an empty set that does not include a conversation center (1370) in accordance with a determination that (i) the user has selected a different dataset or (ii) the user has reset the data visualization.

[0152]

[0166] 14A-14R provide a flow diagram illustrating a method 1400 of using natural language for visual analysis of a dataset applying principles of pragmatics, including processing various forms of pragmatics, according to some implementations. The steps of the method 1400 may be performed by a computer (e.g., computing device 200). In some implementations, the computer includes a display, one or more processors, and a memory (1404). FIGS. 14A-14R correspond to instructions stored in a computer memory or computer-readable storage medium (e.g., memory 206 of computing device 200). The memory stores one or more programs configured for execution by one or more processors (e.g., processor 202) (1406). For example, the operations of the method 1400 are performed, at least in part, by the data visualization generation module 234 and / or the language processing module 238.

[0153]

[0167] In some implementations, the computer displays (1408) a data visualization based on the dataset retrieved from the database using a first set of one or more queries. For example, with reference to FIG. 1, a user can associate one or more data fields from the schema information area 110 with one or more shelves (e.g., column shelf 120 and row shelf 122 in FIG. 1) of the data visualization area 112. In response to receiving a user association, in some implementations, the computer retrieves data for the data fields from the dataset using a set of one or more queries, and then displays a data visualization (e.g., data visualization 408) in the data visualization area 112 corresponding to the received user input. Displaying a data visualization is discussed in more detail above with reference to FIG. 1.

[0154]

[0168] The computer receives (1410) a first user input to specify a first natural language command associated with the displayed data visualization. In some implementations, the user input is received as text input from the user in a data entry area of ​​the display proximate to the displayed data visualization (e.g., via the keyboard 216 or via the touch-sensitive display 214). In some implementations, the user input is received as a voice command using a microphone (e.g., voice input device 220) coupled to the computer. For example, with reference to FIG. 6A, the displayed data visualization 608 relates to a less than 1M home in Ballard. Receiving input (e.g., a command / query) from a user is discussed in more detail above with reference to FIG. 1.

[0155]

[0169] Based on the displayed data visualization, the computer extracts (1412) a first set of one or more independent analytical phrases from the first natural language command. For example, referring to FIG. 6A, the first natural language command received by the computer reads "Houses under 1M in Ballard." The data visualization displayed prior to receiving the first natural language command is about past home sales in Seattle. In some implementations, the computer extracts "Houses" and "Under 1M" and "In Ballard" from the first natural language command because these analytical phrases are relevant to the displayed data visualization. If the phrases in the displayed data visualization directly reference a data field, the extraction (1412) is straightforward. Collect all phrases that directly reference the data field. In some implementations, the computer extracts or removes stop words, filler words, or any predefined set of words from the input query and extracts (1412) all other phrases from the first natural language command because they may be relevant to the displayed data visualization. Some implementations use this approach when phrases in a natural language command refer somewhat indirectly to data fields in the displayed visualization.

[0156]

[0170] According to some implementations, the language processing module 238 calculates (1414) a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analysis phrases. The framework based on the conversational dialogue model is described above with reference to FIG. 3A, FIG. 5, and FIG. 11. A center refers to an entity that helps link the utterance (sometimes referred to herein as a natural language command) to other utterances in a discourse (a sequence of utterances). The conversation center includes data attributes and values, visual characteristics, and analysis actions. Computing the conversation center based on the analysis phrase includes mapping the analysis phrase to one or more conversation centers after necessary transformation and analysis. For the example utterance "House under 1M in Ballard", the language processing module 238 processes the phrase "under 1M" and analyzes the phrase to infer that it refers to the data attribute LAST_SALE_PRICE as shown in FIG. 6C above.

[0157]

[0171] Subsequently, according to some implementations, the language processing module 238 calculates (1416) a first set of analysis functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function phrases. As described above with reference to FIG. 3A, FIG. 5, and FIG. 11, according to some implementations, each analysis function consists of a variable, an operator, and a value. In some implementations, for the example utterance "House under 1M in Ballard", the language processing module 238 creates two functions, F_CAT(House type, == condominium), F_CAT(House type, == townhouse), F_CAT(House type == single), and F_NUMERIC(Price, <, 500000). In this example, the language processing module 238 searches for one or more data attributes associated with the displayed data visualization corresponding to the first set of one or more conversation centers to identify a first set of data attributes, according to some implementations. The language processing module 238 also identifies a first set of operators (e.g., operator ==, operator <) and a first set of values ​​corresponding to the first set of data attributes by examining a first set of one or more conversational centers according to some implementations. Using the first set of variables (attributes), the corresponding first set of operators and the first set of values, the language processing module 238 constructs a first set of one or more analysis functions, thereby creating a first set of one or more function clauses.

[0158]

[0172] In some implementations, the computer updates 1418 a visualization of the data based on the first set of one or more function clauses calculated in step 1416.

[0159]

[0173] Now referring to FIG. 14B, the computer receives (1420) a second user input to specify a second natural language command associated with the displayed data visualization. In some implementations, the user input is received as text input from the user in a data entry area of ​​the display proximate to the displayed data visualization (e.g., via the keyboard 216 or via the touch-sensitive display 214). In some implementations, the user input is received as a voice command using a microphone (e.g., voice input device 220) coupled to the computer. For example, referring to FIG. 6A, the displayed data visualization 608 relates to a less than 1M home in Ballard when the computer receives the second user input "townhome". Receiving input (e.g., commands / queries) from a user is discussed in more detail above with reference to FIG. 1.

[0160]

[0174] Based on the displayed data visualization, the computer extracts (1422) a second set of one or more independent analytical phrases from the second natural language command. For example, referring to FIG. 6A, the second natural language command (620) received by the computer reads "townhomes". In some implementations, in this example, the computer extracts "townhomes" from the second natural language command because this analytical phrase is relevant to the displayed data visualization (related to Ballard's townhomes). If the phrases directly reference data fields in the displayed data visualization, the extraction (1422) is straightforward. Collect all phrases that directly reference data fields. In some implementations, the computer extracts or removes stop words, filler words, or any predefined set of words from the input query and extracts (1422) all other phrases from the second natural language command as they may be relevant to the displayed data visualization. Some implementations use this approach when the phrases in the natural language command somewhat indirectly reference data fields in the displayed visualization.

[0161]

[0175] According to some implementations, the language processing module calculates (1424) a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analysis phrases.

[0162]

[0176] The language processing module, according to some implementations, calculates (1426) cohesion between the first set of one or more analytic phrases and the second set of one or more analytic phrases, and derives the second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers based on the cohesion. As shown in FIG. 14C, in some implementations, calculating cohesion includes identifying (1434) a phrase structure of the second set of one or more analytic phrases. According to some implementations, the calculating phrase structure is described above with reference to FIG. 5 (step 522 or 510), and through the example of FIG. 6A (steps 610 and 622). In some implementations, identifying the phrase structure includes parsing (1436) the second natural language command applying a probabilistic grammar (as described above with reference to 522 or 510 of FIG. 5), thereby obtaining a parsed output. In some implementations, this step further includes inferring syntactic structures by using a part-of-speech API provided by the natural language toolkit (1438), also as described above with reference to FIG. 5. In some implementations, the parsed output is decomposed into corresponding categorical attributes and data attributes by the language processing module (1440). For example, for the utterance "townhome" (620) in FIG. 6A, the language processing module decomposes the categorical attribute into house / (or house) type. In some implementations, although not shown, the language processing module decomposes the parsed output into corresponding categorical attributes and data attributes after step 1442.

[0163]

[0177] Once the phrase structure is identified in step 1434, the language processing module, according to some implementations, identifies one or more pragmatic forms based on the phrase structure. The language processing module then derives one or more second sets of conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers based on the identified one or more pragmatic forms (1446). Figures 14D, 14E, 14H, and 14I described below show how different types of pragmatic forms are identified and how the second set of one or more conversation centers is identified based on the identified pragmatic forms.

[0164]

[0178] In some implementations, the language processing module 238 calculates (1430) a second set of one or more analysis functions associated with the second set of one or more conversation centers, thereby creating a second set of one or more function phrases. The language processing module 238 performs this step in a manner similar to step 1416 described above, using the second set of one or more conversation centers calculated in step 1426.

[0165]

[0179] In some implementations, as shown in FIG. 14P, the language processing module 238 calculates (14.186) semantic relevance between the second set of one or more extracted analytical phrases and one or more attributes of the data included in the updated data visualization, and calculates analytical functions associated with the second set of one or more analytical phrases, (14.188), thereby creating a second set of one or more function phrases based on the semantically related attributes of the data. In some implementations, the language processing module 238 calculates semantically related terms for the lexically cohesive expressions. The process of calculating semantic relevance and calculating analytical functions based on the semantic relevance has been described above with reference to FIGS. 9A-9D. In some implementations, although not shown in FIG. 14B, calculating the second set of one or more analytical functions and the second set of one or more function phrases based on the semantic relevance of the data attributes is performed in addition to (rather than instead of) step 1430 above.

[0166]

[0180] In some implementations, as shown in FIG. 14Q, the language processing module 238 computes semantic associations by training (14.190) a first neural network model on a large corpus of text, thereby learning word embeddings. In some such implementations (14.192), the first neural network model includes a Word2vec™ model. In some implementations, the language processing module 238 uses a second neural network model to compute (1494) a first word vector for a first word of a first phrase of a second set of one or more analysis phrases, where the first word vector maps the first word to the word embedding learning in step 14.190. In some such implementations (14.196), the second neural network model includes a recurrent neural network model. In some implementations, the language processing module uses the second neural network model to calculate second word vectors for the first data attribute of the one or more data attributes (14.198), where the second word vector maps the first data attribute to words of the embedding learning in step 14.190. Although not shown in FIG. 14Q, calculating the first word vector and the second word vector can be performed in parallel with the language processing module 238. The language processing module then calculates the association between the first word vector and the second word vector (14.200) using a similarity metric according to some implementations. In some such implementations (14.202), the similarity metric is based on at least (i) the Woo-Palmer distance between the meanings of words associated with the first word vector and the second word vector, (ii) a weighting factor, and (iii) a pairwise cosine distance between the first word vector and the second word vector.

[0167]

[0181] In some implementations, as shown in FIG. 14R, the language processing module 238 retrieves definitions of words of the second set of one or more analytical phrases from a publicly available dictionary (14.204), determines whether the definitions of the words include one or more predefined definitions while using a part-of-speech API provided by the natural language toolkit (14.206), and maps the one or more predefined adjectives to one or more analytical functions according to the determination that the definitions of the words include one or more predefined adjectives. These operations were described above with reference to FIG. 9B according to some implementations. The language processing module 238 calculates the type of analytical function for one or more terms of the second set of one or more analytical phrases. For example, the term "cheapest" is mapped to Bottom_N (sale price).

[0168]

[0182] Referring now back to FIG. 14B, the computer updates (1432) the visualization of the data based on the second set of one or more function clauses, according to some implementations.

[0169]

[0183] 14D, 14E, 14H, and 14I each show steps performed by the language processing module 238 to process the different types of pragmatic forms identified based on the phrase structure, and how, according to some implementations, a second set of one or more conversation centers is derived based on the pragmatics of the identified forms.

[0170]

[0184] FIG. 14D illustrates steps performed by the language processing module 238 to derive a second set of one or more conversation centers for an incomplete utterance according to some implementations. According to some implementations, the language processing module 238 identifies the pragmatic form as an incomplete utterance by determining whether one or more language elements are not present in the phrase structure (1448). In some implementations, the language processing module 238 then derives the second set of one or more conversation centers (1450) by performing a series of operations including determining a first subset of conversation centers of the first set of one or more conversation centers (1454), where the first subset of conversation centers corresponding to the one or more language elements are not present in the phrase structure, and calculating a second set of one or more conversation centers by combining the temporary set of one or more conversation centers with the first subset of conversation centers (1456). FIG. 6A above illustrates an implementation of deriving a second set of one or more conversation centers for an incomplete utterance.

[0171]

[0185] FIG. 14E illustrates steps performed by the language processing module 238 to derive a second set of one or more conversation centers for a referring expression according to some implementations. According to some implementations, the language processing module 238 identifies the referring expression as a form of pragmatics by determining whether one or more anaphoric references are present in the phrase structure (1458). In some implementations, the language processing module 238 then derives a second set of one or more conversation centers (1460) by performing a series of operations (1462) including searching the first set of one or more conversation centers to find a first subset of conversation centers corresponding to a phrase chunk in the second natural language command that includes a first anaphoric reference of the one or more anaphoric references (1464) and calculating a second set of one or more conversation centers based on the temporary set of one or more conversation centers and the first subset of conversation centers (1466). FIG. 7A above illustrates an implementation of deriving a second set of one or more conversation centers for a referring expression.

[0172]

[0186] In some implementations, the language processing module 238 determines (1468) whether the first anaphoric reference is a reference to a visualization feature of the visualization of the updated data (sometimes referred to herein as a strict reference), and, in accordance with the determination that the anaphoric reference is a strict reference, calculates (1470) a second set of one or more conversation centers based on the temporary set of one or more conversation centers and the data associated with the visualization feature.

[0173]

[0187] In some implementations, as shown in FIG. 14F, the language processing module 238 determines whether the first anaphoric reference accompanies a verb of the second natural language command (1472), and in accordance with the determination that the anaphoric reference accompanies the verb (1474), searches a first set of one or more conversation centers to find a first action conversation center that references the action verb (1476), and calculates a second set of one or more conversation centers based on the temporary set of one or more conversation centers and the first subset of conversation centers (1478).

[0174]

[0188] In some implementations, as shown in FIG. 14G, the language processing module 238 determines whether the first anaphoric reference is a decisional reference indicating some object of the environment (1480), and according to the determination that the anaphoric reference is a strict reference, calculates a second set of one or more conversation centers based on the temporary set of one or more conversation centers and the characteristics of the object (1482).

[0175]

[0189] FIG. 14H illustrates steps performed by the language processing module 238 to derive a second set of one or more conversation centers for a repair utterance according to some implementations. According to some implementations, the language processing module 238 identifies the pragmatic form as a repair utterance by determining whether the phrase structure corresponds to one or more predefined repair utterances (1484). In some implementations, the language processing module 238 then derives the second set of one or more conversation centers by performing a series of operations (1488) including calculating a second set of one or more conversation centers based on the temporary set of one or more conversation centers (1490) and updating one or more data attributes in the second set of one or more conversation centers based on one or more predefined repair utterances and phrase structures (1492). FIG. 10A above illustrates an implementation of deriving a second set of one or more conversation centers for a repair utterance.

[0176]

[0190] In some implementations, the language processing module 238 determines (1494) whether the phrase structure corresponds to the repair utterance and modifies a default behavior regarding the display of the visualization of the data, and modifies (1496) the default behavior regarding the display according to the determination that the phrase structure corresponds to the repair utterance and modifies the default behavior.

[0177]

[0191] FIG. 14I illustrates steps performed by the language processing module 238 to derive a second set of one or more conversation centers for the conjunctive expression according to some implementations. The language processing module 238 identifies the pragmatic form as a conjunctive expression (1498) according to some implementations by determining whether the second natural language command is a conjunctive expression by (i) determining the presence of an explicit or implicit conjunction in the phrase structure, and (ii) determining whether the temporary set of one or more conversation centers includes each conversation center of the first set of one or more conversation centers. In some implementations, following the determination that the second natural language command is a conjunctive expression (14.102), the language processing module 238 subsequently derives a second set of one or more conversation centers (14.100) by calculating a second set of one or more conversation centers based on the temporary set of one or more conversation centers (14.104). FIG. 8A above illustrates an implementation for deriving a second set of one or more conversation centers for utterances having conjunctions.

[0178]

[0192] In some implementations, the language processing module 238 determines whether the second natural language command has multiple conjunctions (14.106) and, in accordance with a determination that the second natural language command has more than one conjunction, calculates a second set of one or more analysis functions by linearizing the second natural language command (14.108).

[0179]

[0193] In some implementations, the language processing module 238 linearizes the second natural language command by performing the sequence of operations shown in Figure 14J. The sequence of operations includes generating a parse tree for the second natural language command (14.110), traversing the parse tree post-order (14.112), and extracting a first analytic phrase and a second analytic phrase, where the first analytic phrase and the second analytic phrase are adjacent nodes in the parse tree, and combining the first analytic function with the second analytic function by applying one or more logical operators based on one or more characteristics of the first analytic function and the second analytic function (14.114), where the one or more characteristics include an attribute type, an operator type, and a value.

[0180]

[0194] Figures 14K-O each show a different example of the final step (14.114) of combining a first analytical function with a second analytical function, according to some implementations. In each case (as shown by labels 14.116, 14.126, 14.36, 14.150, and 14.168 in the respective figures), the first analytical function includes a first attribute, a first operator, and a first value. The second analytical function includes a second attribute, a second operator, and a second value.

[0181]

[0195] In Figure 14K, the language processing module 238, according to some implementations, combines the first analytical function with the second analytical function by performing a series of operations (14.118), including determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute (14.120), determining whether the first attribute and the second attribute are the same and whether the first attribute and the second attribute are the same (14.122), and applying a union operator to combine the first analytical function and the second analytical function according to the determination that both are categorical attributes (14.124).

[0182]

[0196] In Figure 14L, the language processing module 238, according to some implementations, combines the first analytical function with the second analytical function by performing a series of operations (14.128), including determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute (14.130), determining that the first attribute and the second attribute are identical (14.132), and applying a cross operator to combine the first analytical function and the second analytical function according to a determination that the first attribute and the second attribute are not identical (14.134).

[0183]

[0197] In Figure 14M, the language processing module 238, according to some implementations, combines the first analytical function with the second analytical function by performing a series of operations (14.138), including determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute (14.140), determining that the first attribute and the second attribute are identical (14.142), determining the operator types of the first operator and the second operator according to determining that the first attribute and the second attribute are identical and both are ordinal attributes (14.144), and applying a union operator to combine the first analytical function and the second analytical function according to determining that the first operator and the second operator are both equality operators (14.148).

[0184]

[0198] In FIG. 14N, the language processing module 238 combines the first analysis function with the second analysis function by performing a series of operations (14.152), according to some implementations. The set of operations includes determining whether the first attribute is a categorical attribute or an ordinal attribute and determining whether the second attribute is a categorical attribute or an ordinal attribute (14.154); determining that the first attribute and the second attribute are identical (14.156); and in accordance with a determination that the first attribute and the second attribute are identical and both are ordinal attributes, determining operator types for the first operator and the second operator (14.160); in accordance with a determination that the first operator is a "relatively smaller" operator and the second operator is a "relatively larger" operator (14.162), determining whether the first value is less than the second value (14.164); and in accordance with a determination that the first value is less than the second value, applying a union operator to combine the first analytic function and the second analytic function (14.166).

[0185]

[0199] In FIG. 14O, the language processing module 238 combines the first analysis function with the second analysis function by performing a series of operations (14.170), according to some implementations. The sequence of operations includes determining whether the first attribute is a categorical attribute or an ordinal attribute, and determining whether the second attribute is a categorical attribute or an ordinal attribute, determining that the first and second attributes are the same (14.172), determining that the first and second attributes are the same (14.174), determining operator types for the first and second operators in accordance with a determination that the first and second attributes are the same and are both ordinal attributes (14.176), determining whether the first value is less than the second value in accordance with a determination that the first operator is a "relatively greater than" operator and the second operator is a "relatively smaller than" operator (14.180), and applying an intersection operator to combine the first and second analytic functions in accordance with a determination that the first value is less than the second value (14.184).

[0186]

[0200] 15A-15H provide a flowchart of a process (method 1500) of using natural language for visual analysis of a dataset applying principles of pragmatics (1502), including for processing responses and feedback and for handling ambiguity of a user's query, according to some implementations. The steps of method 1500 may be performed by a computer (e.g., computing device 200). In some implementations, the computer includes a display, one or more processors, and a memory (1504). FIGS. 15A-15H correspond to instructions stored in a computer memory or computer-readable storage medium (e.g., memory 206 of computing device 200). The memory stores one or more programs (1506) configured for execution by one or more processors (e.g., processor 202). For example, the operations of method 1500 are performed, at least in part, by data visualization generation module 234 and / or language processing module 238.

[0187]

[0201] In some implementations, the computer displays (1508) a data visualization based on the dataset retrieved from the database using a first set of one or more queries. For example, with reference to FIG. 1, a user can associate one or more data fields from the schema information area 110 with one or more shelves (e.g., column shelf 120 and row shelf 122 in FIG. 1) of the data visualization area 112. In response to receiving a user association, in some implementations, the computer retrieves data for the data fields from the dataset using a set of one or more queries, and then displays a data visualization (e.g., data visualization 408) in the data visualization area 112 corresponding to the received user input. Displaying a data visualization is discussed in more detail above with reference to FIG. 1.

[0188]

[0202] The computer receives (1510) a first user input to specify a first natural language command associated with the displayed data visualization. In some implementations, the user input is received as text input from the user in a data entry area of ​​the display proximate to the displayed data visualization (e.g., via the keyboard 216 or via the touch-sensitive display 214). In some implementations, the user input is received as a voice command using a microphone (e.g., voice input device 220) coupled to the computer. For example, with reference to FIG. 6A, the displayed data visualization 608 relates to a less than 1M home in Ballard. Receiving input (e.g., a command / query) from a user is discussed in more detail above with reference to FIG. 1.

[0189]

[0203] Based on the displayed data visualization, the computer extracts (1512) a first set of one or more independent analytical phrases from the first natural language command. For example, referring to FIG. 6A, the first natural language command received by the computer reads "Houses under 1M in Ballard." The data visualization displayed prior to receiving the first natural language command is about past home sales in Seattle. In some implementations, the computer extracts "Houses" and "Under 1M" and "In Ballard" from the first natural language command because these analytical phrases are relevant to the displayed data visualization. If the phrases directly reference a data field in the displayed data visualization, the extraction (1512) is straightforward. Collect all phrases that directly reference the data field. In some implementations, the computer extracts or removes stop words, filler words, or any predefined set of words from the input query and extracts (1412) all other phrases from the first natural language command because they may be relevant to the displayed data visualization. Some implementations use this approach when phrases in a natural language command refer somewhat indirectly to data fields in the displayed visualization.

[0190]

[0204] According to some implementations, the language processing module 238 calculates (1514) a first set of one or more conversation centers associated with the first natural language command based on the first set of one or more analysis phrases. The framework based on the conversational dialogue model is described above with reference to FIG. 3A, FIG. 5, and FIG. 11. A center refers to an entity that helps link the utterance (sometimes referred to herein as a natural language command) to other utterances in a discourse (a sequence of utterances). The conversation center includes data attributes and values, visual characteristics, and analysis actions. Computing the conversation center based on the analysis phrase includes mapping the analysis phrase to one or more conversation centers after necessary transformation and analysis. For the example utterance "House under 1M in Ballard", the language processing module 238 processes the phrase "under 1M" and analyzes the phrase to infer that it refers to the data attribute LAST_SALE_PRICE as shown in FIG. 6C above.

[0191]

[0205] Subsequently, according to some implementations, the language processing module 238 calculates (1516) a first set of analysis functions associated with the first set of one or more conversation centers, thereby creating a first set of one or more function clauses. As described above with reference to FIG. 3A, FIG. 5, and FIG. 11, according to some implementations, each analysis function consists of a variable, an operator, and a value. In some implementations, for the example utterance "House under 1M in Ballard", the language processing module 238 creates two functions, F_CAT(House type, == condo), F_CAT(House type, == townhouse), F_CAT(House type == single), and F_NUMERIC(Price, <, 500000). In this example, the language processing module 238 searches for one or more data attributes associated with the displayed data visualization corresponding to the first set of one or more conversation centers to identify a first set of data attributes, according to some implementations. The language processing module 238 also identifies a first set of operators (e.g., operator ==, operator <) and a first set of values ​​corresponding to the first set of data attributes by examining a first set of one or more conversational centers according to some implementations. Using the first set of variables (attributes), the corresponding first set of operators and the first set of values, the language processing module 238 constructs a first set of one or more analysis functions, thereby creating a first set of one or more function clauses.

[0192]

[0206] In some implementations, the computer updates 1518 a visualization of the data based on the first set of one or more function clauses calculated in step 1516.

[0193]

[0207] Now referring to FIG. 15B, the computer receives (1520) a second user input to specify a second natural language command associated with the displayed data visualization. In some implementations, the user input is received as text input from the user in a data entry area of ​​the display proximate to the displayed data visualization (e.g., via the keyboard 216 or via the touch-sensitive display 214). In some implementations, the user input is received as a voice command using a microphone (e.g., voice input device 220) coupled to the computer. For example, referring to FIG. 6A, the displayed data visualization 608 relates to a less than 1M home in Ballard when the computer receives the second user input "townhome". Receiving input (e.g., commands / queries) from a user is discussed in more detail above with reference to FIG. 1.

[0194]

[0208] Based on the displayed data visualization, the computer extracts (1522) a second set of one or more independent analytical phrases from the second natural language command. For example, referring to FIG. 6A, the second natural language command (620) received by the computer reads "townhomes". In some implementations, in this example, the computer extracts "townhomes" from the second natural language command because this analytical phrase is relevant to the displayed data visualization (related to Ballard's townhomes). If the phrases directly reference data fields in the displayed data visualization, the extraction (1522) is straightforward. Collect all phrases that directly reference data fields. In some implementations, the computer extracts or removes stop words, filler words, or any predefined set of words from the input query and extracts (1522) all other phrases from the second natural language command as they may be relevant to the displayed data visualization. Some implementations use this approach when the phrases in the natural language command somewhat indirectly reference data fields in the displayed visualization.

[0195]

[0209] According to some implementations, the language processing module calculates (1524) a temporary set of one or more conversation centers associated with the second natural language command based on the second set of one or more analysis phrases.

[0196]

[0210] The language processing module derives (1526) a second set of one or more conversation centers from the first set of one or more conversation centers and the temporary set of one or more conversation centers using one or more transition rules, according to some implementations.

[0197]

[0211] The computer updates (1528) a visualization of the data based on the second set of one or more function clauses according to some implementations.

[0198]

[0212] 15C, in some implementations, the language processing module 238 determines (1530) one or more data attributes corresponding to the one or more conversation center second sets. The language processing module 238 then scans (1532) the displayed data visualizations according to some implementations to identify one or more of the displayed data visualizations that include a mark of data whose characteristics correspond to a first data attribute of the one or more data attributes. In some such implementations (1534), the visualization characteristics include one or more of color, size, and shape. In some such implementations (1536), the visualization characteristics correspond to a visual encoding of the mark of data.

[0199]

[0213] The computer then highlights (1538) the marks of data whose characteristics correspond to the first data attribute according to some implementations. In some such implementations, the computer filters (1540) from the displayed data visualization including the marks of data whose characteristics do not correspond to one or more data attributes. Furthermore, in some such implementations, the computer receives (1542) a user input to determine whether to filter or highlight the marks of data, and filters or highlights the marks of data in the displayed data visualization based on the determination. FIG. 11A above illustrates the application of pragmatic principles to process responses and feedback according to some implementations. The description of FIG. 11A applies to the steps illustrated in FIG. 15C. For example, step 1102 of FIG. 11A for creating a list of all data attributes corresponds to step 1530 for determining one or more data attributes. Similarly, step 1106 for determining which of the existing visualizations encode the respective attribute corresponds to step 1532 for scanning the displayed data visualization.

[0200]

[0214] 15D, in some implementations, the computer determines (1544) whether any of the displayed data visualizations does not include a mark of data whose characteristics correspond to the first data attribute. In some implementations, following the determination (1546) that any of the displayed data visualizations does not include a mark of data whose characteristics correspond to the first data attribute, the computer generates (1548) details of a new data visualization using the first data attribute and displays (1550) the new data visualization. In some implementations, displaying (1550) the new data visualization includes determining (1552) a type of chart based on the details and generating and displaying (1554) the chart. In some such implementations (1556), the chart is arranged using a two-dimensional grid-based layout algorithm that is automatically coordinated with other data visualizations.

[0201]

[0215] Referring to FIG. 15E, in some implementations, the language processing module 238 calculates (1558) a second set of one or more analysis functions associated with a second set of one or more conversation centers, thereby creating a second set of one or more function clauses. In some implementations, the computer selects (1560) a first function clause from the second set of one or more function clauses, where the first function clause includes a parameterized data selection criterion. In some implementations, the computer selects (1562) an initial range for the value of a parameter of the parameterized data selection criterion. In some implementations, the computer displays (1564) an editable user interface control corresponding to the parameterized data selection criterion, where the user interface control displays a current value of the parameter. In some such implementations (1566), the user interface control allows for adjustment of the first function clause. Further, in some such implementations (1568), the user interface control displays a slider that allows the user to adjust the first function clause. In some implementations, the computer orders (1570) the displayed set of one or more editable user interface controls based on an order of queries of the second natural language command, the order of queries being inferred while extracting the second set of one or more analytical phrases from the second natural language command. In some such implementations, the computer uses (1572) a library that facilitates compact placement of small word-scale visualizations in the text. In some such implementations (1574), the library is Sparklificator™. FIG. 12A above illustrates an example diagram of an interface including a collection of selectable widgets presented to a user according to some implementations.

[0202]

[0216] 15F, in some implementations, the computer determines (1576) a first token of the second natural language command that does not correspond to any of the analytical phrases of the second set of one or more analytical phrases. In some implementations, the computer uses a search library to search for a correctly spelled term that corresponds to the first token by comparing the first token to one or more features of the first dataset (1578). In some such implementations (1580), the one or more features include data attributes, cell values, and related keywords of the first dataset. In some such implementations (1582), the search library is Fuse.js™.

[0203]

[0217] In some implementations, the language processing module 238 replaces (1584) the first token of the second natural language command with a correctly spelled term to obtain a third natural language command and extracts (1586) a second set of one or more analysis phrases from the third natural language command.

[0204]

[0218] In some implementations, as shown in FIG. 15G, the language processing module 238 determines whether there is no correctly spelled term corresponding to the first token (1588). Pursuant to determining that there is no correctly spelled term corresponding to the first token (1590), the language processing module parses the second natural language command to obtain a parse tree (1592), prunes the parse tree to remove the portion of the tree corresponding to the first token (1594), and extracts the parse tree to remove the portion of the tree corresponding to the first token (1596). In some such implementations, the language processing module 238 replaces the first token of the second natural language command with the correctly spelled term (1598) to obtain a third natural language command, and the computer displays the first token (15.100).

[0205]

[0219] In some implementations, the computer generates (15.102) text feedback indicating that the correctly spelled term is replaced with the first token of the second natural language command, as shown in Figure 15H. Additionally, in some such implementations, the computer displays (15.104) and highlights the correctly spelled term. Figure 12B above illustrates several example situations and corresponding feedback systems generated by a computer, according to some implementations.

[0206]

[0220] The terms used in the description of the invention herein are intended only to describe particular implementations and are not intended to limit the invention. As used in the description of the invention and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein indicates and encompasses any and all possible combinations of one or more of the associated listed items. As used herein, the terms "comprises" and / or "comprising" specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0207]

[0221] For purposes of explanation, the foregoing description has been set forth with reference to specific implementations. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The implementations have been chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable those skilled in the art to best utilize the invention and various implementations with various modifications suited to the particular applications contemplated.

Claims

1. 1. A method of using natural language for visual analysis of a data set, comprising:

1. A computer having a display, one or more processors, and a memory storing one or more programs configured for execution by the one or more processors, displaying a data visualization application on the display, the data visualization application including displaying a data visualization based on a data set retrieved from a database using a set of one or more queries; receiving, by the data visualization application, a user input specifying a natural language command related to the displayed data visualization; extracting one or more independent analytical phrases from the natural language command based on the displayed data visualization; for a first analytical phrase of the one or more extracted analytical phrases, calculating a semantic association between the first analytical phrase and a data field name of a numerical data field in the data set; identifying one or more numerical data fields having data field names with the highest calculated semantic relevance to the first analytical phrase; and selecting a numeric function in accordance with the first analytical phrase, the numeric function comparing data values ​​in the one or more numeric data fields to a threshold value; re-querying the database using an updated set of one or more queries to retrieve an updated data set, the data set including filtering the one or more identified numeric data fields according to the numeric function; displaying an updated data visualization using the updated data set on the display in the data visualization application; The method includes:

2. Computing semantic relevance generating a plurality of word embeddings for the first analysis phrase using a plurality of neural network models trained on a corpus of text; and Computing the semantic relevance based on the plurality of word embeddings. The method of claim 1 , comprising:

3. 3. The method of claim 2, wherein the plurality of neural network models includes a Word2vec model, the Word2vec model utilizing a continuous bag of words (CBoW) model architecture or a skip-gram model architecture, and the word embeddings are word vectors.

4. 3. The method of claim 2, wherein training the plurality of neural network models comprises a plurality of training methods, the plurality of training methods comprising a training method with negative sampling or a training method without negative sampling.

5. Calculating a semantic association between the first analytical phrase and a data field name of a numeric data field in the data set includes: receiving a lexical database; and extracting synonyms of the first analysis phrase from the received lexical database; The method of claim 1 , comprising:

6. Selecting the numerical function comprises: obtaining definitions of words of the first analysis phrase from a publicly available dictionary; determining whether the word definition includes a predefined adjective; and mapping the predefined adjective to an analytic function in accordance with the determination that the word definition includes the predefined adjective; The method of claim 1 , comprising:

7. 7. The method of claim 6, wherein determining whether the word definition includes a predefined adjective comprises verifying the presence of the predefined adjective using a parts-of-speech API provided by a natural language toolkit library.

8. 2. The method of claim 1 , wherein the numerical function includes a parameterized data selection criterion, and wherein re-querying the database to filter the one or more identified numerical data fields according to the numerical function includes selecting an initial range for values ​​of a parameter of the parameterized data selection criterion.

9. The method of claim 8 , further comprising displaying an editable user interface control corresponding to the parameterized data selection criteria, the user interface control displaying a current value of the parameter.

10. receiving user input at the user interface control; in response to the received user input; modifying a value of a first parameter of the parameterized data selection criteria; and Displaying the modified value in the user interface control. The method of claim 9 further comprising:

11. The method of claim 9 , wherein the user interface control is a slider.

12. If the natural language command references visualization properties of one or more marks of data within the displayed data visualization, the method further comprises: scanning the displayed data visualizations to identify one or more of the displayed data visualizations that include a mark of data whose characteristics correspond to the natural language command; and highlighting a mark in said data whose characteristic corresponds to said natural language command; The method of claim 1 further comprising:

13. The method of claim 12 , wherein the visualization characteristics include one or more of color, size, and shape.

14. The method of claim 12 , wherein the visualization characteristic corresponds to a visual encoding of a mark of data.

15. The method of claim 14 , wherein the visual encoding is one or more of color, size, and shape.

16. display, one or more processors; Memory, and 1. An electronic device including one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: displaying a data visualization application on the display, the data visualization application including displaying a data visualization based on a data set retrieved from a database using a set of one or more queries; receiving, by the data visualization application, a user input specifying a natural language command related to the displayed data visualization; extracting one or more independent analytical phrases from the natural language command based on the displayed data visualization; for a first analytical phrase of the one or more extracted analytical phrases, calculating a semantic association between the first analytical phrase and a data field name of a numerical data field in the data set; identifying one or more numerical data fields having data field names with the highest calculated semantic relevance to the first analytical phrase; and selecting a numeric function in accordance with the first analytical phrase, the numeric function comparing data values ​​in the one or more numeric data fields to a threshold value; re-querying the database using an updated set of one or more queries to retrieve an updated data set, the data set including filtering the one or more identified numeric data fields according to the numeric function; displaying an updated data visualization using the updated data set on the display in the data visualization application; An electronic device that contains instructions for:

17. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by an electronic device having a display, the one or more programs comprising: displaying a data visualization application on the display, the data visualization application including displaying a data visualization based on a data set retrieved from a database using a set of one or more queries; receiving, by the data visualization application, a user input specifying a natural language command related to the displayed data visualization; extracting one or more independent analytical phrases from the natural language command based on the displayed data visualization; for a first analytical phrase of the one or more extracted analytical phrases, calculating a semantic association between the first analytical phrase and a data field name of a numerical data field in the data set; identifying one or more numerical data fields having data field names with the highest calculated semantic relevance to the first analytical phrase; and selecting a numeric function in accordance with the first analytical phrase, the numeric function comparing data values ​​in the one or more numeric data fields to a threshold value; re-querying the database using an updated set of one or more queries to retrieve an updated data set, the data set including filtering the one or more identified numeric data fields according to the numeric function; displaying an updated data visualization using the updated data set on the display in the data visualization application; A non-transitory computer readable storage medium comprising instructions for:

Citation Information

Patent Citations

  • Multidimensional database system using natural language, and operation instruction translation method

    JP2004046394A

  • Method for rendering table by using natural language command

    JP2005267647A

  • Computer systems and methods for visualizing data with generation of marks

    US20060206512A1

  • Computer systems and methods for the query and visualization of multidimensional database

    US20060259509A1

  • Search-based detection, link, and acquisition of data

    US20160217179A1