Application of a Visual Analytics Intentional Language for Generating Data Visualizations

Abstract

The electronic device has one or more processors and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The electronic device receives a request directed to a data source. The request includes one or more intent types and one or more predefined attributes associated with the intent types. Each of the predefined attributes limits a respective data analysis operation of the respective intent type. In response to the request, for each of the intent types, the electronic device formulates a respective intent specification according to the request, including determining a respective property of the respective intent specification. The electronic device also generates a respective output specification. Each output specification includes a respective recommended data visualization type. Each output specification also includes a respective data visualization parameter that specifies how to render a data visualization of the respective recommended data visualization type.

Description

Technical Field

[0001] (Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 168,212, filed Mar. 30, 2021, entitled “Applying a Visual Analytics Intent Language to Generate Data Visualizations,” and is a continuation of U.S. Patent Application No. 17 / 219,784, filed Mar. 31, 2021, entitled “Applying a Visual Analytics Intent Language to Generate Data Visualizations,” the entire contents of each of which are incorporated herein by reference.

[0002] This application is related to the following applications, the entire contents of each of which are incorporated herein by reference. ● (i) U.S. Patent Application No. 16 / 162,332, filed Oct. 16, 2018, entitled “Methods and Systems for Building a View of a Dataset Incrementally According to Characteristics of User-Selected Data Fields,” and ● (ii) U.S. Patent Application No. 15 / 908,709, filed Feb. 28, 2018, entitled “Constructing Data Visualization Options for a Data Set According to User-Selected Data Fields.”

[0003] The disclosed implementations generally relate to data visualization, and more specifically, to systems, methods, and user interfaces for generating data visualizations and applying a visual analytics intent language to analyze data.

Background Art

[0004] Visual analysis tools enable users to visually understand a dataset and make informed decisions. Increasingly, visual analysis tools incorporate features such as natural language interaction and machine learning-based analysis and recommendations, which provide more sophisticated analytical tasks beyond simple selection of data attributes and values for generating visualizations. However, interpreting the user's analysis intent remains a challenge as the intent needs to be translated into lower-level system actions that generate appropriate responses. This process typically involves complex implementations that are hard-coded for each application and difficult to fine-tune.

[0005] Accordingly, there is a need for systems and methods that can capture the user's intent during the analysis workflow. Such systems and methods should also be able to infer insufficient or ambiguous intent and suggest effective outputs. SUMMARY OF THE INVENTION

[0006] Visual analysis tools facilitate data analysis and decision-making. These tools can propose visualization types based on selected data attributes that have perceivable defaults for visually encoding information. Many visual analysis tools currently incorporate graphical user interface conventions that employ heuristic-based approaches for presenting visualizations to users.

[0007] Despite recent advances in visual analysis tools, challenges remain when users attempt to express their higher-level analysis intentions with respect to the lower-level operations of the tools. There are several difficulties in expressing intentions. First, the intention may be vague and the entry point of the question may not match what is provided by the tool. For example, the user may be thinking about data attributes, but the tool requires the user to first select a chart type. Second, there may be a mismatch between the naming of the analysis functions of the tool and the terms used by the user. Third, the user may simply not know what actions are required to answer the question.

[0008] Furthermore, there is a spectrum of analysis intentions that can be expressed. On one hand, the analysis intention can be explicitly expressed, such as through the user's selection of data attributes from a predefined list to generate a visualization. On the other hand, the analysis intention can be vaguely defined or may involve subjective judgments, such as a user request to identify "safe" neighborhoods when looking at housing data and deciding which house to purchase.

[0009] To address these actions, visual analysis application developers have implemented custom business logic to determine appropriate responses to these expressions of intention. However, these tend to be tightly implemented and cannot be generalized across different visual analysis platforms and modalities.

[0010] There is a need to develop an analysis intention system and method for capturing the user's intention during the analysis workflow. The analysis intention system and method should also be able to infer insufficient or ambiguous intentions and suggest effective outputs. They should also be implemented at a high enough level of abstraction to be used across different visual analysis tools and different devices.

[0011] In this disclosure, a Visual Analytics Intent Language (VAIL) is presented. VAIL is a language specification that translates expressive high-level intents into low-level representations that can function as a common layer of abstraction for visual analytics applications. VAIL includes specifications that describe intents, data semantics, and outputs, which are combined with rules for editing intents, inferring ambiguous or lost intents, and proposing valid outputs for a given intent specification. VAIL helps developers manage and represent intents in their visual analytics applications.

[0012] Accordingly, such methods and interfaces reduce the user's cognitive burden and create a more efficient man-machine interface. In the case of battery-powered devices, such methods and interfaces save power and increase the battery charging interval. Such methods and interfaces can complement or replace conventional methods for visualizing data. Other implementations and advantages may become apparent to those skilled in the art in light of the description and drawings of this specification.

[0013] According to some implementations, the method is performed in an electronic device executing a first application. The electronic device includes one or more processors and a memory. The memory stores one or more programs for execution by the one or more processors. The electronic device receives a request directed to a data source. The request includes one or more intent types. Each intent type is selected from a predefined set of data analysis operations directed to the data source. The request includes one or more predefined attributes associated with the one or more intent types. Each of the one or more predefined attributes restricts the respective data analysis operation of the respective intent type. For each of the one or more intent types, the electronic device formulates a respective intent specification according to the request, including determining one or more respective properties of the respective intent specification according to the respective one or more predefined attributes and according to data fields from the data source specified in the request. For each of the one or more intent types, the electronic device generates a respective output specification. Each output specification includes a respective recommended data visualization type selected from a predefined set of data visualization types of the data source. Each output specification includes one or more respective data visualization parameters specifying how to render the data visualization of the respective recommended data visualization type.

[0014] In some implementations, the one or more intent types include one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent.

[0015] In some cases, the request includes a trend intent. Formulating each intent specification includes identifying from the data source (i) a measurement data field and (ii) a first data field representing time.

[0016] In some instances, the requirements include trend intent. Formulating each intent specification involves identifying from the data source (i) all measurement data fields, and (ii) all data fields representing time.

[0017] In some cases, the requirements include focus intent. Formulating each intent specification involves performing one or more of the following operations: (1) a sorting operation that arranges data rows from the data source in a certain order, (2) a filtering operation that filters data rows from the data source to a subset of data rows where a particular data field has a specified data value, and (3) a highlighting operation that highlights a subset of data rows from the data source.

[0018] In some cases, the requirements include encoding intent and specify a first data field. Generating each output specification involves (1) adding an encoding parameter that specifies rendering a data visualization according to the first data field, or (2) overriding an existing encoding parameter for the data visualization according to the first data field.

[0019] In some cases, the requirements include field intent and specify a first data field. Generating each output specification involves applying one or more inference rules based on semantic constraints imposed by the first data field.

[0020] In some implementations, the method further includes, for each of one or more intent types, applying one or more respective rules corresponding to each intent type to determine whether each intent type is ambiguous and / or under-informed. In accordance with the determination that each intent type is ambiguous or under-informed, the electronic device infers information to resolve the ambiguous and / or under-informed intent according to (i) one or more respective rules, (ii) metadata of the data source, and (iii) metadata of the data fields specified in the request. The electronic device updates one or more respective data visualization parameters according to the inferred information.

[0021] In some cases, applying one or more respective rules to determine whether each intent type is ambiguous and / or under-informed further includes, in accordance with the determination that the request specifies a measurable adjective and does not specify a quantity, (1) establishing that each intent type is ambiguous, and (2) inferring a numerical value of the quantity.

[0022] In some cases, the metadata of the data source includes metadata of the semantics of the data field, the data field type corresponding to the data field, the statistics of the data field, the derivation of the data field, and / or the default bin size corresponding to the data field.

[0023] In some cases, the data field type includes one or more of a category field type, a time field type, a geographic field type, a currency field type, a quantity-dependent field type, a quantity-independent field type, a latitude field type, and a longitude field type.

[0024] In some cases, the electronic device infers information to resolve the ambiguous and / or under-informed intent according to the context from previous requests, the data fields identified in previous requests, the attributes identified in previous requests, user-provided interpretations, and / or user-provided definitions.

[0025] In some implementations, each of one or more respective data visualization parameters includes an identifier corresponding to data shaping information, encoding information, and / or intent specifications.

[0026] In some implementations, after generating respective output specifications for each of one or more intent types, the electronic device visually renders data analysis in response to a request according to (i) at least one output specification of one of the one or more intent types, and (ii) the first application and / or the setting parameters of the electronic device.

[0027] According to some implementations of the present disclosure, the method is performed in an electronic device executing a first application. The electronic device includes one or more processors and a memory. The memory stores one or more programs for execution by the one or more processors. The electronic device receives a request directed to a data source. The request includes one or more intent types. Each intent type is selected from a predefined set of data analysis operations directed to the data source. The request includes one or more predefined attributes associated with the one or more intent types. Each of the one or more predefined attributes restricts the respective data analysis operation of the respective intent type. In response to the request, for each of the one or more intent types, the electronic device formulates a respective intent specification according to the request, including determining one or more respective properties of the respective intent specification according to the respective one or more predefined attributes and according to the data fields from the data source specified in the request. The electronic device applies one or more respective rules corresponding to each of the respective intent types to determine whether each of the respective intent types is ambiguous and / or insufficient in information. According to the determination that each of the respective intent types is ambiguous or insufficient in information, the electronic device infers information to resolve the ambiguous and / or insufficient intent according to (i) the one or more respective rules, (ii) the metadata of the data source, and (iii) the metadata of the data fields specified in the request. The electronic device updates each of the intent types according to the inferred information.

[0028] In some implementations, the electronic device generates respective output specifications. Each output specification includes a respective recommended data visualization type selected from a predefined set of data visualization types of the data source. Each output specification also includes (1) the inferred information and (2) one or more respective data visualization parameters that specify how to render the data visualization of the respective recommended data visualization type.

[0029] In some implementations, one or more intent types include one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent.

[0030] In some implementations, determining whether each intent type is ambiguous and / or under-informed by applying one or more respective rules further includes (1) identifying a set of data properties corresponding to each intent specification for each intent type, and (2) determining whether the claim specifies an attribute for each data property of the set of data properties. In some implementations, when the claim specifies two or more attributes of data properties within the set of data properties, the electronic device determines that each intent type is ambiguous. In some implementations, when the claim does not specify an attribute for at least one data property within the set of data properties, the electronic device 102 determines that each intent type is under-informed.

[0031] In some cases, in accordance with the determination that the claim specifies exactly one attribute for each data property of the set of data properties, the electronic device establishes that each intent type is fully specified.

[0032] In some implementations, determining whether each intent type is ambiguous and / or under-informed by applying one or more respective rules further includes establishing that each intent type is ambiguous in accordance with the determination that the claim specifies a measurable adjective and does not specify a quantity, and inferring a numerical value of the quantity.

[0033] In some implementations, each intent type is a focus intent having a focus strategy property. Inferring information to resolve an information deficit intent further includes selecting a highlighting strategy as the default focus strategy according to a determination that the claim does not include a first term selected from the group consisting of "filter", "highlight", and "zoom".

[0034] In some cases, selecting a highlighting strategy as the default strategy further includes (1) performing a sorting operation that arranges data rows from a data table of a data source in ascending or descending order according to data values of data fields in the data table, and (2) performing a highlighting operation that highlights a specified number of data values based on the ranking of the data values.

[0035] In some implementations, each intent type is a focus intent having a field property. Inferring information to resolve an information deficit intent further includes selecting, from a data source, a data field that has a text data type and has a domain greater than or equal to a quantity specified in the claim according to a determination that the claim does not specify a data field.

[0036] In some implementations, each intent type is a focus intent having a field property. Inferring information to resolve an information deficit intent further includes selecting, from a data source, a data field whose domain includes a data value specified in the claim according to a determination that the data source does not include a data field having a text data type and having a domain greater than or equal to a specified quantity.

[0037] In some implementations, each intent type is a focus intent. The method further includes determining whether the data source includes a data field having a currency field type according to a determination that the claim includes an adjective of price. According to a determination that the data source includes a data field having a currency field type, the electronic device performs a sorting operation on the data field. According to a determination that the data source does not include a data field having a currency field type, the electronic device identifies all numerical data fields in the data source and performs a respective sorting operation on each of the numerical data fields.

[0038] In some implementations, the metadata of the data source includes metadata of the semantics of the data field, the data field type corresponding to the data field, the statistics of the data field, the derivation of the data field, and / or the default bin size corresponding to the data field.

[0039] In some cases, the data field type includes one or more of a category field type, a time field type, a geographic field type, a currency field type, a quantity-dependent field type, a quantity-independent field type, a latitude field type, and a longitude field type.

[0040] In some implementations, each intent type is a distribution intent having a grouping property including a grouping field and a bin size. Inferring information to resolve an intent of lack of information further includes, according to a determination that the claim does not specify a grouping field, (1) identifying all data fields in the data source having a quantity-dependent field type, a quantity-independent field type, or a currency field type, and (2) designating each of the identified data fields as inferred information of the grouping field.

[0041] In some cases, according to the determination that the claim specifies a bin size, the electronic device replaces the bin size information in the metadata with the specified bin size.

[0042] In some implementations, the intent type is a trend intent having trend properties including a measurement data field and a date data field. Inferring information to resolve an information-deficient intent further includes, according to the determination that the claim does not include a measurement data field, (1) identifying all quantity-dependent fields in the data source and (2) designating each of the quantity-dependent fields as the inferred information of the measurement data field. Inferring information to resolve an information-deficient intent further includes, according to the determination that the claim does not include a data field having a time field type, (1) identifying all time data fields in the data source and (2) designating each of the time data fields as the inferred information of the date data field.

[0043] In some implementations, the intent type is a correlation intent for correlating two numerical data fields. Inferring information to resolve an information-deficient intent further includes (1) inferring the use of a second data field from all numerical data fields from the data source when the claim specifies only one numerical data field and (2) inferring the use of a first data field and a second data field from all numerical data fields from the data source when the claim does not specify a numerical data field.

[0044] In some implementations, the method further includes inferring information to resolve ambiguous and / or information-deficient intents according to the context from previous claims, the data fields identified in previous claims, the attributes identified in previous claims, user-provided interpretations, and / or user-provided definitions.

[0045] According to some implementations, the method is performed on an electronic device executing a first application. The electronic device includes one or more processors and a memory. The memory stores one or more programs for execution by the one or more processors. The electronic device downloads a visual analysis intent library configured to execute across a plurality of applications including the first application from a server system communicatively coupled to the electronic device. The electronic device sets the visual analysis intent library to execute on the electronic device based on the electronic device executing the first application. After setting, the electronic device receives a request directed to a data source. The request includes one or more intent types. Each of the intent types is selected from a predefined set of data analysis operations directed to the data source. The request includes one or more predefined attributes associated with the one or more intent types. Each of the one or more predefined attributes restricts the respective data analysis operation of the respective intent type. In accordance with the request, for each of the one or more intent types, the electronic device formulates each intent specification according to the visual analysis intent library, including determining one or more properties of each intent specification according to the respective one or more predefined attributes and according to data fields from the data source specified in the request.

[0046] In some implementations, the electronic device has a first device type. In some implementations, the visual analysis intent library is configured to execute across a plurality of device types including the first device type.

[0047] In some implementations, for each of the intent types, after formulating the respective intent specifications, the electronic device generates the respective output specifications according to the visual analysis intent library. Each output specification includes a respective recommended data visualization type selected from a predefined set of data visualization types of the data source. Each output specification also includes one or more respective data visualization parameters that specify how to render the data visualization of the respective recommended data visualization type.

[0048] In some implementations, one or more of the respective data visualization parameters include data shaping information, encoding information, and / or an identifier corresponding to the intent specification.

[0049] In some implementations, after generating the respective output specifications for each of one or more intent types, the electronic device visually renders data analysis in response to a request according to (i) at least one output specification of one of the one or more intent types, and (ii) the first application and / or the setting parameters of the electronic device.

[0050] In some implementations, visually rendering data analysis in response to a request further complies with an electronic device having a first device type.

[0051] In some cases, at least one output specification of one of the one or more intent types includes a first output specification corresponding to the first intent type and having inferred information. Visually rendering data analysis further includes displaying the information inferred in the first application.

[0052] In some cases, the electronic device displays a data widget that allows the inferred information to be modified. The electronic device receives a user interaction with the data widget. In response to the user interaction, the electronic device edits a first intent specification. The electronic device updates a first output specification according to the edited first intent specification. The electronic device generates and visually renders updated data analysis according to the updated first output specification and the first application and / or the setting parameters of the electronic device.

[0053] In some implementations, the electronic device is a voice assistant device. The method further includes, after generating each output specification, (1) converting each output specification into a respective spoken response, and (2) outputting each spoken response.

[0054] In some cases, each output specification includes inferred information. Outputting each spoken response includes outputting a spoken explanation of the inferred information.

[0055] In some implementations, the first application includes a graphical user interface. The method further includes receiving a request via the graphical user interface, including identifying one or more intent types.

[0056] In some cases, the electronic device generates respective data visualizations according to each output specification. The electronic device displays the data visualizations on a graphical user interface.

[0057] In some implementations, the first application is a messaging application. The method further includes, after generating each output specification, (1) converting a visual specification into a text output, and (2) displaying the text output in the messaging application.

[0058] In some implementations, one or more intent types include one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent.

[0059] In some implementations, before receiving a request, the electronic device receives a natural language command. The electronic device analyzes the natural language command according to a visual intent specification language to form a request.

[0060] In some cases, the electronic device determines, according to the analysis, that a trend intent type is included according to a natural language command in which one or more intent types have the term "time" or the term "trend".

[0061] In some cases, the electronic device determines, according to the analysis, that a focus intent type is included according to a natural language command in which one or more intent types have a measurable adjective.

[0062] In some implementations, the method further includes updating a first intent specification formulated from a previous request by correlating the first intent specification with an intent specification formulated in a request.

[0063] In some cases, the previous request includes a first data field from a data source. The request specifies replacing the first data field with a second data field from the data source. Updating the first intent specification includes replacing the first data field with the second data field.

[0064] In some implementations, the electronic device includes one or more processors, a memory, and one or more programs stored in the memory. The programs are configured to be executed by the one or more processors. The one or more programs include instructions for performing any of the methods described herein.

[0065] In some implementations, the non-transitory computer-readable storage medium stores one or more programs configured to be executed by an electronic device having one or more processors and a memory. The one or more programs include instructions for performing any of the methods described herein.

[0066] Accordingly, a method, a system, and a user interface are disclosed for capturing user intent, inferring insufficient or ambiguous intent, and proposing effective outputs.

Brief Description of the Drawings

[0067] To better understand the foregoing system, method, and graphical user interface, as well as additional systems, methods, and graphical user interfaces that provide data visualization analysis, reference is made to the following detailed description in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to corresponding parts throughout the figures.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 14F

Figure 14G

Figure 14H

Figure 14I

Figure 14J

Figure 15

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 16E

Figure 16F

Figure 16G

Figure 17A

Figure 17B

Figure 17C

Figure 18A

Figure 18B

Figure 18C

Figure 18D

Figure 18E

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 19E

Figure 19F

Figure 19G

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 20E

Figure 20F

[0068] Reference is now made to the implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details.

DETAILED DESCRIPTION OF THE INVENTION

[0069] Visual analysis tools are increasingly attempting to make data exploration more accessible and intelligent. The expression of intent in these tools has become more subtle, enabling more abstract and high-level analysis queries. This disclosure describes a Visual Analysis Intent Language (VAIL) to support the development of expressive visual analysis tools.

[0070] VAIL is a language that encapsulates the semantics for analysis intents that developers can incorporate into visual analysis applications to translate expressive high-level intents into low-level representations. VAIL can function as a general layer of abstraction for visual analysis applications.

[0071] VAIL includes specifications that describe intents, data semantics, and outputs, which are combined with rules for editing intents, inferring insufficient (e.g., omitted) or ambiguous (e.g., vague) intents, and proposing valid outputs for a given intent specification. VAIL helps developers manage and represent intents in their visual analysis applications.

[0072] In this disclosure, the terms "VAIL", "VAIL engine", and "visual analysis intent system" are used interchangeably.

[0073] I. Environment Some implementations of this disclosure generate data visualizations by applying VAIL, as shown in FIG. 1. According to some implementations of this disclosure, VAIL can be downloaded from a server (e.g., server system 300) and configured to be executed on an electronic device 102 (e.g., as a visual analysis intent system 230, also referred to as a VAIL engine). The visual analysis intent system 230 is used in conjunction with other visual analysis applications running on the electronic device 102.

[0074] FIG. 1 illustrates that VAIL can be executed on a plurality of electronic devices 102 such as an electronic device 102-1, a mobile electronic device 102-2 having a display screen (e.g., a mobile phone, a smart watch, or a tablet), etc., and on a voice assistant device 102-3 (e.g., having or not having a display screen). In this example, each of the electronic devices 102 has a different device type and includes a respective visual analysis application. For example, the computing device 102-1 includes a data visualization application having a data visualization user interface 104 (e.g., a graphical user interface). The electronic device 102-2 includes a chat application 106. The voice assistant device 102-3 includes a voice assistant application that receives voice input 108 from a user. In some implementations, VAIL can also be used on a recommendation system or other smart tools for visual analysis.

[0075] In an exemplary usage scenario, an electronic device 102 that executes VAIL receives (112) a request directed to a data source 110 from an application on the electronic device 102 (e.g., a client application 224 such as a data visualization application, a chat application, or a voice assistant application). In accordance with the request, the electronic device 102 formulates (114), for each of the intended types, respective intent specifications (e.g., using VAIL). As disclosed herein, VAIL defines a set of data properties for each intent specification having a respective intent type. VAIL also defines rules for editing intents and for inferring ambiguous or under-informed intents. By applying the rules established by VAIL, the electronic device 102 can determine (116) whether the intent type is ambiguous and / or under-informed. For example, the electronic device 102 can determine that each intent type is ambiguous when the request specifies multiple attributes for a data property in the set of data properties, or when the request includes a vague modifier such as "top" or "best". As another example, the electronic device 102 can determine that each intent type is under-informed when the request does not specify an attribute for at least one data property in the set of data properties. In some implementations, in accordance with the determination that the intent type is ambiguous or under-informed, VAIL infers information to resolve the ambiguous and / or under-informed intent, in accordance with one or more rules and the metadata of the data source and / or data fields specified in the request.

[0076] Figure 1 depicts, in some implementations, an electronic device 102 generating (118) one or more respective output specifications corresponding to an intended type. VAIL has a general specification for describing a plurality of possible outputs. For example, the output specification can include one or more respective data visualization parameters such as each recommended data visualization type and the encoding and data shaping parameters that specify how to render the data visualization of each recommended data visualization type. If each intended type is information - deficient or ambiguous and information is being inferred, the output specification includes the inferred information. In some implementations, after each output specification is generated, the electronic device 102 can convert each output specification into a visual grammar such as VizQL or Vega - Lite, or into a non - visual form such as text or audio (120).

[0077] Figure 1 also illustrates an electronic device that renders data analysis in response to a request. For example, computing device 102 - 1, in response to a request, displays a data visualization 122 (e.g., a bar graph) within a data visualization user interface 104. Electronic device 102 - 2 displays a visualization 124 that includes two line graphs in response to the query "How were last year's sales?" In some implementations, the visualization 124 is an image file (e.g., having an image file format such as.jpg or.tif) generated by a data visualization module to be suitable for display on a chat application. Figure 1 also shows a voice assistant device 102 - 3 that outputs a spoken (e.g., audio) response 126 in response to a voice input 108. In some implementations, the spoken response 126 can include an explanation of how information - deficient and / or ambiguous terms were interpreted (e.g., "best" is interpreted as having the highest RP points). In some implementations, the electronic device 102 also follows up with additional questions or prompts to further clarify the user's intent.

[0078] II. Block Diagram FIG. 2 is a block diagram illustrating an electronic device 102 in accordance with some implementations. In some implementations, the electronic device 102 is also known as a computing device. Various examples of the electronic device 102 include desktop computers, laptop computers, tablet computers, display assistant devices, mobile phones, and voice assistant devices.

[0079] The electronic device 102 typically includes one or more processing units (processors or cores) 202, one or more networks or other communication interfaces 204, memory 216, and one or more communication buses 205 for interconnecting these components. In some implementations, the communication bus 205 includes circuitry (sometimes called a chipset) that interconnects and controls communications between system components.

[0080] The electronic device 102 includes a communication interface 206. The user interface 206 optionally includes a display 208. In some implementations, the electronic device 102 includes input devices such as a keyboard, a mouse, and / or other input buttons 210. Alternatively or additionally, in some implementations, the display 208 includes a touch-sensing surface, in which case the display 208 is a touch-sensing display. In some implementations, the touch-sensing surface is configured to detect various swipe gestures (e.g., continuous gestures in the vertical and / or horizontal directions) and / or other gestures (e.g., single / double taps). In an electronic device having a touch-sensing display 208, a physical keyboard is optional (e.g., a soft keyboard can be displayed when keyboard input is required). The user interface 206 also includes an audio output device 214 such as a speaker or an audio output connection connected to a speaker, earphone, or headset. Further, some electronic devices use a microphone and voice recognition to supplement or replace a keyboard. In some implementations, the electronic device 102 includes an audio input device 214 (e.g., a microphone) for capturing audio (e.g., the voice of the user).

[0081] In some implementations, memory 216 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, memory 216 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 216 includes one or more storage devices located remotely from processor 202. Memory 216, or alternatively the non-volatile memory device within memory 216, includes a non-transitory computer-readable storage medium. In some implementations, the computer-readable storage medium of memory 216, or memory 206, stores the following programs, modules, and data structures, or subsets or supersets thereof. ● An operating system 218 that handles various basic system services and includes procedures for performing hardware-dependent tasks. ● A communication module 220. It is used to connect electronic device 102 to other computers and devices, which is performed via one or more communication network interfaces 204 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and the like. ● A web browser 222 (or other client application) that enables a user to communicate with remote computers or devices over a network. One or more client applications 224, such as a chat application, a messaging application, a data visualization application, a voice assistant application, and / or other applications for visual analytics. In some implementations, the client application 224 provides a graphical user interface (GUI) for a user to construct visual graphics (e.g., an individual data visualization or a dashboard with multiple related data visualizations) or enter an intent type. In some implementations, the client application 224 includes a natural language user interface for a user to enter natural language commands or queries. In some implementations, the client application 224 includes a voice assistant interface for a user to enter spoken commands. In some implementations, the client application 224 runs as a stand-alone application. In some implementations, the client application 224 runs within a web browser 222 (e.g., as a web application 322). In some implementations, the client application 224 includes a data visualization generator that automatically generates and displays corresponding visual graphics (also referred to as "data visualizations" or "data viz") using user input (e.g., natural language input) or according to an output specification 236 provided by a visual analytics intent system 230. In some implementations, the client application includes a widget generator that generates widgets including user-selectable options. For example, a "sort" widget is generated in response to a user selecting (e.g., hovering) over a sort field (e.g., a natural language term identified as a sort field).The sorting widget includes user-selectable options such as "ascending", "descending", and / or "alphabetical", and the user can easily select how to sort the fields selected from the widget. ● A parser 226 for parsing natural language commands (e.g., into tokens). In some implementations, the parser 226 returns tokens by removing stop words and lemmatization. In some implementations, the parser 226 parses natural language commands according to the Visual Analysis Intent Language (VAIL) to form requests to the visual analysis intent system 230. In some implementations, the electronic device 102 determines the intent type 229 according to the parsing. ● A converter 228 for converting the output specification 236 into a visual form (e.g., VizQL or Vega-lite) or a non-visual form (e.g., text or audio). ● An intent type 229 for identifying the type of user intent. In some implementations, the intent type 229 includes one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent. ● A visual analysis intent system 230 (e.g., a VAIL engine) including the following. ○ One or more intent specifications 234 formulated by the visual analysis intent system 230. The intent specifications 234 capture information regarding one or more different types of intents expressed by a user. In some implementations, the intent specifications 234 include a set of an intent type corresponding to the intent specification, a unique identifier, and properties (e.g., corresponding to the intent type). In some implementations, the visual analysis intent system 230 formulates each intent specification in accordance with a request (e.g., from the client application 224). In some implementations, formulating each intent specification includes determining one or more respective properties of each intent specification in accordance with predefined attributes and / or data fields from the data source 110 specified in the request. In some implementations, formulating each intent specification includes inferring one or more properties (e.g., by applying one or more rules defined by the VAIL engine). ○ One or more output specifications 236 generated by the visual analysis intent system 230. In some implementations, the visual analysis intent system 230 generates one or more output specifications 236 for each intent type included in the request. ○ An intent editing subsystem 238 for adding, modifying, or deleting intent specifications. ○ An inferring intent subsystem 240 for inferring information - insufficient or ambiguous intents in accordance with one or more rules corresponding to each intent type. In some implementations, the intent inference subsystem 240 utilizes metadata regarding the data source (e.g., data semantics) 246 (e.g., metadata of a data table corresponding to the data source) and / or metadata of the data fields specified in the request to infer information - insufficient and / or ambiguous intents. ○ An output proposal subsystem 242 for proposing a valid output for a specified intent. In some implementations, VAIL defines rules for converting an intent type into information in an output specification. ○ An output conversion subsystem 244 for converting the output specification 236 into modalities. Further details of the visual analysis intent system 230 and its use are described in FIGS. 4-17. ● Zero or more databases or data sources 110 (e.g., the first data source 110-1) used by the application 224 and / or the data visualization application 230. In some implementations, the data source may be stored as a spreadsheet file, a CSV file, an XML file, a flat file, a JSON file, a table of a relational database, a cloud database, or a statistical database. ● Metadata / data semantics 246 of the data source (e.g., metadata / data semantics 246-1 corresponding to the first data source) used by the visual analysis intent system 230 to infer insufficient and / or ambiguous intent. In some implementations, the metadata 246 of the data source includes semantics of data fields, data field types corresponding to the data fields, statistics of the data fields, derivations of the data fields, and / or metadata of default bin sizes corresponding to the data fields. The semantics of the data fields of the data source, or the data semantics, describe a particular set of data with which the user is interacting. This includes a list of fields, their roles, and summary statistics. In some implementations, the data semantics may be retrieved from the database or curated by the user to enable the visual analysis intent system 230 to better infer intent. In some implementations, the metadata 246 is utilized by the intent inference subsystem 240 and the output proposal subsystem 242 to infer ambiguous and / or insufficient intent and propose an output. ● An API 248 that receives API calls from one or more applications (e.g., the web browser 222, the client application 224, and the visual analysis intent system 230), translates the API calls into appropriate actions, and performs one or more actions.

[0082] Each of the identified executable modules, applications, or sets of procedures described above may be stored in one or more of the memory devices described above and corresponds to a set of instructions for performing the functions described above. The identified module or program (i.e., set of instructions) described above need not be implemented as a separate software program, procedure, or module, and thus various subsets of these modules may be combined in various implementations or otherwise rearranged. In some implementations, memory 216 stores a subset of the modules and data structures identified above. In some implementations, memory 216 stores additional modules or data structures not described above.

[0083] FIG. 2 shows electronic device 102, but FIG. 2 is intended more as a functional illustration of various features that may exist rather than as a structural overview of the implementations described herein. In fact, as will be recognized by those of ordinary skill in the art, it is possible to combine the items shown separately, and some items may be separated.

[0084] FIG. 3 is a block diagram illustrating an exemplary server system 300 according to some implementations. In some implementations, server system 300 is a data visualization server. In some implementations, server system 300 may host one or more libraries 322, one or more databases / data sources 110, and / or metadata 246 corresponding to the data sources. In some implementations, server system 300 may provide various executable applications or modules. Server system 300 typically includes one or more processing units / cores (CPUs) 302, one or more communication network interfaces 304, memory 314, and one or more communication buses 312 for interconnecting these components. In some implementations, server system 300 includes a user interface 306 that includes a display device 308 and one or more input devices 310 such as a keyboard and a mouse. In some implementations, communication bus 312 may include circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components.

[0085] In some implementations, memory 314 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 314 includes one or more storage devices located remotely from CPU 302. Memory 314, or alternatively the non-volatile memory device within memory 314, includes a non-transitory computer-readable storage medium.

[0086] In some implementations, memory 314, or the computer-readable storage medium of memory 314, stores the following programs, modules, and data structures, or subsets thereof. ● An operating system 316 that handles various basic system services and includes procedures for performing hardware-dependent tasks. ● A network communication module 318 used to connect to other computers or electronic devices (e.g., electronic device 102) such as servers through one or more communication network interfaces 304 (wired or wireless) and one or more communication networks such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. ● A web server 320 (such as an HTTP server) that receives web requests from users and responds by providing a response web page or other resources. ● A library 322. In some implementations, the library 322 may include a visual analysis intent library 323 that is downloaded onto the electronic device 102 and configured for use (e.g., as the visual analysis intent system 230 or VAIL). ● An intent type 229. ● A data visualization web application 324 that can be downloaded and executed by a web browser 428 on a user's electronic device 102. Generally, the data visualization web application 324 has the same functions as a desktop data visualization application (e.g., a data visualization application including a data visualization user interface 104), which is described in a patent application incorporated herein by reference. However, it provides flexibility of access from any device located anywhere with network connectivity and does not require installation and maintenance. In some implementations, the data visualization web application 324 includes various software modules for performing specific tasks. In some implementations, the web application 324 includes a user interface module 326 that provides a user interface for all aspects of the web application 324. In some implementations, the web application 324 includes a data retrieval module 328 that builds and executes queries to retrieve data from one or more data sources 110. The data source 110 can be stored locally on the server system 300 or in an external database. In some implementations, data from two or more data sources can be mixed. In some implementations, the data retrieval module 328 builds queries using a visual specification 344. In some implementations, the data visualization web application 324 also includes a data visualization compiler and a data visualization virtual machine (not shown). ● A natural language processing interface 330 that receives and analyzes queries provided by a computer system or electronic device 102 including a natural language input function. In some implementations, the query includes natural language input provided by a user of the electronic device 102 (e.g., via a chat application 106). In some implementations, the natural language processing interface 330 can identify analysis expressions 332 such as aggregate expressions, group expressions, filter expressions, restriction expressions, and / or sort expressions. ● The natural language processing module 162 may also include a dependency determination module 334, which searches for dependencies in the database / data source 110 to determine how specific terms and / or phrases are related (e.g., dependent). ● In some implementations, the natural language processing interface 162 includes a filter generation module 532 that determines whether one or more filters are associated with fields modified by the user. The filter generation module 532 generates one or more filters based on user selections. ● A data visualization platform 338 including the following. ○ A data visualization generation module 340 that automatically generates and displays a corresponding visual graphic (also referred to as "data visualization" or "data viz") using user input (e.g., natural language input). ○ An optional visual representation generation module 342 that generates a representation of the data visualization and transmits the generated representation to the electronic device 102. In some implementations, the representation of the data visualization is an image file of the data visualization (e.g., the image file is a.png,.tiff, or.jpg file). In some implementations, the representation of the data visualization includes a link to an image. In some implementations, the representation of the data visualization is a link to the data visualization (e.g., a URL link). ○ A visual specification 346 used to define the characteristics of the desired data visualization. In some implementations, the information provided by the electronic device 102 (e.g., user input) is stored as a visual specification. In some implementations, the visual specification 346 includes previous natural language commands received from the user or properties specified by the user via natural language commands. In some implementations, the visual specification 346 includes two or more aggregations based on different levels of detail. ●Zero or more databases or data sources 110. In some implementations, the data source is stored as a spreadsheet file, CSV file, XML file, flat file, or JSON file, or is stored in a relational database. For example, a user selects one or more databases or data sources 110 (which may be stored in the server system 300 or remotely), selects data fields from the data source, and uses the selected fields to define visual graphics. ●Metadata / data semantics 246 of the data source (e.g., metadata / data semantics 246-1 corresponding to the first data source), used by the server system 300 to infer insufficient information and / or ambiguous intent. In some implementations, the metadata 246 of the data source includes semantics of data fields, data field types corresponding to the data fields, statistics of the data fields, derivation of the data fields, and / or metadata of default bin sizes corresponding to the data fields. ●An API 348 that receives API calls, translates the API calls into appropriate actions, and performs one or more actions.

[0087] In some implementations, the server system 300 further includes an inference module (not shown) that uses one or more inference rules to resolve insufficient information (e.g., omitted information) or ambiguous (e.g., vague) natural language commands (e.g., expressions or utterances) directed to the database or data source 110. In some implementations, the one or more inference rules are defined according to the Visual Analytics Intent Language (VAIL).

[0088] In some implementations, a canonical representation is assigned to the parse representation 332 (e.g., by the natural language processing module 330) to address the problem of the sudden increase in ambiguous parsing of natural language queries. The canonical structure is clear from the perspective of the parser, and the natural language processing module 330 can quickly select among multiple parses to form an intermediate representation.

[0089] Each of the identified executable modules, applications, or sets of procedures described above can be stored in one or more memory devices and corresponds to a set of instructions for performing the functions described above. The identified modules or programs (i.e., sets of instructions) described above need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of these modules can be combined in various implementation forms or otherwise rearranged. In some implementation forms, the memory 314 stores a subset of the modules and data structures identified above. In some implementation forms, the memory 314 stores additional modules or data structures not described above.

[0090] FIG. 3 shows the server system 300, but FIG. 3 is intended more as a functional explanation of various features that can exist rather than as a structural overview of the implementation forms described herein. In fact, as will be recognized by those skilled in the art, it is possible to combine the separately shown items and some items can be separated. In addition, some of the programs, functions, procedures, or data shown above with respect to the server 300 can be stored or executed on the computing device 102. In some implementation forms, the functions and / or data can be allocated between the computing device 200 and one or more servers 300. Further, those skilled in the art will recognize that FIG. 3 need not represent a single physical device. In some implementation forms, the server functions are allocated across a plurality of physical devices including the server system. As used herein, references to a "server" or "data visualization server" include various groups, collections, or arrays of servers that provide the described functions, and physical servers need not be physically collocated (e.g., individual physical devices can be spread throughout the United States or the world).

[0091] III. Visual Analysis Intent System Figure 4 shows the connection to the visual analysis intent system 230 (e.g., VAIL engine) and the client application according to some implementation forms.

[0092] In Figure 4, the arrows entering the visual analysis intent system 230 (e.g., arrows 402, 404, and 406) represent commands (e.g., requests or instructions) from the client application 224, and the arrows exiting the visual analysis intent system 230 (e.g., arrows 408, 410, and 412) represent specifications (e.g., generated by the visual intent analysis system 230 that the client application uses to present responses to the user).

[0093] In some implementation forms, the client application 224 includes a user interface (e.g., user interface 206). In a typical cycle of interaction, the user interface 206 accepts input from the user. According to the user input, the client application 224 issues a request (e.g., a command) describing the user intent to the visual analysis intent system 230. The request includes the identification of one or more intent types. According to the request, the visual analysis intent system 230 formulates (e.g., creates, updates, or deletes) one or more intent specifications 234. In some implementation forms, the user intent can be insufficient in information and / or ambiguous. The visual analysis intent system 230 lacks the details of the insufficient intent (e.g., the attributes corresponding to the properties) and resolves any ambiguity where the intent is ambiguous or vague.

[0094] In some implementations, after an intent is inferred, the user interface requests the visual analysis intent system 230 to propose an output. The visual analysis intent system 230 then generates one or more output specifications 236, which can be converted into various types of outputs that are adaptable to the display modality of the client application 224 (and the electronic device 102), e.g., via the output conversion subsystem 244. FIG. 4 shows that the output specification 236 can be converted into various types of outputs such as visualizations 411, text 413, audio, or images that the client application 224 can display to the user. In some situations, based on the inferred intent and the output response, the user can decide to modify the intent and repeat the cycle when repeating in the analysis of the data.

[0095] In some implementations, the client application 224 and the visual analysis intent system 230 are part of a single application that includes several subsystems, and the interaction represents an interaction within the subsystems.

[0096] As shown in FIG. 4, the visual analysis intent system 230 includes an intent specification 234, an output specification 236, data semantics / metadata 246 corresponding to one or more data sources 110, a subsystem for editing the intent (e.g., the edit intent subsystem 238), a subsystem for proposing an output (e.g., the output proposal subsystem 242), and a subsystem for converting the output (e.g., the output conversion subsystem 244). Each of these components is discussed below.

[0097] A. Field Many parts of the visual analysis intent system 230 refer to data fields from the data source 110 (or the data table corresponding to the data source 100).

[0098] Figure 5 illustrates BNF (Backus-Naur Form) generation for defining data fields (e.g., field) according to some implementations. A field reference describes either an explicit field-instance or a reference to a variable that refers to multiple possible field instances (e.g., var-name, [index]).

[0099] In some implementations, a field-instance may include a name (e.g., the field name) to be used when binning values, an optional derivation (e.g., an aggregation type such as sum or fraction, or a time derivation such as year or time), and an optional bin-count.

[0100] A variable field (e.g., field-var) associates a variable field name (e.g., var-name) with a list of field instances (e.g., {field}), enabling the list to be referenced from various specifications. A field-var is referenced by name. An optional index value is used when two or more intents or output properties reference the field variable, and each reference should be to a different field instance within the list of variables.

[0101] B. Intent Specification 234 Figure 6 illustrates BNF generation for defining an intent specification according to some implementations.

[0102] A user's intent is described by vail-intent, which captures information about one or more different types of intents expressed by the user. In some implementations, the intent type is provided by the client application 224 (e.g., using editing rules). As illustrated in Figure 6, vail-intent is defined as the following tuple. ● vail-intent := (intents, field-vars)

[0103] The vail-intent includes field-vars if there are lists of intent specifications and those created by inferring the rules of the visual analysis intent system 230. FIG. 6 shows that a single intent is defined by the following tuple. ● intent := (id, inferred, specific-intent)

[0104] A single intent (e.g., each intent specification 234) includes a unique id, a list of inferred properties, and a specific-intent. FIG. 6 shows that the specific-intent can include a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coded intent, or a field intent. In some implementations, the specific-intent is also referred to as an intent type (e.g., intent type 229).

[0105] In some implementations, each intent specification 234 has a corresponding set of properties based on the intent type to which each intent specification corresponds. This is illustrated in FIG. 6. For example, a correlation intent includes a first field property (e.g., [field1]) and a second field property (e.g., [field2]). A distribution intent includes a grouped field property (e.g., [bin-field]) and a bin size property (e.g., [bin-count]). A fields intent includes a field property (e.g., (field)).

[0106] In some implementations, the intent type includes a focus intent. The focus intent can include a field property (e.g., [field]), a data value property (e.g., [values]), a sort-by field property for each (e.g., a data field for performing a sorting operation) (e.g., [sortby-field]), an adjective (e.g., [adjective]), a quantity (e.g., [quantity]), and a focus strategy property (e.g., [strategy]).

[0107] In some implementations, the intent type includes a trend intent. The trend intent can include a measurement data field property (e.g., [measure-field]) and a time field property (or date / time field property) (e.g., [time-field]).

[0108] In some implementations, the visual analysis intent system 230 receives a request that includes an intent type where not all properties are specified in the request. The visual analysis intent system 230 has rules for filling in missing or lacking properties as described below.

[0109] C. Metadata / Data Semantics 246 In some implementations, the visual analysis intent system 230 applies rules that utilize metadata (e.g., data semantics) 246 regarding a data source (e.g., a data table corresponding to a data source such as data source 110) to infer information about an ambiguous or information-lacking intent. The metadata 246 of the data source can include semantics of a data field, a data field type corresponding to the data field, statistics of the data field, derivation of the data field, and / or metadata of a default bin size corresponding to the data field. In some implementations, the metadata 246 is provided by the client application 224.

[0110] In some implementations, the metadata of the data source is described using data-semantics. FIG. 7 illustrates the BNF generation for defining data-semantics according to some implementations.

[0111] FIG. 7 shows that data-semantics includes a list of field names with associated information about the fields (e.g., {(name, field-info)}). Field-info includes field-type, which encapsulates important semantics regarding the field, default derivation, default binning size (e.g., [bin-count]), and statistical values of the field (e.g., [stats]). Field-type includes a categorical field type, a time field type, a geographic field type, a currency field type, a quantity-dependent field type, a quantity-independent field type, a latitude field type, and / or a longitude field type. In some implementations, the visual analysis intent system 230 uses default derivation properties and / or bin size properties when they are not specified in the requirements. The statistical values of the field (e.g., [stats]) include the data type and corresponding domain of the data field (e.g., (data-type, domain)). In some implementations, data-type includes text or numbers.

[0112] In some implementations, the metadata 238 is stored on the electronic device 102, and all VAIL engine processes are executed locally on the electronic device 102, thereby avoiding the need to send information to and / or execute queries on the server system, thereby enabling data analysis on large datasets. In some implementations, the metadata 246 is stored on the server system 300 and is fetched on-the-fly by the visual analysis intent system 230 according to the requirements.

[0113] D. Output Specification 236 VAIL has a general specification for describing multiple possible outputs. The output specification 236 describes how data should be presented to the user and is not restricted to a particular output modality. In some implementations, the visual analysis intent system 230 generates one or more respective output specifications 236 corresponding to each intent type. The output specification 236 is generated from vail-intent through the suggest-output rules as described below. The output specification 236 can be converted into a visual grammar such as VizQL or Vega-lite, or into a non-visual form such as text, speech, or audio.

[0114] According to some implementations, the output specification 236 (e.g., vail-output) is defined as the following tuple. ● vail-output := (output-sorted, output-by-id, field-vars)

[0115] output-sorted includes a list of output specifications in which the most highly recommended output is enumerated first, output-by-id provides a lookup from intent id to different output specifications generated from the corresponding intent, and field-vars includes any field variables referenced by the output specification.

[0116] Figure 8 shows a BNF production for defining a single output specification (e.g., output) according to some implementations.

[0117] In some implementations, as illustrated in Figure 8, a single output specification includes weight, intent-ids, encoding, and data-shape.

[0118] In some implementations, the output specification (e.g., output) includes weights from 0 to 100, with higher numbers being more highly recommended. The intent-ids include the ids of all the intents used to generate the output. The encoding includes the viz-type and a collection of different encoding properties. The encoding can enumerate fields greater than or equal to 0 and does not include explicit details about mark color, size, or shape, because it leaves these decisions to the output implementation. In some implementations, the encoding type is simply a suggestion and may not be relevant. For example, an output implementation that generates text may choose to use encodings to simply notify how the data should be formatted rather than how it should be encoded.

[0119] In some implementations, the entire list of encoding fields is used to determine the aggregation and level of detail of the data to be used for the output. For example, consider a data table that has points, prices, names, and varieties, with each row describing a different wine. If the encodings include only the average price and variety, then only the price and variety fields should be included, with one row per variety containing the aggregated average price.

[0120] The data-shape describes how the source data should be further modified before being used. As further illustrated in Figure 8, the data-shape includes focus and sort. The focus describes which part of the data to focus on.

[0121] In some implementations, focus includes recommended strategies on how to focus, such as filter or highlight. Output - adjective has several values with special semantics that allow the visual analysis intent system 230 to specify a broad range without the need to query data. "top" and "bottom" indicate that the output implementation should focus on only a single value, and "high" and "low" indicate that the visual analysis intent system 230 should focus on the range of the highest or lowest values. The sort formatting property describes which field to use to sort the data.

[0122] E. Output Transformation 244 In some implementations, the visual analysis intent system 230 includes an output transformation subsystem 244 for converting the output specification 236 into a visual grammar such as VizQL or Vega - Lite, or into a non - visual form such as text, audio, or spoken output.

[0123] F. Intent Editing Subsystem 238 VAIL edits the vail - intent to infer ambiguous or missing intents and defines rules for proposing vail - output based on the intent. These rules utilize information (such as metadata 246) contained within data - semantics that describe the semantics of the data fields (or data tables of the data source) within the data source 110.

[0124] In some implementations, the visual analysis intent system 230 includes an intent editing subsystem 238 that applies one or more rules defined by VAIL to edit the intent specification.

[0125] FIG. 9 illustrates BNF production rules for editing an intent specification (e.g., set - intent) according to some implementations.

[0126] As discussed above, vail-intent can include multiple intent specifications. A particular intent can be added or changed with the set-intent rule, or removed with the remove-intent rule. In some implementations, these rules use an id to identify a particular intent. All occurrences of a field instance in vail-intent can be swapped out with another field instance using replace-field, thereby enabling the user to iterate over their intents.

[0127] G. Intent Inference Subsystem 240 In some implementations, the visual analysis intent system 230 includes an intent inference subsystem 240 that applies one or more rules defined by VAIL to infer ambiguous or under-specified (e.g., missing) intents.

[0128] In some implementations, there is a set of data properties corresponding to each intent specification of each intent type. The visual analysis intent system 230 can determine that each intent type is ambiguous when the request specifies multiple attributes for a data property in the set of data properties. Each intent type can also be considered ambiguous when the request specifies a measurable adjective (e.g., "top") but no quantity. In some implementations, the visual analysis intent system 230 can determine that each intent type is under-specified when the request does not specify an attribute for at least one data property in the set of data properties. In some implementations, when the request specifies exactly one attribute for each data property in the set of data properties, the visual analysis intent system 230 establishes that the intent type is fully specified.

[0129] Many intent types have rules on how to infer missing or ambiguous intent properties. The rules examine data-semantics to make information-based decisions. In some implementations, if the visual analysis intent system 230 finds exactly one possible field for an unspecified property, the visual analysis intent system 230 fills it directly. In some implementations, if the visual analysis intent system 230 finds multiple possible fields for an unspecified property, it creates a field-var that includes a list from the intent properties and its references.

[0130] Figure 10 illustrates BNF production rules for inferring information to resolve under-specified focus intents, according to some implementations.

[0131] The focut intent is related to directing the analysis towards a subset of data values. As previously explained in FIG. 6, the input specification for the focus intent type can include field properties (e.g., [field]), data value properties (e.g., [values]), data fields for performing sorting operations (e.g., [sortby-field]), adjectives (e.g., [adjective]), quantity properties (e.g., [quantity]), and focus strategy properties (e.g., [strategy]).

[0132] The strategy property is related to an approach for directing the user's attention to a subset of data values. FIG. 10 shows that in some implementations, the visual analysis intent system 230 can choose (e.g., infer) to highlight the default strategy property according to a determination that a request does not specify the strategy property (e.g., the request does not contain terms such as "filter", "highlight", or "zoom"). This is because research has shown that it is often important to show data within context. In some implementations, according to the "highlight" strategy, the visual analysis intent system 230 generates an output specification that includes one or more recommended actions for highlighting the value described by dataShape.focus in the output specification. For example, the recommended actions can include a sorting action that arranges data rows from a data table of a data source in ascending or descending order according to the data values of data fields in the data table, and a highlighting action that highlights a specified number of data values based on the ranking of the data values.

[0133] In some implementations, according to a determination that a focus intent does not specify the strategy property, the visual analysis intent system 230 can select "zoom" as the strategy property. In some implementations, according to the "zoom" strategy, the visual analysis intent system 230 generates an output specification that includes a recommended action for zooming in on the value described by dataShape.focus in the output specification.

[0134] In some implementations, according to a determination that a focus intent does not specify the strategy property, the visual analysis intent system 230 can select "filter" as the strategy property. In some implementations, according to the "filter" strategy, the visual analysis intent system 230 generates an output specification that includes a recommended filtering action for filtering the data rows of a data source to a subset of data rows where a specific data field has a specific data value.

[0135] Referring back to FIG. 10, in some implementations, in accordance with the determination that a field property (e.g., field) is not specified in the request, the visual analysis intent system 230 can select a data field that has a text data type and has more domains than the quantity specified in the request from the data source. In some implementations, in accordance with the determination that a field property (e.g., field) is not specified, the visual analysis intent system 230 can select a data field whose domain contains the data value specified in the request. In some implementations, when sort-by is not specified, if the request contains an adjective referring to price (e.g., "cheap" or "expensive"), the visual analysis intent system 230 can use all data fields within a data source having a currency data type (e.g., CCurrency). In some implementations, in accordance with the determination that the data source does not contain a data field having a currency field type, the visual analysis intent system 230 infers the sort-by field by selecting all numeric data fields.

[0136] FIG. 11 shows BNF generation rules for inferring information to resolve the correlation intent of insufficient information according to some implementations.

[0137] The correlation intent describes the user's desire to understand the correlation between data fields. An exemplary request corresponding to the correlation intent can be "Show the correlation between horsepower and MPG" where the two data fields are "horsepower" and "MPG". As shown in FIG. 6, the correlation intent includes a first field property (e.g., [field1]) and a second field property (e.g., [field2]). In some implementations, the correlation intent is information-insufficient because at least one of the two field properties is not specified in the request.

[0138] In some implementations, in accordance with the determination that the correlation intent is information - insufficient, the visual analysis intent system 230 identifies a list of all numerical fields within the data source 110. Here, the variable "list" in FIG. 11 is an array. That is, the visual analysis intent system 230 creates an array that is a list of all numerical data fields. Next, within the intent specification, if the first data field is not specified (i.e., "!i.field1"), the visual analysis intent system 230 fills this field with element 0 from the list (i.e., the first field within the field list). If the second field is not specified, the visual analysis intent system 230 fills the field with element 1 from the list (the second numerical data field). If neither field property is specified, the visual analysis intent system 230 sets the indices within the field1 and field2 properties to indicate that they should refer to different fields that are the same field variable within the list.

[0139] FIG. 12 illustrates BNF production rules for inferring information to resolve an information - insufficient distribution intent, according to some implementations. A distribution intent describes a user's desire to see the spread of values of a data field. An exemplary requirement corresponding to a distribution intent is "Show me the distribution of wine prices." As described in FIG. 6, in some implementations, a distribution intent includes a grouping field property (e.g., [bin - field]) and a bin size property (e.g., [bin - count]).

[0140] In some implementations, in accordance with the determination that the request does not specify a grouping field property, the visual analysis intent system 230 searches all quantity fields (e.g., Qi, Qd, or CCurrency). In some implementations, in accordance with the determination that the request specifies a bin size, the visual analysis intent system 230 overrides (e.g., replaces) the bin size information (e.g., [bin-count]) specified in data-semantics with the bin size specified in the request.

[0141] FIG. 13 illustrates BNF generation rules for inferring information to resolve an insufficient information trend intent, according to some implementations.

[0142] A trend intent is used to describe a user's desire to view the value of a field over time. Exemplary requests that include a trend intent include statements such as "Show me the trend of COVID cases." or questions such as "What is the price of one-bedroom apartments in San Francisco over the past five years?"

[0143] As described in FIG. 6, according to some implementations, a trend intent includes a measurement data field property (e.g., [measure-field]) and a time field property (or date / time field property) (e.g., [time-field]). In some implementations, in accordance with the determination that the request does not include a measurement field, the visual analysis intent system 230 finds all quantity-dependent fields (e.g., Qd) in the data source, creates a field-var to include all quantity-dependent fields, and references it from the measure-field property. In some implementations, in accordance with the determination that the request does not specify a time field property, the visual analysis intent system 230 identifies all time fields in the data source, creates a field-var to include all time fields, and references it from the time-field property.

[0144] In some implementations, the intent type includes a geographic intent. An exemplary request that includes a geographic intent is "Where is my sales?" In some implementations, the inference rule for an under-specified geographic intent is the same as the inference rule for an under-specified trend intent as described above, except that the inference rule for the geographic intent finds fields of type CGeo, QLat, and QLon.

[0145] In some implementations, the intent type includes an encoding intent. The encoding intent expresses a desire to modify the output or a previous intent. An exemplary request that includes an encoding intent is "Color by X" (where X is a field). In some implementations, VAIL does not define an inference rule for the encoding intent.

[0146] In some implementations, the request includes an encoding intent and specifies a first data field. In accordance with the request, the visual analysis intent system 230 generates respective output specifications by adding encoding parameters that specify rendering data visualization according to the first data field. In some implementations, the visual analysis intent system 230 generates respective output specifications that override existing encoding parameters for data visualization according to the first data field.

[0147] In some implementations, the intent type includes a fields intent. The fields intent expresses an interest in fields without an explicitly defined analysis intent. An exemplary request that includes a fields intent is "Show me sales and profit." In some implementations, VAIL does not define an inference rule for the fields intent.

[0148] In some implementations, the request includes a field intent and specifies a first data field. In accordance with the request, the visual analysis intent system 230 generates respective output specifications by applying one or more inference rules based on the semantic constraints imposed by the first data field.

[0149] H. Output Proposal Subsystem 242 In some implementations, the visual analysis intent system 230 includes an output proposal subsystem 242 that applies one or more rules for proposing vail-output based on the intent.

[0150] To propose an effective output for the intent specified by the user, VAIL defines rules for converting each intent type into information within the output specification. In some implementations, the visual analysis intent system 230 creates new output specifications from correlation, distribution, trend, geographic, and field intents.

[0151] In some implementations, the correlation intent is used to generate a scatter plot. ●vizType: “scatterPlot”, x:i.field1, y:i.field2

[0152] In some implementations, the distribution intent is used to generate a histogram.

[0153] In some implementations, the trend intent creates a line graph.

[0154] In some implementations, the encoding and focus intents modify the output specifications generated by the previous intent. If there is no output specification corresponding to these intent types, the fields from these intents can be extracted and fed into the suggest-output rules for the fields intent, leading to further refinement from the user.

[0155] The encoding intent type is used to modify the encodings of other output specifications and to add or override existing encodings. The focus intent type is used to modify the data-shape of other output specifications. This uses fields, adjectives, values, and quantities to create an output.data-shape.focus specification (e.g., "Show me the top 10 COVID cases [count]."). This creates an output.data-shape.sort specification using fields, sort-by, and adjectives.

[0156] Since the user's goals are not well understood, the output rules for fields intent depend on the data-semantics of the fields of the intent. This can leverage systems like Show Me or Voyager that are designed to generate effective visualizations using field semantics.

[0157] I. Scalability and Customization In some implementations, the visual analytics intent system 230 can also be customized according to the needs of the user. For example, developers can add their own custom intents to fit domain-specific concepts such as "best neighborhood" or technical jargon that is specific in a business context. This involves providing rules for inferring missing or ambiguous properties, along with rules for generating the corresponding output specifications. Intent properties can be inferred through an understanding of the intent type (e.g., a particular intent), such as knowing the types of fields to be used and identifying information about the data source to find appropriate matches (e.g., finding all fields that represent time).

[0158] In some implementations, the intent specification corresponding to a particular intent type can be extended to add new properties. For example, in the case of the focus intent, since there are many options for describing high or low numerical values, the attributes within the adjective property can be extended to include new adjectives. Additionally or alternatively, the infer-intent rules corresponding to the focus intent can be adjusted to include new options. As an example, in some implementations, to enable the focus intent to indicate a zoom into data, the developer can add a new strategy called "zoom", and then add custom logic to the output specification handler to zoom into the value described by dataShape.focus in output specification 236.

[0159] In some implementations, the visual analysis intent system 230 can be extended by adding new possibilities to vail-output for utilization from the suggest-output rules. For example, if a developer wants to create an infer-intent rule for creating a tree map visualization, the developer can add a new viz-type to the output specification and utilize existing encoding properties to describe how the visualization should be constructed.

[0160] In light of these principles, we know to look towards specific implementations.

[0161] IV. Implementations In some implementations, the visual analysis intent system 230 (e.g., the VAIL engine) is written in TypeScript and compiled to JavaScript.

[0162] In some implementations, the visual analysis intent system 230 includes a graphical user interface (GUI) built using React components that can be used to input and modify its built-in intent specifications.

[0163] In some implementations, the client application 224 can interact with the visual analysis intent system 230 through input commands and output specifications, and can also provide its own data sources. Inputs are communicated via VAIL commands to set intents for the engine, infer missing ambiguous intents, and propose outputs. The client application 224 can choose to show the user the inferred intents returned by the visual analysis intent system 230 and, optionally, provide a way for an analyst to refine the intent.

[0164] In some implementations, the output specifications can be interpreted in a client-specific way. The visual analysis intent system 230 also includes a built-in Vega-lite converter that can be used instead of (or in addition to) the client-specific interpretation. Data sources can be manually curated to improve the ability of the visual analysis intent system 230 to properly infer intents and propose effective outputs. Each field can specify a type such as currency or time, any preferred derivation such as average or total, and an optional bin size.

[0165] In some implementations, as a performance consideration, the visual analysis intent system 230 only looks at the metadata 246 regarding fields rather than executing live queries.

[0166] A. GUI-based application Figures 14A - 14J provide a series of screenshots for interaction with a graphical user interface-based application 224 using VAIL, according to some implementations.

[0167] FIG. 14A illustrates a graphical user interface 1400 of an application 224 displayed on an electronic device 102. The graphical user interface 1400 includes a graphical control element 1402 (e.g., a "pick data" affordance) that, when selected by a user, causes a list of data sets 1404 (e.g., data sets corresponding to respective data sources 110) to be displayed as a drop-down menu 1406. The graphical user interface 1400 also includes a graphical control element 1408 (e.g., an "add intent" affordance) that, when selected by a user, identifies one or more respective intent types (e.g., intent type 229 of FIG. 2 and specific-intents of FIG. 6). In the example of FIG. 6, the affordance 1408 includes a focus intent affordance 1408-1, a correlation intent affordance 1408-2, a trend intent affordance 1408-3, and a distribution intent affordance 1408-4.

[0168] In some implementations, as illustrated in FIG. 14A, the graphical user interface 1400 includes an intent specification region 1412 for displaying one or more intent specifications formulated by VAIL in accordance with user interaction. The graphical user interface 1400 includes an output specification region 1414 for displaying one or more output specifications generated by VAIL. The graphical user interface 1400 also includes a command region 1416 for displaying one or more commands (e.g., from the client application 224). The graphical user interface 1400 also includes a data region 1418 for displaying data (e.g., properties) specified by a user.

[0169] FIG. 14A illustrates a user interaction with the graphical user interface 1400. In this example, the user selects (1410) a data set 1404-3 corresponding to "wine". The user also selects the focus intent affordance 1408-1.

[0170] FIG. 14B illustrates that in response to a user selection of the "Wine" dataset, the graphical user interface 1400 displays a graphical control element 1420 (e.g., a "Pick a Field" affordance). Each of the elements 1420 corresponds to a respective data field within the "Wine" data source.

[0171] FIG. 14B also illustrates that in response to a user selection of the "Focus" intent, the graphical user interface 1400 displays a set 1422 of properties corresponding to the focus intent. In this example, the properties 1422 include a "Field" property, a "Value" property, a "Sort per" property, an "Adjective" property, a "Quantity" property, and a focus strategy property. The intent specification area 1412 displays the selected intent type (1424).

[0172] FIG. 14C illustrates that in response to a user selection of the drop-down button 1426 adjacent to the "Adjective" property 1422-1 in FIG. 14B, the graphical user interface 1400 displays a drop-down menu 1428 that includes a list of adjectives (e.g., measurable adjectives). The user selects a first adjective (e.g., "expensive") from the list (1430).

[0173] FIG. 14D illustrates that the graphical user interface 1400 displays a data visualization 1432 (e.g., a bar graph) in response to a user selection (e.g., user specification) of the adjective "expensive". Each of the bars in the bar graph corresponds to a wine variety. The data bars are sorted (e.g., arranged) in descending order, and the top three rows are displayed so as to be visually distinguishable from the other rows within the bar graph. In this example, the visual analysis intent system 230 infers the currency attribute "price" according to the user selection of the adjective "expensive". Then, it generates an output specification for generating a bar graph that highlights the upper range of price values. FIG. 14 also shows that the data field "variety" is the inferred field of the "field" property 1422-2, and the strategy "highlight" is inferred as the strategy property.

[0174] FIGS. 14E and 14F illustrate user interactions with the graphical user interface 1400. In FIG. 14E, the user inputs (e.g., specifies) a first value 1434 (e.g., "55") for the "value" property 1422-5. In response to the user input, the graphical user interface 1400 displays an updated data visualization 1436 (e.g., a bar graph) having data bars corresponding to wine varieties having a price of at least 55 dollars. FIG. 14F illustrates that the user is specifying a second value 1438 (e.g., "55, 65, 75"). In this example, VAIL interprets the input "55, 65, 75" as the user's intent to filter the view based on the specific values "55", "65", and "75". The visual analysis intent system 230 applies this new intent to the existing context to be able to filter by the specific values "55", "65", and "75". Then, it generates an output specification including a filtering operation that filters the data rows to a subset of the data rows where the price values are "55", "65", and "75". The graphical user interface 1400 displays an updated data visualization 1440 that reflects the output specification.

[0175] FIG. 14G illustrates a user selection of the "Titanic" data source 1404-5 and a user selection of the "Distribution" intent type 1408-4. FIG. 14H shows that in response to the user selection, the visual analysis intent system 230 automatically infers the data field "Age" within the data source 1404-5 as a binned field 1444 (e.g., grouped field property, or [bin-field]) and displays a histogram 1442. In some implementations, the visual analysis intent system 230 also picks a custom-defined bin size 1446 (e.g., bin interval or [bin-count]) for the dataset.

[0176] FIG. 14I illustrates a user selection of the "Covid" data source 1404-4 corresponding to Covid-19 cases in California. FIG. 14I also illustrates a user selection of the "Trend" intent type 1408-3.

[0177] FIG. 14J illustrates that in response to the user selection, the graphical user interface 1400 displays a time series 1446 showing the new Covid death count over time (e.g., daily). As described in FIG. 6, the trend intent includes a measurement data field property and a time field property. Here, the visual analysis intent system 230 infers the measurement field "New death count" as the measurement data field property and the field "Date" as the time field property.

[0178] B. Slackbot Application According to some implementations of the present disclosure, VAIL can support natural language intents in a Slack application (e.g., client application 224 or chat application).

[0179] FIG. 15 illustrates a system overview 1500 of a Slackbot application using VAIL according to some implementations.

[0180] The Slack application includes a Slack chatbot that employs a node.js client-server architecture. In some implementations, as illustrated in FIG. 15, the Slack chatbot includes the following components. ● Slack client 1502. In some implementations, the Slack client 1502 listens for user interaction events within the Slack interface and message events from the Slack server. In some implementations, the Slack client 1502 includes a user interface that also displays native interactive widgets for surfacing ambiguity, as illustrated in FIG. 16. ● Slack server 1504. In some implementations, the Slack server 1504 is a main application-specific server that bridges between the Slack client 1502 and the other components of the Slack application. The Slack server 1504 converts input client events (e.g., Slack messages) into appropriate API requests and, similarly, converts API responses into the correct format for responses to be sent to the client. ● Parser 1506 (e.g., Parser 226 of FIG. 2). In some implementations, the parser 1506 parses natural language input queries into tokens based on a context-free grammar (such as implemented in Eviza). These tokens are resolved as data attributes and values (with information from data sources), or as intent vocabulary such as "trend" or "best". ● VAIL engine 1508 (e.g., Visual Analytics Intent System 230). The VAIL 1508 processes intent processing and infers insufficient or ambiguous information. The server passes the parsed tokens to the VAIL, which generates one or more intent specifications and one or more output specifications, as discussed with respect to FIGS. 1, 2, and 4-13. ● VizQL module 1510. The VizQL module 1510 uses Tableau's VizQL commands to generate an image of the data visualization result based on information such as chart type, intended strategy, data attributes, and values. In some implementations, the VizQL module 1510 converts the data visualization into a format (e.g., jpg, tif, png, gif, bmp, pdf format, etc.) that can be adapted for display on the Slack client 1502. ● Natural language generation (NLG) module 1512. The NLG module 1512 uses a simple language template for NLG with predefined text and placeholders to insert information from the output specification. The grammar rules from the parser 1506 are further used to assist in the linguistic realization.

[0181] Figures 16A - 16G provide a series of screenshots for interaction with a Slack chatbot (e.g., Slackbot) according to some implementations. In this example, the Slack chatbot uses VAIL to answer questions regarding the "Wine" dataset (e.g., data source 110).

[0182] Figure 16A illustrates an exemplary graphical user interface 1600 of the Slack application displayed on the electronic device 102.

[0183] In some implementations, according to the user identification (e.g., user-specified or user-selected) of the "wine" data source, the Slack application generates message 1602, displays it on the graphical user interface 1600, and prompts the user to ask questions about the data source. In some implementations, the Slack application also displays a representation 1604 (e.g., an image file such as a.png,.tiff,.bmp, or.jpg file) indicating the attributes of the data fields in the data source. In the example of FIG. 16A, the user is interacting with the Slack application for the first time, and thus, the user selection of the "wine" data source starts a new conversation thread 1606.

[0184] FIG. 16B illustrates a user interaction with the graphical user interface 1600 according to some implementations. In this example, the user interacts with the user interface 1600 by composing (e.g., typing) a message in the input box 1607. In some implementations, the user can indirectly interact with the input box 1607 by speaking towards the microphone of the electronic device 102 (e.g., the audio input device 212) to provide a natural language command. In the example of FIG. 16B, the user inputs a natural language command (e.g., a natural language expression) "show me the expensive varieties". Here, "expensive varieties" contains an ambiguous intention because the user mentions "expensive" but does not define the number of expensive varieties that the user is interested in.

[0185] In some implementations, in accordance with user input, a Slack event with a command (e.g., a query) is passed to the Slack server 1504. The Slack server 1504 recognizes that the event is a query type event and passes the query to the parser 1506 for tokenization. In some implementations, in response to receiving the query, the parser 1506 removes stop words and lemmatization (e.g., "show" and "me") and returns tokens (["expensive", "variety"]) to the Slack server 1504. The Slack server 1504 then passes the tokenized parser output to the VAIL engine 1508 (e.g., the visual analysis intent system 230). In this example, the VAIL engine 1508 infers that "expensive" resolves to variety sorted by the currency attribute price, and the strategy indicates highlighting the top 3 items. Next, the VAIL engine 1508 generates the intent specification 1608 illustrated in FIG. 16C. The VAIL engine 1508 also generates the output specification 1610 illustrated in FIG. 16D. In some implementations, after generating the output specification, the VAIL engine 1508 passes the generated specification to the VizQL module 1510 and the NLG module 1512, which generate a visualization and a text response.

[0186] FIG. 16E illustrates a view of the user interface 1600 updated in response to user input. In this example, the conversation thread 1606 is updated to include a visualization representation 1610 (e.g., a bar graph with data bars sorted in descending order) and a text response 1612 (e.g., "The top 3 varieties are Cabernet Franc Merlot, Petit Verdot, and Bordeaux-style Red Blend. "Expensive" is interpreted as an average price of $109 or more.").

[0187] In some implementations, the Slack application applies relevant templates based on the output specifications. In the example of FIG. 16, the relevant template is "[The] [modifier] [dimension] of [N] is [data - values]. [Modifier] is interpreted as [filter - value]." and corresponds to the text response "The [modifier] varieties (dimension) of 3 ([N]) are Cabernet Franc Merlot ([data - value1]), Petit Verdot ([data - value2]), and Bordeaux - style Red Blend ([data - value3]). '[Modifier]' (expensive) is interpreted as an average price of $109 or more ([filter - value])."

[0188] As described in FIG. 16B, the phrase "expensive varieties" contains an ambiguous intention because the user mentions "expensive" but does not define the number of expensive varieties they are interested in. In this example, the VAIL engine 1508 infers, based on the ranking of data values (e.g., in descending order), the number of data values for which the quantity "3" should be highlighted. The representation 1610 in FIG. 16E shows that the top 3 data rows 1614 of the bar graph are displayed to be visually distinct from the other data rows 1616 of the bar graph (e.g., the top 3 data rows 1614 are highlighted and the other data rows are not). In some implementations, the inferred information (e.g., "top N") is displayed as a data widget 1618 that the user can interact with to refine and update the response.

[0189] As an example, assume that the user selects the data widget 1618 in FIG. 16E and changes the filter from "top 3" to "top 5". In some implementations, in response to the user selection (2042), VAIL edits each intended specification by changing the [quantity] property from the inferred value "3" to the user-specified value "5". VAIL updates each output specification according to each edited intended specification. The client application 224 generates (2048) and visually renders an updated data analysis according to each output specification. For example, the user interface 1600 can display an updated image of a bar graph in which the top 5 data bars are highlighted.

[0190] FIG. 16F illustrates another user interaction with the user interface 1600 according to some implementations. In this example, the user interacts with the user interface 1600 by typing a message (e.g., "What do you think about the best wine?") into the input box 1607. In this example, the term "best" is ambiguous. This is because "best" can be defined, for example, with respect to the number of wines sold, the highest average customer review score, or the amount of revenue generated. The term "wine" is also ambiguous because it can mean a type of wine such as chardonnay or cabernet, or a winery.

[0191] FIG. 16G illustrates a view of user interface 1600 updated in response to user input. In this example, user interface 1600 displays a visualization representation 1620 (e.g., a bar graph with data bars sorted in descending order) and a text response 1622 (e.g., "Here, the wineries are sorted in descending order by average points. 'Best' is interpreted as the top 1."). In this example, response 1622 includes an explanation of representation 1620 and an explanation of how an ambiguous term (e.g., "Best") was interpreted. In some implementations, user interface 1600 can display a follow-up question 1624 (e.g., "Is this what was expected?") that includes a user-selectable feedback button 1626 (e.g., interface element 1626). In some implementations, a user selection of one of buttons 1626 can cause a follow-up question related to the Slack application to be proposed, thereby potentially providing the possibility of undiscovered analytical capabilities.

[0192] C. Voice-Enabled Smart Assistant Chatbot According to some implementations of the present disclosure, VAIL can be used with a voice assistant device running a voice assistant application (e.g., a voice chatbot application). In some implementations, the voice assistant device includes a display screen. In some implementations, the voice assistant device does not include a display screen.

[0193] Figures 17A - 17C provide a series of screenshots for interaction with a voice assistant device including a display screen 1700 according to some implementation forms. In this example, in response to a user selection (e.g., orally or via a touch screen) of the "wine" dataset, the voice assistant device displays the attributes 1702 of the data fields within the dataset. Figure 17A also illustrates that the voice assistant device outputs an oral response 1704 (e.g., via the audio output device 214) (e.g., "Hello, please ask questions about wine data, I will help.").

[0194] Figure 17B illustrates a user interaction with the voice assistant device. In this example, the user provides the voice command "Show me wines by location."

[0195] In some implementation forms, the voice assistant application is similar to the Slackbot application described in Figures 16A - 16G with respect to its implementation architecture. However, in some implementation forms, the voice assistant application uses the Alexa API to implement conversational interactions rather than using a custom parser and NLG module. The voice assistant application uses the output specification to determine the primary response and any follow - up responses and ignores the specific visual encodings included in the output specification.

[0196] Referring back to the example of Figure 17B, in some implementation forms, in response to the voice command, the voice assistant device outputs an oral response 1706 (e.g., "The most expensive wineries are located in the United States.") according to the output specification generated by VAIL. Figure 17B shows a display screen 1700 that displays a visualization 1708 (e.g., a map) including a shaded area corresponding to the country where the most expensive wineries are located.

[0197] FIG. 17C illustrates a user interaction with a voice assistant device. In this example, the user provides a voice command: "Show me the most expensive variety." In some implementations, in response to the voice command, VAIL updates the intent specification and generates an updated output specification according to the updated intent specification. The voice assistant device outputs a spoken response 1710 (e.g., "The most expensive variety is Cabernet Sauvignon."). FIG. 17C also shows a display screen 1700 that displays an updated visualization 1712 (e.g., a bar graph).

[0198] V. FLOWCHART FIGS. 18A - 18E provide a flowchart of method 1800. Method 1800 is also referred to as a process.

[0199] Method 1800 is performed (1802) on an electronic device 102 that is executing a first application (e.g., client application 224).

[0200] The electronic device 102 includes one or more processors 202 and a memory 216 (1804). The memory stores one or more programs for execution by the one or more processors (1806). In some implementations, the operations shown in FIGS. 4 - 17 correspond to instructions stored in the memory 216 or other non - transient computer - readable storage media. The computer - readable storage media can include magnetic or optical disk storage devices, solid - state storage devices such as flash memory, or other one or more non - volatile memory devices. The instructions stored in the computer - readable storage media can include one or more of source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some of the operations in method 1800 can be combined and / or the order of some operations can be changed.

[0201] The electronic device 102 receives (1808) a request directed to a data source (e.g., data source 110). The request includes (1810) one or more intent types (e.g., intent type 229). Each intent type is selected from a predefined set of timing definitions of data analysis operations directed to the data source (1811).

[0202] In some implementations, the one or more intent types include one or more of a focus intent (e.g., focusing on a subset of data points), a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent (1812). This is illustrated in FIG. 6.

[0203] The request also includes one or more predefined attributes associated with the one or more intent types (1814). Each of the one or more predefined attributes restricts the respective data analysis operation of the respective intent type (1815).

[0204] For each of the one or more intent types, the electronic device 102 formulates (1818) a respective intent specification (e.g., intent specification 234) according to the request, including determining one or more respective properties of the respective intent specification according to the respective one or more predefined attributes and according to the data fields from the data source specified in the request.

[0205] For each of the one or more intent types, the electronic device 102 also generates (1820) a respective output specification (e.g., output specification 236). Each output specification includes a respective recommended data visualization type selected from a predefined set of data visualization types of the data source (1822). Each output specification also includes one or more respective data visualization parameters that specify how to render the data visualization of the respective recommended data visualization type (1824). This is illustrated in FIG. 8.

[0206] In some implementations, one or more respective data visualization parameters include identifiers corresponding to data shaping information, encoding information, and / or intent specifications (1826). For example, data shaping includes information that modifies a data source (e.g., sorts or filters) before the data source is used. Encoding information enumerates the type of visualization recommended (e.g., bar graph or scatter plot) and the visual encodings recommended (e.g., x, y, color, and size) for each of the fields to be displayed. This is illustrated in FIG. 8.

[0207] In some implementations, after generating respective output specifications for each of one or more intent types, electronic device 102 visually renders data analysis in response to a request in accordance with (i) at least one output specification of one of the one or more intent types, and (ii) the settings parameters of the first application and / or the electronic device (1828).

[0208] In some cases, the request includes a trend intent (1830). Formulating each intent specification includes identifying from the data source (i) measurement data fields, and (ii) first data that identifies data fields that represent time (1832). This is illustrated in FIG. 6. In some implementations, the first data field can be a time field, a date field, or a date / time field.

[0209] In some cases, the request includes a trend intent (1834). Formulating each intent specification includes identifying from the data source (i) all measurement data fields, and (ii) all data fields that represent time (1836).

[0210] In some cases, the requirements include a focus intent (1838). Formulating each intent specification includes performing one or more of the following operations: (1) a sorting operation that arranges data rows from a data source in a certain order, (2) a filtering operation that filters data rows from a data source into a subset of data rows where a specific data field has a specified data value, and (3) a highlighting operation that highlights a subset of data rows from a data source (e.g., the top row, the top three rows, etc.) (1840).

[0211] In some cases, the requirements include a focus intent. Formulating each intent specification includes performing a "zoom" operation that zooms in on a subset of data rows (e.g., highlights, enlarges, draws the user's attention).

[0212] In some cases, the requirements include an encoding intent (1842) and specify a first data field. Generating each output specification includes adding an encoding parameter that specifies rendering a data visualization according to the first data field (1844), or overriding an existing encoding parameter for the data visualization according to the first data field (1846).

[0213] In some cases, the requirements include a field intent (1848) and specify a first data field. Generating each output specification includes applying one or more inference rules based on semantic constraints imposed by the first data field (1850).

[0214] In some implementations, method 1800 further includes, for each of one or more intent types, applying one or more respective rules corresponding to each intent type to determine whether each intent type is ambiguous and / or under-informed (1854). In accordance with the determination that each intent type is ambiguous or under-informed, electronic device 102 infers information (1856) to resolve the ambiguous and / or under-informed intent according to (i) one or more respective rules (e.g., rules defined by visual analysis within tent system 230 and described in FIGS. 4-13), (ii) metadata of the data source (e.g., metadata 246), and (iii) metadata of the data fields specified in the request. Electronic device 102 updates one or more respective data visualization parameters according to the inferred information (1858).

[0215] In some cases, applying one or more respective rules to determine whether each intent type is ambiguous and / or under-informed further includes establishing that each intent type is ambiguous (1862) and inferring a numerical value of the quantity (1864) according to the determination that the request specifies a measurable adjective and does not specify a quantity (e.g., the request specifies "top" but does not specify a number) (1860).

[0216] In some cases, the metadata of the data source includes metadata of the meaning of the data field, the data field type corresponding to the data field, the statistics of the data field, the derivation of the data field, and / or the default bin size corresponding to the data field (1866).

[0217] In some implementations, the semantics of the data fields of a data source, or data semantics, describe a particular set of data with which the user is interacting. This includes a list of fields, their roles, and summary statistics. In some implementations, the data semantics can be retrieved from a database or curated by a user to enable the visual analytics intent system 230 to better infer the intent.

[0218] In some cases, the data field type includes one or more of a category field type, a time field type, a geographic field type, a currency field type, a quantity-dependent field type, a quantity-independent field type, a latitude field type, and a longitude field type (1868).

[0219] In some implementations, the electronic device 102 infers information to resolve ambiguous and / or information-deficient intents according to the context from previous requests, the data fields identified in previous requests, the attributes identified in previous requests, user-provided interpretations, and / or user-provided definitions (1870).

[0220] Figures 19A - 19G provide a flowchart of a method 1900. The method 1900 is also referred to as a process.

[0221] The method 1900 is performed in an electronic device 102 executing a first application (e.g., the client application 224) (1902).

[0222] The electronic device 102 includes one or more processors 202 and a memory 216 (1904). The memory stores one or more programs configured to be executed by the one or more processors 202 (1906). In some implementations, the operations shown in FIGS. 4-17 correspond to instructions stored in the memory 216 or other non-transitory computer-readable storage media. The computer-readable storage media may include magnetic or optical disk storage devices, solid-state storage devices such as flash memory, or one or more other non-volatile memory devices. The instructions stored in the computer-readable storage media may include one or more of source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some of the operations in method 1800 may be combined and / or the order of some operations may be changed.

[0223] The electronic device 102 receives a request directed to a data source (e.g., data source 110) (1908). The request includes one or more intent types (e.g., intent type 229) (1910). Each intent type is selected from a defined set of data analysis operations directed to the data source (1911).

[0224] In some implementations, the one or more intent types include one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent (1912). This is illustrated in FIG. 6.

[0225] In some implementations, the request includes one or more predefined attributes associated with the one or more intent types (1914). Each of the one or more predefined attributes restricts the respective data analysis operation of the respective intent type (1915).

[0226] In response to the request (1916), for each of one or more intent types, the electronic device 102 formulates each intent specification according to the request, including determining one or more respective properties of each intent specification according to each of one or more predefined attributes and according to data fields from a data source specified in the request (1918).

[0227] The electronic device 102 applies one or more respective rules corresponding to each intent type to determine whether each intent type is ambiguous and / or under - informed (1920).

[0228] In accordance with the determination (1922) that each intent type is ambiguous or under - informed, the electronic device 102 infers information to resolve the ambiguous and / or under - informed intents according to (i) one or more respective rules, (ii) the metadata of the data source, and (iii) the metadata of the data fields specified in the request (1924).

[0229] In some implementations, the metadata of the data source includes metadata of the meaning of the data field, the data field type corresponding to the data field, the statistics of the data field, the derivation of the data field, and / or the default bin size corresponding to the data field (1926).

[0230] In some cases, the data field type includes one or more of a categorical field type, a time field type, a geographic field type, a currency field type, a quantity - dependent field type, a quantity - independent field type, a latitude field type, and a longitude field type (1928).

[0231] The electronic device 102 updates each intent type according to the inferred information (1930).

[0232] In some implementations, method 1900 further includes inferring information (1932) to resolve ambiguous and / or insufficient intent according to context from previous requests, data fields identified in previous requests, attributes identified in previous requests, user-provided interpretations, and / or user-provided definitions. For example, a user can curate content added to a data table of a data source. In some implementations, the user can also include definitions such as "this data field XX can be interpreted as a currency field" as metadata.

[0233] In some implementations, electronic device 102 generates respective output specifications (1934). Each output specification includes a respective recommended data visualization type selected from a pre-defined set of data visualization types of the data source (1936). Each output specification includes (1) the inferred information and (2) one or more respective data visualization parameters that specify how to render the data visualization of each respective recommended data visualization type (1938).

[0234] In some implementations, determining whether each intent type is ambiguous and / or insufficient by applying one or more respective rules includes identifying a set of data properties corresponding to each intent specification of each intent type (1940) and determining whether the request specifies an attribute of each data property of the set of data properties (1942). In some implementations, when the request specifies two or more attributes of a data property within the set of data properties, electronic device 102 determines that each intent type is ambiguous (1944). In some implementations, when the request does not specify an attribute of at least one data property within the set of data properties, electronic device 102 determines that each intent type is insufficient (1946).

[0235] In some implementations, in accordance with the determination that a claim specifies exactly one attribute for each data property of a set of data properties, the electronic device 102 establishes that each intended type is fully specified (1948).

[0236] In some embodiments, applying one or more respective rules to determine whether each intended type is ambiguous and / or under-specified further includes that, in accordance with the determination that a claim specifies a measurable adjective (1950) and does not specify a quantity, the electronic device 102 determines that each intended type is ambiguous (1952). The electronic device 102 infers a numerical value of the quantity (1954).

[0237] In some embodiments, each intended type is a focus intent having a focus strategy property. Inferring information to resolve an under-specified intent further includes selecting a highlighting strategy as the default focus strategy (1958) in accordance with the determination that a claim does not include a first term selected from the group consisting of "filter", "highlight", and "zoom".

[0238] In some cases, selecting a highlighting strategy as the default strategy includes performing a sorting operation that arranges data rows from a data table of a data source in ascending or descending order according to data values of data fields in the data table (1962), and performing a highlighting operation that highlights a specified number of data values based on the ranking of the data values (1964).

[0239] In some implementations, each intent type is a focus intent having field properties (1966). Inferring information to resolve an information deficit intent further includes selecting, from a data source (e.g., based on metadata), a data field having a text data type and a domain larger than the quantity specified in the request, according to a determination that the request does not specify a data field (1968). In some implementations, the domain can be the same as the quantity specified in the request. In some implementations, if the quantity specified in the request is larger than the domain, the electronic device 102 can modify the quantity in the request to be the size of the domain.

[0240] In some implementations, each intent type is a focus intent having field properties (1970). Inferring information to resolve an information deficit intent further includes selecting, from a data source, a data field whose domain includes the data value specified in the request, according to a determination that the data source does not include a data field having a text data type and a domain larger than the specified quantity (1972).

[0241] In some implementations, each intent type is a focus intent (1974). Method 1900 further includes determining (e.g., at least partially based on metadata) whether the data source includes a data field having a currency field type according to a determination that the request includes an adjective of price (e.g., the adjective includes "cheap", "expensive", "most expensive", "cheapest", "extravagant", etc.) (1976). In some implementations, according to a determination that the data source includes a data field having a currency field type, electronic device 102 performs a sorting operation on the data field (1978). In some implementations, according to a determination (1980) that the data source does not include a data field having a currency field type, electronic device 102 identifies all numerical data fields (e.g., measurement fields) within the data source (1982). Electronic device 102 performs a respective sorting operation on each of the numerical data fields (1984).

[0242] In some implementations, each intent type is a delivery intent having a grouping property including a grouping field and a bin size (1986). Inferring information to resolve an information-lacking intent includes, according to a determination (1988) that the request does not specify a grouping field, (1) identifying all data fields within a data source having a quantity-dependent field type, a quantity-independent field type, or a currency field type (1990), and (2) designating each of the identified data fields as inferred information of the grouping field (1992).

[0243] In some cases, according to a determination that the request specifies a bin size, electronic device 102 replaces the bin size information in the metadata with the specified bin size (1994).

[0244] In some implementations, the intent type is a trend intent having trend properties that include a measurement data field and a date data field (1996). Inferring information to resolve an information deficit intent further includes, in accordance with a determination that the claim does not include a measurement data field (1998), (1) identifying all quantity-dependent fields in the data source (19100), and (2) designating each of the quantity-dependent fields as inferred information for the measurement data field (19102). In some implementations, inferring information to resolve an information deficit intent further includes, in accordance with a determination that the claim does not include a data field having a time field type (19104), (1) identifying all time data fields in the data source (19106), and (2) designating each of the time data fields as inferred information for the date data field (19108).

[0245] In some implementations, the intent type is a correlation intent for correlating two numerical data fields (19110). Inferring information to resolve an information deficit intent further includes (1) inferring the use of a second data field from all numerical data fields in the data source when the claim specifies only one numerical data field (19112), and (2) inferring the use of a first and a second data field from all numerical data fields in the data source when the claim does not specify a numerical data field (19114).

[0246] Figures 20A - 20F provide a flowchart of method 2000. Method 2000 is also referred to as a process.

[0247] Method 2000 is performed at an electronic device 102 executing a first application (e.g., client application 224) (2002).

[0248] The electronic device 102 includes one or more processors 202 and a memory 216 (2004). The memory stores one or more programs configured to be executed by one or more processors 202 (2006). In some implementations, the operations shown in FIGS. 4-17 correspond to instructions stored in the memory 216 or other non-transitory computer-readable storage media. The computer-readable storage media may include magnetic or optical disk storage devices, solid-state storage devices such as flash memory, or other one or more non-volatile memory devices. The instructions stored in the computer-readable storage media may include one or more of source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some of the operations in method 1800 may be combined and / or the order of some of the operations may be changed.

[0249] The electronic device downloads a visual analysis intent library (e.g., visual analysis intent library 323) configured to execute across a plurality of applications including a first application from a server system (e.g., server system 300) communicatively coupled to the electronic device (2008).

[0250] The electronic device 102 sets the visual analysis intent library to execute on the electronic device based on the electronic device executing the first application (2010).

[0251] After setting, the electronic device 102 receives a request directed to a data source (2012). The request includes one or more intent types (2014). Each of the intent types is selected from a predefined set of data analysis operations directed to the data source (2015).

[0252] In some implementations, the one or more intent types include one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent (2016).

[0253] The requirements also include one or more pre - defined attributes associated with one or more intent types (2018). Each of the one or more pre - defined attributes restricts the respective data analysis operation of each intent type (2019).

[0254] In accordance with the requirements, for each of the one or more intent types, the electronic device 102 formulates each intent specification according to a visual analysis intent library, including determining one or more respective properties of each intent specification according to each of the one or more pre - defined attributes and according to data fields from a data source specified in the requirements (2020).

[0255] In some implementations, the electronic device has a first device type. In some implementations, the visual analysis intent library is configured to execute across a plurality of device types including the first device type.

[0256] In some implementations, for each of the intent types, after formulating each intent specification, the electronic device 102 generates each output specification according to a visual analysis intent library (2022). Each output specification includes a respective recommended data visualization type selected from a pre - defined set of data visualization types of the data source (2024). Each output specification also includes one or more respective data visualization parameters that specify how to render the data visualization of each respective recommended data visualization type (2026).

[0257] In some implementations, one or more of the respective data visualization parameters include data shaping information, encoding information, and / or an identifier corresponding to the intent specification (2028).

[0258] In some implementations, after generating respective output specifications for each of one or more intent types, the electronic device 102 visually renders data analysis in response to a request according to (i) at least one output specification of one of the one or more intent types, and (ii) the first application and / or the setting parameters of the electronic device (2030).

[0259] In some implementations, visually rendering data analysis in response to a request further complies with the electronic device 102 having a first device type (2032). For example, in some implementations, visually rendering data analysis in response to a request further complies with an electronic device having a display screen, or the size of the display screen), or peripheral devices such as audio input and output devices.

[0260] In some cases, at least one output specification of one of the one or more intent types corresponds to a first intent type and includes a first output specification having inferred information (2034). Visually rendering data analysis further includes displaying the inferred information in the first application (2036).

[0261] In some implementations, the electronic device 102 displays the inferred information in a manner that is visually distinct from the information included in the request. In some implementations, displaying the inferred information includes displaying an identification of the inferred data field. In some implementations, displaying the inferred information can include providing an explanation of how terms within the terms in the request were interpreted.

[0262] In some cases, the electronic device 102 displays (2038) a data widget (e.g., data widget 1618 in FIG. 16E) that enables the inferred information to be corrected. The electronic device 102 receives (2040) a user interaction with the data widget. In response to the user interaction (2042), the electronic device 102 edits (2044) the first intended specification. The electronic device 102 updates (2046) the first output specification according to the edited first intended specification. The electronic device 102 generates (2048) updated data analysis according to the updated first output specification and the first application and / or the setting parameters of the electronic device, and visually renders it.

[0263] In some implementations, the electronic device 102 is a voice assistant device (e.g., the voice assistant device 102-3 in FIG. 1 that does not include a display screen, or the voice assistant device in FIG. 17 that includes a display screen 1700) (2050). The method 2000 further includes, after generating each output specification, converting each output specification into each spoken response (2054), and outputting each spoken response (2056).

[0264] For example, FIG. 17C shows that in response to a user request of "Show me the most expensive variety.", the electronic device converts the output specification into each spoken response and outputs a spoken response 1710 of "The most expensive variety is Cabernet Sauvignon.".

[0265] In some implementations, the spoken response can include a response that verbally describes the visualization. In some implementations, the spoken response can include a response that verbally describes how the ambiguous terms in the request were inferred.

[0266] In some cases, each output specification includes the inferred information (2058). Outputting each spoken response includes outputting a spoken explanation of the inferred information (e.g., explanation, clarification, etc.) (2060).

[0267] In some implementations, after outputting a response, the voice assistant device can explicitly ask the user a follow-up question to clarify or confirm that the inferred information matches the user's request.

[0268] In some implementations, the first application includes a graphical user interface (2062). Method 2000 further includes receiving a request via the graphical user interface, including identifying one or more intent types (2064).

[0269] For example, as shown in FIGS. 14A - 14J, the first application 224 includes a graphical user interface 1400. The method further includes receiving a request via the graphical user interface 1400, including identifying one or more intent types 1408.

[0270] In some cases, the electronic device 102 generates respective data visualizations according to each output specification (2066). The electronic device 102 displays the data visualizations on a graphical user interface (2068). This is illustrated in FIGS. 14D, 14E, 14F, and 14H.

[0271] In some implementations, the first application is a messaging application (2070). Method 2000 further includes, after generating each output specification (2072), (1) converting the visual specification to text output (2074), and (2) displaying the text output in the messaging application (2076).

[0272] For example, in some implementations, the first application is a messaging application such as the chat application 106 of FIG. 1 or the Slackbot application of FIGS. 15 and 16. The method further includes converting the visual specification into a text output such as the text output 1612 of FIG. 16E or the text output 1622 of FIG. 16G after generating each output specification. The method also includes displaying the text output on the user interface 1600 of the messaging application.

[0273] In some implementations, before receiving the request, the electronic device 102 receives a natural language command (2078). The electronic device 102 analyzes the natural language command (e.g., using parser 226 or parser 1506) according to a visual intent specification language for forming the request (2080).

[0274] In some cases, the electronic device 102 determines, according to the analysis, that the natural language command having one or more intent types with the term "time" or the term "trend" includes a trend intent type (2082).

[0275] In some cases, the electronic device 102 determines, according to the analysis, that the natural language command having one or more intent types with a measurable adjective (e.g., adjectives such as "highest", "lowest", "high", "low", or "most expensive") includes a focus intent type (2084). In some implementations, the measurable adjective refers to an adjective having an attribute that can be numerically quantified, ranked, or arranged in a specific order.

[0276] In some implementations, the method 2000 further includes updating the first intent specification formulated from a previous request by correlating the first intent specification with the intent specification formulated in the request (2086).

[0277] In some cases, a previous claim includes (2088) a first data field from a data source. The claim specifies (2090) replacing the first data field with a second data field from the data source. Updating the first intended specification includes (2092) replacing the first data field with the second data field.

[0278] Each of the executable modules, applications, or sets of procedures identified above can be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The identified modules or programs (i.e., sets of instructions) above need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of these modules can be combined in various implementations or otherwise rearranged. In some implementations, the memory 216 stores a subset of the modules and data structures identified above. Further, the memory 216 can store additional modules or data structures not described above.

[0279] The terms used in the description of the present invention herein are for the purpose of describing particular implementations only and are not intended to be limiting of the present invention. As used in the description of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" refers to any and all possible combinations of one or more of the associated listed items and is understood to encompass them. The terms "comprises" and / or "comprising", when used herein, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0280] The foregoing description has been presented for purposes of illustration and description in connection with specific implementations. However, the above exemplary considerations are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The implementations were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and various implementations with various modifications suited to the particular use contemplated.

Claims

Claim 1 A method comprising: In an electronic device that is executing a first application, the electronic device including one or more processors and a memory storing one or more programs for execution by the one or more processors, Receiving a request directed to a data source, the request including (i) one or more intent types selected from a predefined set of data analysis operations directed to the data source, and (ii) one or more predefined attributes associated with the one or more intent types, each of the one or more predefined attributes restricting a respective data analysis operation of a respective intent type, For each of the one or more intent types, Formulating each respective intent specification according to the request, including determining one or more properties of each respective intent specification according to (a) each respective intent type to which each respective intent specification corresponds, (b) each of the one or more predefined attributes, and (c) data fields from the data source specified in the request, Generating each respective output specification, the each respective output specification including (i) a respective recommended data visualization type selected from a predefined set of data visualization types of the data source, and (ii) one or more respective data visualization parameters specifying how to render a data visualization of the respective recommended data visualization type, Converting each respective output specification to a format according to the electronic device having a particular device type, A method as described above. Claim 2 The method of claim 1, wherein the one or more intent types include one or more of a focus intent, a trend intent, a correlation intent, a distribution intent, a geographic intent, a coding intent, and a field intent. Claim 3 The request includes the trend intent, Formulating each respective intent specification includes identifying from the data source (i) a measurement data field, and (ii) a first data field representing time, The method of claim 2, as described above. Claim 4 The request includes the trend intent, Formulating each of the respective intended specifications involves identifying, from the data source, (i) all measurement data fields, and (ii) all data fields representing time. The method according to claim 2, comprising.

5. The request includes the focus intention. Formulating each of the respective intended specifications involves. A sorting operation that arranges data rows from the data source in a certain order, A filtering operation that filters the data rows from the data source into a subset of data rows where a specific data field has a specific data value, Performing one or more of a highlighting operation that highlights a subset of data rows from the data source. The method according to claim 2, comprising.

6. The request includes the encoding intention, specifies a first data field, Generating each of the respective output specifications involves. Adding an encoding parameter that specifies rendering the data visualization according to the first data field, or Overriding an existing encoding parameter of the data visualization according to the first data field. The method according to claim 2, comprising.

7. The request includes the field intention, specifies a first data field, Generating each output specification involves applying one or more inference rules based on semantic constraints imposed by the field type of the first data field. The method according to claim 2, comprising.

8. For each of the one or more intention types, Applying one or more respective rules corresponding to each of the respective intention types to determine whether each of the respective intention types is ambiguous and / or lacks information, In accordance with the determination that each of the respective intention types is ambiguous or lacks information, inferring information to resolve the ambiguous and / or information - lacking intention according to (i) the one or more respective rules, (ii) the metadata of the data source, and (iii) the metadata of the data fields specified in the request, Updating the one or more respective data visualization parameters according to the inferred information. The method according to claim 1, further comprising.

9. Applying each of the one or more rules to determine whether each of the intent types is ambiguous and / or lacks sufficient information, According to the determination that the claim specifies a measurable adjective and does not specify a quantity, Establishing that each of the intent types is ambiguous, Inferring the numerical value of the quantity, further comprising the method according to claim 8.

10. The metadata of the data source includes metadata of the meaning of the data field, the data field type corresponding to the data field, the statistics of the data field, the derivation of the data field, and / or the default bin size corresponding to the data field, the method according to claim 8.

11. The data field type includes one or more of a category field type, a time field type, a geographic field type, a currency field type, a quantity-dependent field type, a quantity-independent field type, a latitude field type, and a longitude field type, the method according to claim 10.

12. Further comprising inferring information to resolve the ambiguous and / or insufficient information intent according to the context from previous claims, the data fields identified in the previous claims, the attributes identified in the previous claims, user-provided interpretations, and / or user-provided definitions, the method according to claim 8.

13. Each of the one or more data visualization parameters includes data shaping information, encoding information, and / or an identifier corresponding to each intent specification, the method according to claim 1.

14. After generating each of the respective output specifications for each of the one or more intent types, (i) at least one output specification of one of the one or more intent types, and (ii) visually rendering data analysis in response to the request according to the setting parameters of the first application and / or the electronic device, Further comprising the method according to claim 1.

15. An electronic device, One or more processors, A memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs being, Receiving a request directed to a data source, the request comprising: (i) one or more intent types selected from a predefined set of data analysis operations directed to the data source; and (ii) one or more predefined attributes associated with the one or more intent types, each of the one or more predefined attributes restricting a respective data analysis operation of a respective intent type. For each of the one or more intent types, formulating each intent specification according to the request, including determining one or more properties of each intent specification: (a) according to a respective intent type to which a respective intent specification corresponds; (b) according to each of the one or more predefined attributes; and (c) according to data fields from the data source specified in the request. generating each output specification, the each output specification including: (i) a respective recommended data visualization type selected from a predefined set of data visualization types of the data source; and (ii) one or more respective data visualization parameters specifying how to render a data visualization of the respective recommended data visualization type. converting each output specification to a format according to an electronic device having a specific device type. An electronic device comprising instructions for performing the above. Claim 16 The electronic device according to claim 15, wherein the one or more programs further comprise instructions for performing the method according to any one of claims 2 to 14. Claim 17 An electronic device, when executed by the electronic device, causes the electronic device to receive a request directed to a data source, the request comprising: (i) one or more intent types selected from a predefined set of data analysis operations directed to the data source; and (ii) one or more predefined attributes associated with the one or more intent types, each of the one or more predefined attributes restricting a respective data analysis operation of a respective intent type. For each of the one or more intent types, Formulating each intent specification according to the request, including determining one or more respective properties of each intent specification (a) according to each respective intent type to which each intent specification corresponds, (b) according to each of the one or more predefined attributes, and (c) according to data fields from the data source specified in the request. Generating each output specification, where each output specification includes (i) each recommended data visualization type selected from a predefined set of data visualization types of the data source, and (ii) one or more respective data visualization parameters specifying how to render the data visualization of each recommended data visualization type. Converting each output specification into a format according to the electronic device having a specific device type. A program including instructions for causing an operation including the above. The program according to claim 17, which causes the electronic device to perform the method according to any one of claims 2 to 14 when executed by the electronic device.

Images