Natural language for customizing data visualizations

Abstract

A system, method, and device for generating data visualizations are disclosed. The method includes the steps of (i) obtaining a natural language query, (ii) determining the intent of the natural language query, (iii) generating one or more data requests to one or more selected data sources, the one or more data requests being at least partially based on the intent, (iv) abstracting the resulting data to obtain a data abstraction, the resulting data being in response to one or more data requests, and (v) generating visualizations of the resulting data, at least partially based on the data abstraction.

Description

Background Art

[0001] In today's information-driven society, organizations and individuals rely on data analysis to uncover meaningful patterns, identify trends, and extract actionable insights. Traditional data analysis methods often involve complex spreadsheets, long reports, and static charts, which can be time-consuming and difficult to interpret, especially for non-technical users. There is a growing need for more efficient and user-friendly techniques for visualizing data in a way that is understandable and visually appealing.

[0002] Existing data visualization tools often require manual configuration and lack the ability to dynamically adapt to different data sets and user queries. Furthermore, the process of generating visualizations typically requires a deep understanding of data structures and programming skills, limiting accessibility to only specialized user groups. Therefore, there is a need for an automated system that can seamlessly generate insightful and interactive visualizations from diverse data sources to meet the needs of users with varying levels of technical expertise.

Brief Description of the Drawings

[0003] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings. [Figure 1] A block diagram of a system for generating data visualizations according to various embodiments. [Figure 2] A block diagram of a system for generating data visualizations according to various embodiments. [Figure 3] A block diagram of a system for generating data visualizations according to various embodiments. [Figure 4] A block diagram of a system for generating data visualizations according to various embodiments. [Figure 5] This is an example of determining visualization definitions based on natural language user input using various embodiments. [Figure 6A] This is an example of a set of data retrieved in relation to natural language queries using various embodiments. [Figure 6B] This is an example of data abstraction performed on a set of data for a query, using various embodiments. [Figure 7A] This is an example of a set of data retrieved in relation to natural language queries using various embodiments. [Figure 7B] This is an example of data abstraction performed on a set of data for a query, using various embodiments. [Figure 8] This document describes the process for generating visualizations using various embodiments. [Figure 9] Examples of visualization definitions and corresponding visualizations are shown using various embodiments. [Figure 10] This document describes the process for generating visualizations using various embodiments. [Figure 11] This document describes the process for generating visualizations using various embodiments. [Figure 12] This document describes a process for determining the visualization type using various embodiments. [Figure 13] This document describes the process for generating visualizations using various embodiments. [Figure 14] This is a flowchart illustrating methods for generating visualizations based on natural language queries using various embodiments. [Figure 15] This is a flowchart illustrating methods for generating visualizations based on natural language queries using various embodiments. [Figure 16] This is a flowchart illustrating methods for determining the query intent of a natural language query using various embodiments. [Figure 17] This is a flowchart illustrating various embodiments of methods for obtaining data that responds to queries based on query intents. [Figure 18] This is a flowchart illustrating various embodiments of methods for obtaining data that responds to queries based on query intents. [Figure 19] This is a flowchart of a method for determining visualization definitions for data abstraction using various embodiments. [Figure 20] This is a flowchart of a method for determining visualization definitions for data abstraction using various embodiments. [Figure 21] This is a flowchart of a method for determining the visualization type for data abstraction using various embodiments. [Figure 22] This is a flowchart of a method for determining visualization definitions for data abstraction using various embodiments. [Figure 23] This is a flowchart illustrating various methods for translating visualization definitions into another language. [Figure 24] This is a flowchart of methods for training a model using various embodiments. [Modes for carrying out the invention]

[0004] The present invention can be implemented in numerous ways, including processes, apparatus, systems, compositions, computer program products embodied on computer-readable storage media, and / or processors, such as processors, which are stored on and / or provided by memory coupled to a processor. These implementations, or any other forms the invention may take, may be referred to as techniques. Generally, the order of the steps of the disclosed process may be modified within the scope of the invention. Unless otherwise specified, components such as processors or memory described as configured to perform a task may be implemented as general components temporarily configured to perform a task at a given time, or as specific components manufactured to perform a task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0005] A detailed description of one or more embodiments of the present invention, along with accompanying diagrams illustrating the principles of the present invention, is provided below. While the present invention is described in relation to such embodiments, it is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous alternative forms, modifications, and equivalents. In order to provide a complete understanding of the present invention, numerous specific details are described in the following description. These details are provided for illustrative purposes, and the present invention may be carried out in accordance with the claims without some or all of these specific details. For clarity, technical materials known in the art relating to the present invention are not described in detail so as not to unnecessarily obscure the present invention.

[0006] The systems and methods of related technologies for generating data visualizations in response to queries generally involve predefined visualizations or dashboards that require a considerable degree of customization by the developer. A prominent method for providing visibility and insight into data is through custom dashboards that provide a visual representation of data collected from various sources. Conventional custom dashboards generally provide a limited set of pre-built widgets, either provided by a specific application or built using an application / service (e.g., Grafana). In most cases, such custom dashboards require explicit engineering effort and lack flexibility. Predefined widgets provide users with limited visualizations. Furthermore, if the user requires new filters or different data representations, such requirements generally require engineering effort.

[0007] Systems and processes of related technologies for generating data visualizations generally require the user to input queries that are highly specific or follow a particular syntax. In contrast, various embodiments allow the user to input queries as natural language queries, and the system automatically infers the intent of the query, determines the type of visualization to be used to represent the resulting data, and generates the corresponding visualization.

[0008] Various embodiments address the inflexibility and engineering resource requirements inherent in solutions of related technologies. According to various embodiments, a system or process for generating visualizations enables users to build "no-code" natural language-driven customized data visualizations / data visualization dashboards. The user inputs a natural language query, and in response to receiving the natural language, the system uses a machine learning model to infer the intent of the query (e.g., query parameters, task, design, etc.) and automatically generates a visualization based on the query intent. In relation to automatically generating visualizations, the system represents or describes the data visualization in a computable way. For example, the system / process for generating data visualizations represents the query in a high-level language (e.g., the query is abstracted to a high level and represented in a given data visualization language). A high-level language (e.g., a given data visualization language) can function as an intermediate language for expressing visualization queries or definitions for use across multiple libraries, applications, or services for generating visualizations. A visualization definition in a given data visualization language can be translated into another high-level language used by a specific library, application, or service, or otherwise used to generate a specific type of visualization.

[0009] The various embodiments disclosed herein provide systems, methods, and devices for generating data visualizations. The method includes: (i) obtaining a natural language query; (ii) determining the intent of the natural language query; (iii) generating one or more data requests to one or more selected data sources, the one or more data requests being generated at least in part based on the intent; (iv) abstracting result data to obtain a data abstraction, the result data responding to the one or more data requests; and (v) generating a visualization for the result data based at least in part on the data abstraction.

[0010] The various embodiments disclosed herein provide systems, methods, and devices for generating data visualizations. The method includes: (i) obtaining a natural language query; (ii) determining the intent of the natural language query; (iii) generating one or more data requests to one or more selected data sources, the one or more data requests being generated at least in part based on the intent; (iv) obtaining a predicted visualization definition based at least in part on abstracting the result data; and (v) generating a visualization for the result data based at least in part on the predicted visualization definition. The result data can respond to the one or more data requests. In some embodiments, obtaining the predicted visualization definition includes querying a prediction engine for the predicted visualization definition based at least in part on a data abstraction for the result data.

[0011] In some embodiments, the intent is determined at least in part based on a machine learning model. For example, determining the intent of the natural language query includes querying a large language model based on the natural language query.

[0012] In some embodiments, the prediction engine is a machine learning model. The machine learning model can be trained based on a training dataset that includes a set of data abstractions and corresponding data visualization language expressions for the data abstractions. As an example, the prediction engine includes a machine learning model, and the predicted visualization definition is obtained based on querying the machine learning model at least partially based on the data abstraction. The system can update the machine learning model at least partially based on user feedback received in response to the visualization being provided to the user.

[0013] In some embodiments, the intent and the predicted visualization definition are obtained based on different machine learning models.

[0014] In some embodiments, the predicted visualization definition is obtained at least partially based on querying a large language model. The large language model is trained based on a training set that includes (i) a set of natural language queries or data abstractions, and (ii) a corresponding set of visualization definitions in a given data visualization language.

[0015] In some embodiments, one or more selected data sources are determined at least partially based on the intent. The system can obtain result data from the one or more selected data sources.

[0016] In some embodiments, abstracting the resulting data in connection with obtaining a predicted visualization definition includes determining one or more statistical properties of the resulting data. These one or more statistical properties may include one or more of columns, data within columns, outlier data, and numerical distributions. Determining one or more statistical properties of the resulting data may include analyzing the resulting data, which includes applying one or more predetermined rules to obtain one or more statistical properties. The predicted data abstraction may be determined at least in part on one or more statistical properties.

[0017] In some embodiments, generating visualizations for resulting data based at least partially on a predicted visualization definition includes determining the type of visualization based at least partially on a predicted visualization definition and creating the visualization based at least partially on the type of visualization.

[0018] In some embodiments, the predicted visualization definition corresponds to a data visualization language representation of a natural language query according to a given data visualization language. The given data visualization language may include a first dimension of the data to be visualized, a second dimension of the data to be visualized, and an indication of the type of visualization. Generating a visualization for the resulting data may include (i) translating the predicted visualization definition into another high-level programming language to obtain the translated representation, and (ii) generating the visualization based on the translated representation. Exemplary examples of other high-level programming languages ​​may include Python or Go.

[0019] In some embodiments, the predicted visualization definition indicates the type of visualization to be generated.

[0020] In some embodiments, abstracting the resulting data to obtain a data abstraction includes (a) determining all possible ordered subsets of the data for the resulting data; (b) analyzing each subset of the data with a specific rule for each rule in a given set of rules to obtain a corresponding rule check; (c) creating a score vector for each rule in a given set of rules that includes the corresponding rule check for the subset of data; (d) ranking the rules in a given set of rules based on the corresponding score vector; and (e) selecting a data abstraction based on the ranking of the rules. Each rule in a given set of rules may correspond to a specific type of visualization. Creating a score vector may include scoring the fit of the type of visualization for a specific rule to a subset of data, the fit being scored according to a predetermined scoring criterion. For example, the scoring criterion may include scores for one or more of the following: compatibility, coverage, fanciness, and user preference. In some embodiments, user preferences are determined at least in part on user feedback provided in response to one or more previous visualizations.

[0021] Figure 1 is a block diagram of a system for generating data visualizations according to various embodiments. In some embodiments, system 100 implements at least a portion of process 1000 in Figure 10, process 1100 in Figure 11, process 1200 in Figure 12, process 1300 in Figure 13, process 1400 in Figure 14, process 1500 in Figure 15, process 1600 in Figure 16, process 1700 in Figure 17, process 1800 in Figure 18, process 1900 in Figure 19, process 2000 in Figure 20, process 2100 in Figure 21, process 2200 in Figure 22, process 2300 in Figure 23, and / or process 2400 in Figure 24.

[0022] In the illustrated example, system 100 allows a user to input a natural language query, and a service (e.g., a cloud service) to generate a data visualization based on the natural language query. The service interprets the natural language query, determines the query's intent (e.g., the data to be used, the task to be performed, the design configuration), and generates the data visualization.

[0023] As shown in the figure, system 100 comprises a client system 110, a DataService API server 120, a router layer 130, a cognitive service 140, an LLM service 150, and a data query service 160.

[0024] The client system 110 is configured to present a user interface to the user. The user interface is configured to allow the user to input natural language queries, for example, in the form of a chat with a chatbot. In 181, the user inputs a request into the user interface, and the request is communicated to the DataService API server 120. In response to the DataService API server 120 receiving a request containing a natural language query, in 182, the DataService API server 120 stores the request in association with the user, and so the system 100 can track the conversation / interaction with the user. The request may be stored in a table along with the user interaction / communication. In 183, the DataService API server 120 posts the request to the router layer 130, which is configured to determine which service or system will process the request. In 184, the router layer 130 communicates / posts a message indicating that a new request has been received. In response to receiving the message indicating that a new request has been received and will be processed, the cognitive service 140 interprets the natural language query associated with the request, retrieves the associated data, and generates a visualization.

[0025] In step 185, the cognitive service 140 sends a request to the LLM service 150 to interpret a natural language query. The LLM service 150 may be a cloud service that provides machine learning models, such as a Large-Scale Language Model (LLM) as a service. Examples of LLMs provided as a service include ChatGPT or GPT-3 provided by Azure OpenAI, or Bard provided by Google. Other models may be implemented. The cognitive service 140 may send a request to the LLM service 150 by posting the user request to the LLM service 150 (or a specific LLM provided by the LLM service 150). In response to receiving a request to interpret a natural language query, the LLM service 150 queries a specific machine learning model to interpret the natural language query. For example, the LLM service 150 infers the intent and context of the natural language query (e.g., the LLM service 150 infers what is being asked in the natural language query). Inferring intent and context may include determining query intents, task intents, and / or design intents. A query intent may include instructions for data that are invoked in connection with obtaining / determining result data, and logic to be executed with respect to that data. The LLM service 150 provides the cognitive service 140 with an interpretation of the natural language query (e.g., query intents, task intents, and / or design intents). In response to receiving the interpretation of the natural language query, the cognitive service 140 determines one or more data sources that contain data corresponding to the result data, or data used to generate the result data of the natural language query. The cognitive service 140 may determine one or more data sources based on the query intents obtained from the LLM service 150, including query intents that identify the type of data to be analyzed.In response to determining one or more data sources to be queried, the cognitive service 140 generates one or more corresponding requests (e.g., subqueries of a natural language query) to retrieve data (e.g., each subset of data) from one or more data sources. In 186, the cognitive service 140 communicates one or more requests to the data query service 160 and receives data in response to one or more requests. The data query service 160 is configured to receive data requests, manage queries for the identified data sources, retrieve data from the data sources, and provide responsive data to the service that requested the data (e.g., the cognitive service 140).

[0026] The cognitive service 140 determines how to present data in response to a natural language query, based at least partially on the natural language query and the resulting data for the query. For example, the cognitive service 140 determines the type of visualization to generate, based at least partially on one or more of the following: the query intent, the task intent, the design intent, and the resulting data. The cognitive service 140 may determine the visualization to be generated, or the type of visualization to be generated, based on calling process 1000 in Figure 10, process 1100 in Figure 11, and / or process 1300 in Figure 13. In response to determining the visualization type, the cognitive service 140 renders the visualization. The cognitive service 140 may render the visualization directly or call another application or service to generate the visualization. In response to the rendering of the visualization, in 187, the cognitive service 140 stores the results (e.g., visualizations and / or UI recommendations) associated with the user in a table used, for example, to track the user's conversation / interaction. In 188, the DataService API server 120 retrieves the results for a natural language query, such as the generated visualization and user interface (or configuration for the user interface to be presented to the user along with the visualization). In 189, the DataService API server 120 renders the results (e.g., the generated visualization) and provides the results to the client system 110.

[0027] Figure 2 is a block diagram of a system for generating data visualizations according to various embodiments. In some embodiments, system 200 is implemented in relation to at least a portion of system 100 in Figure 1. In some embodiments, system 200 implements at least a portion of process 1000 in Figure 10, process 1400 in Figure 14, process 1500 in Figure 15, process 1600 in Figure 16, process 1700 in Figure 17, process 1800 in Figure 18, process 1900 in Figure 19, process 2300 in Figure 23, and / or process 2400 in Figure 24. System 200 may be implemented by a cloud service (e.g., a set of one or more cloud resources that provide services to a user, such as providing visualizations to a user).

[0028] In some embodiments, system 200 implements the cognitive services 140 of system 100.

[0029] In the illustrated example, system 200 implements one or more modules related to interpreting natural language queries and providing visualizations based on natural language queries. System 200 comprises a communication interface 205, one or more processors 210, storage 215, and / or memory 220. One or more processors 210 comprises one or more of the following: a communication module 225, a query receiving module 227, a query interpretation engine module 229, a data retrieval module 231, a data abstraction module 233, a rule lookup module 235, a visualization generation module 237, a user feedback module 239, and / or a user interface module 241.

[0030] In some embodiments, the system 200 includes a communication module 225. The system 200 uses the communication module 225 to communicate with various nodes or endpoints (e.g., client terminals, firewalls, DNS resolvers, data appliances, other security entities, etc.) or user systems such as an administrator system. For example, the communication module 225 provides information to be communicated (e.g., to another node, security entity, etc.) to the communication interface 205. In another example, the communication interface 205 provides information received by the system 200 to the communication module 225. The communication module 225 is configured to receive natural language queries, selections of visualization types, user feedback for visualizations, etc. The communication module 225 is configured to query third-party services or data sources for data that is expected to respond to queries, such as subqueries of natural language queries. The communication module 225 is further configured to receive one or more settings or configurations from an administrator. Examples of one or more settings or configurations include the configuration of a machine learning model (e.g., a model used to interpret natural language queries), the configuration of a visualization, etc.

[0031] In some embodiments, the system 200 includes a query receiving module 227. The system 200 uses the query receiving module 227 to receive user input. User input includes natural language queries, such as queries entered by a user interface, such as a user interface configured by a user interface module 241.

[0032] In some embodiments, the system 200 includes a query interpretation engine module 229. The system 200 uses the query interpretation engine module 229 to interpret queries, such as natural language queries, received by the query receiving module 227. The query interpretation engine module 229 is configured to infer the intent or context of the natural language query. For example, the query interpretation engine module 229 determines the query intent (e.g., instructions for the data to be retrieved and / or the logic to be executed to provide the result data of the query), the task intent, and / or the design intent.

[0033] The query interpretation engine module 229 queries a machine learning model based on the natural language query. In some embodiments, the machine learning model is a large-scale language model (LLM), such as a common LLM model like ChatGPT, GPT-3, or Bard. The LLM analyzes the natural language query and determines the intent and context.

[0034] In some embodiments, the query interpretation engine module 229 determines the tasks of a natural language query. For example, the query interpretation engine module 229 further abstracts the tasks(s) of the corresponding natural language query. For example, the query interpretation engine module 229 determines a set of corresponding domain-specific tasks. Examples of domain-specific tasks associated with a natural language query may include (i) whether more incidents occurred yesterday, (ii) on which day this week had the most incidents, and (iii) how the number of incidents changed over the past week. The system may abstract the set of domain-specific tasks into an abstracted set of tasks: (i) comparing values, (ii) identifying outliers, and (iii) discovering distributions.

[0035] In some embodiments, the query interpretation engine module 229 determines the design criteria for the visualizations generated for natural language queries. The query interpretation engine module 229 may further abstract the design intent, which allows the user to define / modify the design aspects of the desired visualization. Examples of design abstractions for natural language input include size, aspect ratio, labels and / or legends, colors (e.g., color scheme), orientation, scale, accessibility, interaction, and layout structure. Various other design abstractions may be implemented.

[0036] In some embodiments, the system 200 includes a data retrieval module 231. The system 200 uses the data retrieval module 231 to retrieve data to be used in connection with generating result data for a natural language query. The data retrieval module 231 is configured to determine the data source(s) from which data is retrieved in order to determine the result data. For example, the data retrieval module 231 determines the data source(s) based on a query intent. For example, the system 200 may have (or be connected to) multiple data sources, and based on the query intent, the data retrieval module 231 determines one or more subsets of the multiple data sources to query for data (e.g., result data and / or data from which the result data is derived). In response to determining the data source(s), the data retrieval module 231 generates one or more requests to be sent to the data source(s) to retrieve the corresponding data. The data retrieval module 231 sends one or more requests to the data source(s) and receives the requested data accordingly.

[0037] In some embodiments, the system 200 includes a data abstraction module 233. The system 200 uses the data abstraction module 233 to abstract data extracted from one or more data sources. For example, the data abstraction module 233 abstracts result data associated with a natural language query. In some embodiments, abstracting data includes determining one or more properties or statistics associated with the query. The data abstraction may be configured according to a predetermined abstraction format.

[0038] Data abstraction can correspond to characteristic-based data descriptions, such as high-level descriptions of data based on data statistics. Examples of statistics for response data include mean, median, standard deviation, range, maximum, minimum, and number of data elements / records. Various other statistics may be determined and used in connection with the generation of data abstractions. In some embodiments, statistics are selected based on the type of data contained in the response data, the amount of data / records in the response data, etc.

[0039] In some embodiments, the system 200 includes a rulebook lookup module 235. The system 200 uses the rulebook lookup module 235 to perform a lookup of a given set of rules against a rulebook. The rulebook lookup module 235 performs the lookup to determine which rule in the given set of rules matches or most closely matches a data abstraction.

[0040] The rulebook (for example, stored in rulebook data 270) contains rules for bar charts (for example, rules corresponding to horizontal bar chart visualization types), line charts (for example, rules corresponding to line chart visualization types), donut charts (for example, rules corresponding to donut chart visualization types), map charts (for example, rules corresponding to map visualization types), sunburst charts (for example, rules corresponding to sunburst visualization types), parallel set charts (for example, rules corresponding to parallel set visualization types), pie charts (for example, rules corresponding to pie chart visualization types), histograms (for example, rules corresponding to histogram visualization types), and single values ​​(for example, single values This may include rules corresponding to various other types of visualizations and their corresponding rules.

[0041] The rulebook may contain specific rules that match the desired visualization. For example, if the intent is to show relationships between specific data / variables inferred from a natural language query, the rulebook may contain rules for scatter plot visualizations and rules for bubble chart visualizations. Once it is determined that relationships between variables should be visualized, the rulebook lookup module 235 may determine the number of variables whose relationships should be visualized and look up rules (or multiple rules) that match that number of variables (e.g., rules from a set of rules configured to show relationships between variables). As an example, if the relationship between two variables should be visualized, the selected visualization type (e.g., rules matching data abstraction) is a scatter plot. As another example, if the relationship between three variables should be visualized, the selected visualization type is a bubble chart.

[0042] The rulebook may include a rule hierarchy through which the rulebook lookup module 235 walks a decision tree. The first level of the rule hierarchy may include determining whether the desired visualization is relational, comparative, distributional, or composite. In response to determining the desired visualization, the rulebook lookup module 235 determines the rules that should be evaluated for the desired visualization.

[0043] In some embodiments, in response to determining that the desired visualization is a comparison, the rule hierarchy includes determining whether the comparison is between items or over time.

[0044] In response to determining that the comparison is between items, the rule hierarchy includes determining the number of variables for each item being compared. If the number of variables is two per item, the rule matching the data abstraction is a variable-width volume chart. In response to determining that the comparison between items includes one variable per item, the rule hierarchy includes determining whether the data includes many categories or few categories. In response to determining that the data includes many categories, the rule matching the data abstraction is a table or a table with an embedded chart. Conversely, in response to determining that the data includes few categories, the rule hierarchy includes determining whether the data includes many items or few items. If the data includes many items, the rule matching the data abstraction is a horizontal bar chart. If the data includes few items, the rule matching the data abstraction is a vertical bar chart. The number of categories considered to be many or few categories may be predefined by user preferences, etc. Similarly, the number of items considered to be many or few items may be predefined.

[0045] In response to the determination that the comparison is over time, the rule hierarchy includes determining whether the time period over which the comparison is performed is a large number of periods or a small number of periods. The number of periods considered to be a large number or a small number of periods may be predefined by user preferences, etc. In response to the determination that the time period includes many periods, the rule hierarchy includes determining whether the data is periodic or aperiodic. In response to the determination that the data is cyclical, the rule matching the data abstraction corresponds to a circular area chart. Conversely, in response to the determination that the data is aperiodic, the rule matching the data abstraction corresponds to a line graph. In response to the determination that the time period includes a small number of periods, the rule hierarchy includes determining whether the data contains a single category or a small number of categories, or whether the data contains many categories. The number of categories considered to be many or a small number of categories may be predefined by user preferences, etc. In response to the determination that the number of categories is a single category or a small number of categories, the rule matching the data abstraction corresponds to a bar chart (e.g., visualization type = bar chart). Conversely, in response to the determination that the data contains multiple categories, the rules that match the data abstraction correspond to line graphs.

[0046] In response to the decision that the data distribution should be visualized, the rule hierarchy includes determining whether the number of variables is one, two, or three. In response to the decision that the number of variables is a single variable, the rule hierarchy includes determining whether the data contains a small number of data points or a large number of data points. The number of data points considered to be a large or small number may be predefined by user preference, etc. In response to the decision that the data contains a small number of data points, the rule matching the data abstraction is considered a rule for column histogram visualization. Conversely, in response to the decision that the data contains a large number of data points, the rule matching the data abstraction is considered a rule for line histogram visualization. If the data is determined to contain two variables, the rule matching the data abstraction is considered a rule for scatter plot visualization. If the data is determined to contain three variables, the rule matching the data abstraction is considered a rule for 3D area chart.

[0047] In response to the decision that the data structure should be visualized, the rule hierarchy includes determining whether the data structure is static or changes over time.

[0048] In response to the decision that the data structure is static, the rule hierarchy includes determining whether a simple share of the total should be visualized, whether an accumulation or subtraction of the total should be visualized, or whether one of the components should be visualized. In response to the decision that a simple share of the total should be visualized, the rule matching the data abstraction is considered a rule for a pie chart visualization. In response to the decision that an accumulation or subtraction of the total should be visualized, the rule matching the data abstraction is considered a rule for a waterfall chart visualization. In response to the decision that one of the components should be visualized, the rule matching the data abstraction is considered a rule for a stacked 100% bar chart with visualizations of sub-components.

[0049] In response to the determination that the data structure changes over time, the rule hierarchy includes determining whether the data includes a small number of periods or a large number of periods. The number of periods considered to be a large number or a small number may be predefined by user preference, etc. In response to the determination that the data includes a large number of periods, the rule hierarchy includes determining whether only the relative differences between data structures matter, or whether the relative and absolute differences between data structures matter. In response to the determination that only the relative differences between data structures matter, rules matching the data abstraction are considered rules for a stacked 100% chart visualization. In response to the determination that the relative and absolute differences between data structures matter, rules matching the data abstraction are considered rules for a stacked area chart visualization. In response to the determination that the data includes several periods, the rule hierarchy includes determining whether only the relative differences between data structures matter, or whether the relative and absolute differences between data structures matter. In response to the determination that only the relative differences between data configurations matter, the rules that match the data abstraction are considered to be the rules for a stacked 100% bar chart visualization. In response to the determination that both the relative and absolute differences between data configurations matter, the rules that match the data abstraction are considered to be the rules for a stacked bar chart visualization.

[0050] The rulebook may include various other rule hierarchies or rules for selecting visualization types.

[0051] In some embodiments, the system 200 includes a visualization generation module 237. The system 200 uses the visualization generation module 237 to generate visualizations based on visualization types corresponding to rules that are considered to match the data abstraction.

[0052] In some embodiments, the visualization generation module 237 generates a visualization definition. In some embodiments, the visualization definition is generated in a data visualization language, which is a high-level language closely related to natural language. For example, the visualization definition is a high-level description of the visualization, such as a representation of the requested visualization in a language closely related to natural language. The visualization generation module 237 generates the visualization definition based on one or more of the data abstractions, task abstractions, design abstractions, and rules identified by the rulebook lookup module 235. In some embodiments, the visualization definition includes a first dimension of the data to be visualized, a second dimension of the data to be visualized, and an indication of the type of visualization. The visualization may be multidimensional (e.g., n dimensions, where n is a positive integer). Thus, the visualization may include indications of other dimensions of the data to be visualized.

[0053] In response to determining the visualization definition, the visualization generation module 237 generates or causes a visualization to be generated. For example, the visualization generation module 237 directly generates a visualization from the visualization definition. As another example, the visualization generation module 237 calls another application or service to generate and provide a visualization based at least partially on the visualization definition. Generating a visualization may include translating the visualization definition into another language, such as the language associated with the particular visualization type to be generated, or the language used by the associated application / service to be called to generate the visualization.

[0054] In some embodiments, the system 200 includes a user feedback module 239. The system 200 uses the user feedback module 239 to receive user feedback on whether the visualizations provided in response to natural language queries are correct, engaging, or useful, etc. The user feedback module 239 may use the user interface module 241 to prompt the user to provide feedback. Developers can use the feedback in connection with updating rulebooks (e.g., rule hierarchies) or decision trees to update visualization types with specific characteristics or sets of statistics in the data abstraction.

[0055] In some embodiments, the system 200 includes a user interface module 241. The system 200 uses the user interface module 241 to configure a user interface and provide it to the user, for example, to a client system used by the user. The user interface module 241 is configured to provide visualizations generated in response to natural language queries. Additionally, the user interface module 241 may include various input fields or selectable elements from which the user can provide user feedback. In some embodiments, the user interface module 241 provides an interface from which the user can select from several visualization types for a particular natural language query (for example, if several rules match a data abstraction), and in response to receiving the user selection, the system 200 uses a visualization generation module 237 to generate a visualization according to the selected visualization type.

[0056] According to various embodiments, storage 215 includes one or more of file system data 260, query data 265, and rulebook data 270. Storage 215 also includes shared storage (e.g., a network storage system) and / or database data and / or user activity data.

[0057] In some embodiments, the file system data 260 includes a database such as one or more datasets (e.g., one or more datasets of domains, a dataset containing samples of network traffic, a dataset containing sample classifications, a dataset of security events / expirations, a mapping of network traffic instructions or predicted network traffic classifications to network traffic or hashes, a mapping of network traffic signatures or other unique identifiers such as domain signatures, hashes, signatures or benign traffic indicators to network traffic, etc.).

[0058] Query data 265 contains information about one or more natural language queries received by the system 200. For example, query data 265 stores natural language queries received by the query receiving module 227. In some embodiments, query data 265 includes query intents, task intents, design intents, and / or result data corresponding to the natural language queries. Query data 265 may optionally store historical query data, such as result data, visualization data, and intent data for previously processed queries.

[0059] Rulebook data 270 contains information about one or more rules for determining the visualization type. Rulebook data 270 may include a rule hierarchy or decision tree that is walked through to determine a rule (e.g., visualization type) that matches a data abstraction. Rulebook data 270 may include user feedback regarding visualization types or visualization definitions for a particular data abstraction, query (e.g., query intent, task intent, etc.).

[0060] According to various embodiments, memory 220 includes execution application data 275. Execution application data 275 includes data obtained or used in connection with executing applications such as applications for determining or predicting whether a particular sample corresponds to malicious or benign traffic, applications for extracting information from web page content, input strings, applications for extracting information from files, or other samples. In embodiments, applications include one or more applications that perform one or more of the following: receiving and / or executing queries or tasks, generating reports and / or configuration information in response to executed queries or tasks, and / or providing user information in response to queries or tasks. Other applications include any other appropriate applications (e.g., index maintenance applications, communication applications, machine learning model applications, applications for detecting suspicious input strings, applications for detecting suspicious files, suspicious or unparked domains, applications for detecting malicious network traffic, or malicious / non-compliant applications with respect to corporate security policies, etc., document creation applications, reporting applications, user interface applications, data analysis applications, anomaly detection applications, user authentication applications, security policy management / update applications, etc.).

[0061] Figure 3 is a block diagram of a system for generating data visualizations according to various embodiments. In some embodiments, system 300 is implemented in relation to at least a portion of system 100 in Figure 1. In some embodiments, system 300 implements at least a portion of process 1100 in Figure 11, process 1200 in Figure 12, process 1400 in Figure 14, process 1500 in Figure 15, process 1600 in Figure 16, process 1700 in Figure 17, process 1800 in Figure 18, process 2000 in Figure 20, process 2300 in Figure 23, and / or process 2400 in Figure 24. System 300 may be implemented by a cloud service (a set of one or more cloud resources that provide services to a user, such as providing visualizations to a user).

[0062] In some embodiments, system 300 implements the cognitive services 140 of system 100.

[0063] In the illustrated example, system 300 implements one or more modules related to interpreting natural language queries and providing visualizations based on natural language queries. System 300 comprises a communication interface 305, one or more processors 310, storage 315, and / or memory 320. One or more processors 310 comprises one or more of the following: a communication module 325, a query receiving module 327, a query interpretation engine module 329, a data retrieval module 331, a data abstraction module 333, a rule scoring module 235, a rule selection module 337, a visualization generation module 339, a user feedback module 341, and / or a user interface module 343.

[0064] In some embodiments, system 300 includes a communication module 325. System 300 uses the communication module 325 to communicate with various nodes or endpoints (e.g., client terminals, firewalls, DNS resolvers, data appliances, other security entities, etc.) or user systems such as an administrator system. For example, the communication module 325 provides information to be communicated (e.g., to another node, security entity, etc.) to a communication interface 305. In another example, the communication interface 305 provides information received by system 300 to the communication module 325. The communication module 325 is configured to receive natural language queries, selections of visualization types, user feedback for visualizations, etc. The communication module 325 is configured to query third-party services or data sources for data that is expected to respond to queries, such as subqueries of natural language queries. The communication module 325 is further configured to receive one or more settings or configurations from an administrator. Examples of one or more settings or configurations include the configuration of a machine learning model (e.g., a model used to interpret natural language queries), the configuration of a visualization, etc.

[0065] In some embodiments, system 300 includes a query receiving module 327. System 300 uses the query receiving module 327 to receive user input, which includes natural language queries such as queries entered by the user into the user interface. The query receiving module 327 may correspond to or be similar to the query receiving module 227 of system 200.

[0066] In some embodiments, system 300 includes a query interpretation engine module 329. System 300 uses the query interpretation engine module 329 to interpret queries, such as natural language queries, received by the query receiving module 327. The query interpretation engine module 329 may correspond to or be similar to the query interpretation engine module 229 of system 200.

[0067] In some embodiments, system 300 includes a data retrieval module 331. System 300 uses the data retrieval module 331 to retrieve data to be used in connection with generating result data for natural language queries. The data retrieval module 331 may correspond to or be similar to the data retrieval module 231 of system 200.

[0068] In some embodiments, system 300 includes a data abstraction module 333. System 300 uses the data abstraction module 333 to abstract data extracted from one or more data sources. The data abstraction module 333 may correspond to or be similar to the data abstraction module 233 of system 200.

[0069] In some embodiments, the system 300 includes a rule scoring module 335. The system 300 uses the rule scoring module 335 to score rules for visualizing result data of natural language queries. The rule scoring module 335 may rank the rules for visualizing result data according to a score (e.g., an aggregate score of multiple scoring criteria). Examples of scoring criteria include relevance, scope, decorativeness, and user preference.

[0070] In some embodiments, scoring rules for visualizing the resulting data (e.g., data summarized / represented by data abstraction) involves calling process 1200 in Figure 12. For example, the rule scoring module 335 determines all possible ordered subsets of the resulting data, checks each of the given rules (e.g., rules in a rulebook) against each subset of the data, and creates a score vector for each rule. The score vector may be created along dimensions defined by the scoring criteria. In response to determining the score vector for each rule, the rules are ranked, for example, by Pareto front optimization.

[0071] In some embodiments, the system 300 includes a rule selection module 337. The system 300 uses the rule selection module 337 to select rules that match, or most closely match, the resulting data represented / summarized by the data abstraction. For example, the rule selection module 337 performs a Pareto front optimization algorithm on the score vector of each rule and then selects rules corresponding to data points on the Pareto front.

[0072] In some embodiments, system 300 includes a visualization generation module 339. System 300 uses the visualization generation module 339 to generate visualizations based on visualization types corresponding to rules considered to match a data abstraction. The rules considered to match a data abstraction are those selected by the rule selection module 337 (e.g., the rule with the highest ranking). The visualization generation module 339 may correspond to or be similar to the visualization generation module 237 of system 200.

[0073] In some embodiments, system 300 includes a user feedback module 341. System 300 uses the user feedback module 341 to obtain user feedback for visualizations provided in response to natural language queries. User feedback module 341 may correspond to or be similar to the user feedback module 239 of system 200.

[0074] In some embodiments, the system 300 uses user feedback to update the user preference score in the associated score vector(s).

[0075] In some embodiments, system 300 includes a user interface module 343. System 300 uses the user interface module 343 to configure and provide a user interface to the user, such as a client system used by the user. User interface module 343 may correspond to or be similar to user interface module 241 of system 200.

[0076] According to various embodiments, storage 315 includes one or more of file system data 360, query data 365, and rulebook data 370. Storage 315 also includes shared storage (e.g., a network storage system) and / or database data and / or user activity data.

[0077] In some embodiments, the file system data 360 includes a database such as one or more datasets (e.g., one or more datasets of domains, a dataset containing samples of network traffic, a dataset containing sample classifications, a dataset of security events / expirations, a mapping of network traffic instructions or predicted network traffic classifications to network traffic or hashes, a mapping of network traffic signatures or other unique identifiers such as domain signatures, hashes, signatures or benign traffic indicators to network traffic, etc.).

[0078] Query data 365 contains information about one or more natural language queries received by the system 300. For example, query data 365 stores natural language queries received by the query receiving module 327. In some embodiments, query data 365 includes query intents, task intents, design intents, and / or result data corresponding to the natural language queries. Query data 365 may optionally store historical query data, such as result data, visualization data, and intent data for previously processed queries.

[0079] The rulebook data 370 contains information about one or more rules for determining the visualization type. The rulebook data 370 may include a rule hierarchy or decision tree that is walked through to determine a rule (e.g., visualization type) that matches a data abstraction. The rulebook data 370 may include user feedback regarding visualization types or visualization definitions for specific data abstractions, queries (e.g., query intents, task intents, etc.). The rulebook data 370 may store user preference values ​​for rules, which can be updated in an associated score vector for using the rule with respect to an ordered subset of the data.

[0080] According to various embodiments, memory 320 includes execution application data 375. Execution application data 375 includes data obtained or used in connection with executing applications such as applications for determining or predicting whether a particular sample corresponds to malicious or benign traffic, applications for extracting information from web page content, input strings, applications for extracting information from files, or other samples. In embodiments, applications include one or more applications that perform one or more of the following: receiving and / or executing queries or tasks, generating reports and / or configuration information in response to executed queries or tasks, and / or providing user information in response to queries or tasks. Other applications include any other appropriate applications (e.g., index maintenance applications, communication applications, machine learning model applications, applications for detecting suspicious input strings, applications for detecting suspicious files, suspicious or unparked domains, applications for detecting malicious network traffic, or malicious / non-compliant applications with respect to corporate security policies, etc., document creation applications, reporting applications, user interface applications, data analysis applications, anomaly detection applications, user authentication applications, security policy management / update applications, etc.).

[0081] Figure 4 is a block diagram of a system for generating data visualizations according to various embodiments. In some embodiments, system 400 is implemented in relation to at least a portion of system 100 in Figure 1. In some embodiments, system 400 implements at least a portion of process 1300 in Figure 13, process 1500 in Figure 15, process 1600 in Figure 16, process 1700 in Figure 17, process 1800 in Figure 18, process 2200 in Figure 22, process 2300 in Figure 23, and / or process 2400 in Figure 24. System 400 may be implemented by a cloud service (a set of one or more cloud resources that provide services to a user, such as providing visualizations to a user).

[0082] In some embodiments, system 400 implements the cognitive services 140 of system 100.

[0083] In the illustrated example, system 400 implements one or more modules related to interpreting natural language queries and providing visualizations based on natural language queries. System 400 comprises a communication interface 405, one or more processors 410, storage 415, and / or memory 420. One or more processors 410 comprises one or more of the following: a communication module 425, a query receiving module 427, a query interpretation engine module 429, a data retrieval module 431, a data abstraction module 433, a model training module 435, a prediction engine module 437, a visualization generation module 439, a user feedback module 441, and / or a user interface module 443.

[0084] In some embodiments, system 400 includes a communication module 425. System 400 uses the communication module 425 to communicate with various nodes or endpoints (e.g., client terminals, firewalls, DNS resolvers, data appliances, other security entities, etc.) or user systems such as an administrator system. For example, the communication module 425 provides information to be communicated (e.g., to another node, security entity, etc.) to a communication interface 405. In another example, the communication interface 405 provides information received by system 400 to the communication module 425. The communication module 425 is configured to receive natural language queries, selections of visualization types, user feedback for visualizations, etc. The communication module 425 is configured to query third-party services or data sources for data that is expected to respond to queries, such as subqueries of natural language queries. The communication module 425 is further configured to receive one or more settings or configurations from an administrator. Examples of one or more settings or configurations include the configuration of a machine learning model (e.g., a model used to interpret natural language queries), the configuration of a visualization, etc.

[0085] In some embodiments, system 400 includes a query receiving module 427. System 400 uses the query receiving module 427 to receive user input, which includes natural language queries such as queries entered by the user into the user interface. The query receiving module 427 may correspond to or be similar to the query receiving module 227 of system 200.

[0086] In some embodiments, system 400 includes a query interpretation engine module 429. System 400 uses the query interpretation engine module 429 to interpret queries, such as natural language queries, received by the query receiving module 427. The query interpretation engine module 429 may correspond to or be similar to the query interpretation engine module 229 of system 200.

[0087] In some embodiments, system 400 includes a data retrieval module 431. System 400 uses the data retrieval module 431 to retrieve data to be used in connection with generating result data for natural language queries. The data retrieval module 431 may correspond to or be similar to the data retrieval module 231 of system 200.

[0088] In some embodiments, system 400 includes a data abstraction module 433. System 400 uses the data abstraction module 433 to abstract data extracted from one or more data sources. The data abstraction module 433 may correspond to or be similar to the data abstraction module 233 of system 200.

[0089] In some embodiments, the system 400 includes a model training module 435. The system 400 uses the model training module 435 to train a machine learning model(s). The model training module 435 may train or otherwise acquire a model for interpreting natural language queries. For example, the model training module 435 trains a large-scale language model to infer query intents, task intents, and / or design intents from a natural language model. The model used to interpret natural language queries may also be used to predict visualization types or to generate predicted visualization definitions.

[0090] In some embodiments, the model used to predict the visualization type or to generate the predicted visualization definition is different from the model used to interpret the natural language query. For example, the model training module 435 may train a model specifically to predict the visualization type based on one or more of the data abstraction, query intent, task intent, and / or design intent. As another example, the model training module 435 may train a model specifically to predict the visualization definition based on one or more of the data abstraction, query intent, task intent, and / or design intent.

[0091] In some embodiments, system 400 includes a prediction engine module 437. System 400 uses the prediction engine module 437 to implement a model and obtain a predicted visualization definition based on one or more of the data abstractions, query intents, task intents, and / or design intents. Alternatively, the prediction engine module 437 queries the model for the predicted visualization types of system 200 that should be used to generate the corresponding visualization definitions based on one or more of the data abstractions, query intents, task intents, and / or design intents.

[0092] In some embodiments, system 400 includes a visualization generation module 439. System 400 uses the visualization generation module 439 to generate visualizations based on visualization types corresponding to rules considered to match the data abstraction. The visualization generation module 439 may correspond to or be similar to the visualization generation module 237 of system 200.

[0093] In some embodiments, system 400 includes a user feedback module 441. System 400 uses the user feedback module 441 to obtain user feedback for visualizations provided in response to natural language queries. User feedback module 441 may correspond to or be similar to the user feedback module 239 of system 200.

[0094] In some embodiments, system 400 includes a user interface module 443. System 400 uses the user interface module 443 to configure and provide a user interface to the user, such as a client system used by the user. User interface module 443 may correspond to or be similar to user interface module 241 of system 200.

[0095] According to various embodiments, storage 415 includes one or more of file system data 460, query data 465, and model data 470. Storage 415 also includes shared storage (e.g., a network storage system) and / or database data and / or user activity data.

[0096] In some embodiments, the file system data 460 includes a database such as one or more datasets (e.g., one or more datasets of domains, a dataset containing samples of network traffic, a dataset containing sample classifications, a dataset of security events / expirations, a mapping of network traffic instructions or predicted network traffic classifications to network traffic or hashes, a mapping of network traffic signatures or other unique identifiers such as domain signatures, hashes, signatures or benign traffic indicators to network traffic, etc.).

[0097] Query data 465 contains information about one or more natural language queries received by the system 400. For example, query data 465 stores natural language queries received by the query receiving module 427. In some embodiments, query data 465 includes query intents, task intents, design intents, and / or result data corresponding to the natural language queries. Query data 465 may optionally store historical query data, such as result data, visualization data, and intent data for previously processed queries.

[0098] Model data 470 contains information about one or more models used to predict visualization types or visualization definitions. Model data 470 may include historical predictions of visualization types or visualization definitions. For example, model data 470 stores historical predictions in relation to a natural language query, or one or more of the following: query intent, data abstraction, task abstraction, and / or design abstraction. Model data 470 may include the models used to generate predictions, and / or embeddings or feature vectors used to generate predictions using the models. In some embodiments, model data 470 includes machine learning models for interpreting natural language queries (e.g., for inferring the intent / context of a query). For example, model data 470 may include large language models (e.g., GPT-3, GPT-4, ChatGPT, Bard, etc.).

[0099] According to various embodiments, memory 420 includes execution application data 475. Execution application data 475 includes data obtained or used in connection with executing applications such as applications for determining or predicting whether a particular sample corresponds to malicious or benign traffic, applications for extracting information from web page content, applications for extracting information from input strings, files, or other samples. In embodiments, applications include one or more applications that perform one or more of the following: receiving and / or executing queries or tasks, generating reports and / or configuration information in response to executed queries or tasks, and / or providing user information in response to queries or tasks. Other applications include any other appropriate applications (e.g., index maintenance applications, communication applications, machine learning model applications, applications for detecting suspicious input strings, applications for detecting suspicious files, suspicious or unparked domains, applications for detecting malicious network traffic, or malicious / non-compliant applications with respect to corporate security policies, etc., document creation applications, reporting applications, user interface applications, data analysis applications, anomaly detection applications, user authentication applications, security policy management / update applications, etc.).

[0100] Figure 5 shows an example of determining a visualization definition based on natural language user input in various embodiments. System 500 receives a natural language query, generates a visualization definition, and the visualization definition can then be used to generate a visualization. User 510 provides a natural language query to System 500. User 510 may communicate the natural language query using a client terminal. For example, User 510 inputs a natural language query into a user interface 520 presented on a client terminal used by User 510. In response to receiving a natural language query (e.g., "I would like to see the number of incidents over last week in the USA"), System 500 provides the natural language query to the cognitive service 530 to generate a visualization definition 540 that defines the visualization to be provided in response to the natural language query. System 500 may additionally generate a visualization based on the visualization definition.

[0101] In some embodiments, the cognitive service 530 analyzes a natural language query, queries one or more models based at least partially on the natural language query, and generates a visualization definition.

[0102] In response to receiving a natural language query, the cognitive service 530 determines a query intent, which may include one or more of the data intent, task intent, and design intent. Determining the query intent involves querying a machine learning model, such as a large-scale language model, to interpret the natural language query and obtain the query intent. Various other machine learning models may be used to determine the query intent.

[0103] In response to the determination of the query intent, the cognitive service 530 determines one or more requests (e.g., one or more queries) to retrieve data that responds to the natural language query. The system 500 determines one or more sets of data sources that store data that responds to the queries (e.g., data identified in the query intent) and generates one or more requests based on the set of data sources. The cognitive service 530 communicates one or more requests to the corresponding data source(s) to retrieve the response data.

[0104] In response to receiving response data, the cognitive service 530 analyzes the data, abstracts the response data, and obtains a data abstraction. The data abstraction may be a high-level description of one or more properties / features of the data. The data abstraction may be a statistical summary of the data.

[0105] In response to obtaining a data abstraction for the response data, the cognitive service 530 determines a visualization definition for the natural language query, at least partially based on the data abstraction. The visualization definition is generated according to a predetermined language, such as a data visualization language. In some embodiments, the visualization definition is generated in the data visualization language in an accurate and computable manner. The generation of the visualization definition in the data visualization language compels the cognitive service 530 to accurately describe the space of all possible visualizations. The data visualization language may be an intermediate language that can be interpreted by a machine learning model, such as a large-scale language model (e.g., ChatGPT). For example, the data visualization language is relatively close to natural language, making it interpretable by a large-scale language model.

[0106] The Cognitive Service 530 may determine visualization definitions based at least partially on a rulebook, perform scoring of rules within the rulebook, or query a model or prediction engine to predict the visualization type from which a visualization definition will be generated, or to predict a visualization definition. Using a rulebook may involve performing lookups to determine rules that match a natural language query (e.g., data abstractions performed on query result data, query intents, etc.) and determining the visualization type associated with such rules. Performing scoring of rules may involve ranking rules according to their scores (e.g., scores determined based on one or more scoring criteria) and selecting the rule with the highest ranking. The visualization type corresponding to the selected rule is used to determine the visualization definition.

[0107] Figure 6A shows an example of a set of data retrieved in relation to a natural language query in various embodiments. In the illustrated example, the response data 600 includes a mapping of country, time, number of incidents, and device type. The system retrieves the response data 600 for a natural language query. The natural language query is processed to determine the query intent, and the system generates a set of one or more requests to a data store that stores the information to be retrieved.

[0108] Figure 6B shows an example of data abstraction performed with respect to a set of data for a query, according to various embodiments. In the illustrated example, the system generates a data abstraction 650 for response data 600. In some embodiments, generating a data abstraction involves analyzing the response data 600 and determining one or more properties of the response data, such as statistics for the response data 600. As an example, the data abstraction 650 is generated by processing a request for a view of summary statistics for the response data 600. For the response data 600, the request for a view of summary statistics may include "viewof summary_data = SummaryTable(data, {label: “Incident Data”})".

[0109] Figure 7A shows an example of a set of data retrieved in relation to a natural language query in various embodiments. In the illustrated example, the response data 700 includes a mapping of time and the number of incidents (e.g., the number of incidents as a function of time). The system retrieves the response data 700 for a natural language query. The natural language query is processed to determine the query intent, and the system generates a set of one or more requests to a data store that stores the information to be retrieved.

[0110] Figure 7B shows an example of data abstraction performed with respect to a set of data for a query, according to various embodiments. In the illustrated example, the system generates a data abstraction 750 for response data 700. In some embodiments, generating a data abstraction involves analyzing the response data 700 and determining one or more properties of the response data, such as statistics for the response data 700. As an example, the data abstraction 750 is generated by processing a request for a view of summary statistics for the response data 700. For the response data 700, the request for a view of summary statistics may include "viewof summary_data = SummaryTable(data, {label: “Incident Data”})".

[0111] Data abstraction can correspond to characteristic-based data descriptions, such as high-level descriptions of data based on data statistics. Examples of statistics for response data include mean, median, standard deviation, range, maximum, minimum, and number of data elements / records. Various other statistics may be determined and used in connection with the generation of data abstractions. In some embodiments, statistics are selected based on the type of data contained in the response data, the amount of data / records in the response data, etc.

[0112] In some embodiments, the system further abstracts the task(s) of the corresponding natural language query. For example, the system determines a set of corresponding domain-specific tasks. Examples of domain-specific tasks associated with a natural language query might include (i) whether more incidents occurred yesterday, (ii) which day of the week had the most incidents, and (iii) how the number of incidents changed over the past week. The system may abstract the set of domain-specific tasks into an abstracted set of tasks: (i) compare values, (ii) identify outliers, and (iii) discover distributions.

[0113] Figure 8 shows the process for generating visualizations according to various embodiments. Process 800 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4.

[0114] In 810, the system obtains a data abstraction of a natural language query. The system obtains a data abstraction at least in part based on abstracting data in response to a request generated for a particular natural language query.

[0115] In step 820, the system determines a visualization definition for the abstracted data. The visualization definition may correspond to a description or definition of the visualization in a data visualization language.

[0116] In some embodiments, the visualization definition is determined at least in part on the basis of applying a specific rule from a given set of rules that map data abstractions to visualization types. The specific rule may be selected from a rulebook based on one or more decision rules, or based on scoring various rules for a particular data abstraction and selecting the highest-ranking rule.

[0117] In some embodiments, the visualization definition is determined at least in part on a prediction engine. The prediction engine may comprise a model such as a machine learning model. In response to having determined the data abstractions, the system queries the model for the predicted visualization definitions. The model is trained on a training set that includes a set of data abstractions and the corresponding visualizations or visualization definitions. The model may be further retrained based on user feedback, such as feedback provided in response to a user receiving data visualizations for a particular natural language query, or feedback in the form of selection of a desired visualization when a user is presented with a set of different types of visualizations for a particular natural language query.

[0118] In 830, the system generates a visualization. In some embodiments, the system generates a visualization based at least partially on a visualization definition. The system may process the visualization definition in relation to the generation of the visualization. For example, the system may translate the visualization definition (e.g., from a data visualization language) into another high-level language. The other high-level language may be selected based on the type of visualization to be generated, etc.

[0119] Figure 9 shows examples of visualization definitions and corresponding visualizations according to various embodiments. In some embodiments, system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4 can generate definitions and corresponding visualizations.

[0120] In the illustrated example, visualization set 900 includes various examples of the types of visualizations that can be generated. Examples of visualization types include bar chart representation 910 (e.g., a representation in which results are stacked within a bar chart along a particular dimension), parallel set representation 920, bubble chart representation 930, donut representation 940, bar chart 950 (e.g., a representation in which a separate bar is provided for each variable along a particular dimension), and line chart 960. The visualization definitions shown with each visualization type are provided in a relatively high-level data visualization language that is close to natural language. In some implementations, a language model may be used to generate visualization definitions and in relation to using a predictive engine that interprets data abstractions to predict the type of visualization that should be generated to represent the data. In some implementations, a language model is used to generate visualization definitions for query intents based on a given visualization type (for example, the visualization type may be selected based on a rule set or rule ranking).

[0121] Figure 10 shows a process for generating visualizations according to various embodiments. Process 1000 is carried out by system 100 in Figure 1 and / or system 200 in Figure 2.

[0122] In 1005, natural language input is received. The input may correspond to a natural language query. In some embodiments, the natural language input is received in a user interface presented to the user on a client device. In 1010, a query intent is determined based on the natural language input. In some embodiments, the system queries a model to interpret the natural language input in relation to determining the query intent. Determining the query intent may include determining the data intent (e.g., determining the parameters of the data to respond to the query). The model is a machine learning model such as a Large-Scale Language Model (LLM) (e.g., GPT-3, ChatGPT, Bard, or other LLMs). The model may parse the natural language input to determine the query. For example, the system infers one or more intents or contexts for visualization, such as a query intent, a task intent, and / or a design intent.

[0123] In step 1015, the system determines the data to respond to the query. In response to determining the query, the system retrieves the data to respond to the query (e.g., result data). The system may determine a set of one or more data sources to store the data to respond to the query, and in response to determining a set of one or more data sources, the system may retrieve the result data from that set of data sources. For example, the system generates one or more requests for the corresponding data in each of the data sources within the set of one or more data sources.

[0124] In 1020, the system determines a task intent for natural language input. In some embodiments, the system determines the task intent based on querying a model (e.g., an LLM) to analyze / interpret the natural language input and derive the task intent. The task intent indicates why the visualization is being used. Examples of domain-specific tasks that may be derived from natural language input include (i) "Were there more incidents yesterday?", (ii) "Which day had the most incidents this week?", and (iii) "How did the number of incidents change over the past week?".

[0125] In 1025, the system determines the design intent for the natural language input. In some embodiments, the system determines the design intent based on querying a model (e.g., an LLM) to analyze / interpret the natural language input and derive a task intent. The design intent indicates elements of the desired visualization, such as labeling and color scheme.

[0126] Although process 1000 indicates that steps 1015, 1020, and 1025 are executed in different steps, the system may use a single query to the LLM to interpret the natural language query and determine the data intent, task intent, and design intent.

[0127] After determining the query intent (e.g., data intent, task intent, and / or design intent), the system performs abstraction to translate, for example, a natural language query into a high-level description of a visualization (e.g., a representation of the visualization in a data visualization language). The high-level description of the visualization (e.g., a visualization definition) may be a representation of the requested visualization in a language closely related to natural language. Based on the query intent, the system accurately describes the space of all possible visualizations. In some embodiments, the system uses the query intent (e.g., by performing query abstraction to determine the data intent, task intent, and / or design intent) to describe the visualization in an accurate and computable manner. In some embodiments, the visualization definition includes a first dimension of the data to be visualized, a second dimension of the data to be visualized, and an indication of the type of visualization. Visualizations can be multidimensional (for example, n-dimensional, where n is a positive integer). Therefore, a visualization may include indications of other dimensions of the data to be visualized.

[0128] In 1030, the system determines the data abstraction. In some embodiments, the system abstracts the data, for example, by determining the characteristics / statistics of the data in response to the query. Data abstraction may include determining whether the response data is numerical, categorical, hierarchical, multidimensional, time-series, map, etc.

[0129] In 1035, the system determines a task abstraction. In some embodiments, the system performs an abstraction with respect to the task intent in order to determine a task abstraction. An example of a task abstraction is determining that the task intent corresponds to a determination of distribution, comparison, relation, and / or composition. Various other task abstractions may be determined. Using the domain-specific task example described in relation to 1020, the corresponding task abstraction may include (i) comparing values, (ii) identifying outliers, and / or (iii) discovering a distribution.

[0130] In 1040, the system determines the design abstraction. By abstracting the design intent, the user can define / modify the design aspects of the desired visualization. Examples of design abstractions for natural language input include size, aspect ratio, labels and / or legends, colors (e.g., color scheme), orientation, scale, accessibility, interaction, and layout structure. Various other design abstractions may be implemented.

[0131] In 1045, the system performs a lookup against the rulebook to determine the rule that matches (or most closely matches) the data abstraction. Alternatively, the system performs a lookup against the rulebook to determine the rule that matches (for example, corresponds to) the data abstraction, task abstraction, and design abstraction (e.g., corresponding to a specific visualization type). The rulebook may contain a predetermined set of rules corresponding to each visualization type.

[0132] In 1050, the system generates a visualization definition (e.g., a representation of a natural language query). In some embodiments, the visualization definition is generated in a data visualization language, which is a high-level language closely related to natural language. The system generates the visualization definition based on one or more of the data abstractions, task abstractions, design abstractions, and rules identified in 1045. For example, the system determines the visualization definition using the visualization type associated with the identified rule. In some embodiments, the visualization definition includes a first dimension of the data to be visualized, a second dimension of the data to be visualized, and an indication of the type of visualization. The visualization may be multidimensional (e.g., n dimensions, where n is a positive integer). Thus, the visualization may include indications of other dimensions of the data to be visualized.

[0133] In 1055, the system generates a visualization. The visualization is generated at least in part based on the visualization definition. In some embodiments, the system translates the visualization definition into another high-level language, such as the language associated with the particular visualization type to be generated.

[0134] Figure 11 shows a process for generating visualizations according to various embodiments. Process 1100 is carried out by system 100 in Figure 1, system 200 in Figure 2, and / or system 300 in Figure 3.

[0135] Process 1100 may include two main steps 1110 and 1120. Step 1110 may include receiving a natural language query and interpreting the natural language query to determine query criteria such as query intent, task intent, and / or design intent. Step 1120 may include using the query interpretation (e.g., query intent, task intent, and / or design intent) to determine the type of visualization to be produced for the natural language query and producing the visualization accordingly.

[0136] In the illustrated example, step 1110 includes steps 1111 to 1116, which are described below.

[0137] In 1111, a natural language query is received. 1111 may correspond to or be similar to 1005. In 1112, the natural language query is interpreted using a machine learning model (e.g., LLM) to determine the query (e.g., query criteria / query intent). 1112 may correspond to or be similar to 1010. In 1113, the query criteria (e.g., query intent or other definition of the query) are determined based on the output from the machine learning model. In 1114, data considered to respond to the query is retrieved. The system determines one or more data sources to be queried to retrieve the resulting data, at least partially based on the criteria. In 1115, task criteria (e.g., task intent) are determined. The task criteria may be determined at least partially based on the interpretation of the natural language query using a machine learning model (e.g., LLM). 1115 may correspond to or be similar to 1020. In 1116, design criteria (e.g., design intent) are determined. The design criteria may be determined directly from the natural language query, or they may be determined based on the interpretation of the natural language query using a machine learning model (e.g., LLM). 1116 may correspond to or be similar to 1025.

[0138] In some embodiments, steps 1114 and 1115 (e.g., determining task criteria and design criteria) may be performed at an optional rate. For example, the visualization may be generated based on the response data during data abstraction. The system may generate a visualization definition based on the data abstraction and then generate the visualization.

[0139] In the illustrated example, step 1120 includes steps 1121 to 1126, which are described below.

[0140] In 1121, the system abstracts criteria / parameters for natural language. In the illustrated example, in 1121, the system determines a data abstraction based on the resulting data obtained from a set of one or more data sources identified based on the query intent. In other implementations, 1121 may include using the task criteria and design criteria determined in 1115 and 1116, respectively, to determine the task abstraction and design abstraction. 1121 may correspond to or be similar to 1030, 1035, and / or 1040. In 1122, the system queries an expert system / service to analyze the data abstraction and determine the type of visualization to be generated for the resulting data. In some embodiments, the expert system or service includes checking a set of rules against a subset of the data in the resulting data, scoring the rules, and selecting a visualization type corresponding to the highest-ranked / scored rule. For example, 1112 includes calling process 1200 in Figure 12. In response to determining the type of visualization to represent the result data in 1123, the system generates a visualization definition. The system may determine the visualization definition based at least partially on one or more of the query intent, task intent, and / or design intent. 1123 may correspond to or be similar to 1050. In some embodiments, the system generates a visualization based at least partially on the visualization definition. For example, the system selects an application or service to be called in connection with the generation of the visualization. In the example shown, the system selects whether to perform 1124 (e.g., generate the visualization using Plotly), 1125 (e.g., generate the visualization using D3.js), or 1126 (e.g., generate the visualization using ggplot2).Various other visualization applications or services can be implemented.

[0141] In some embodiments, 1115 and 1116 (e.g., determining task criteria and design criteria) may be implemented optionally. For example, the visualization may be generated based on the response data during data abstraction. The system may generate a visualization definition based on the data abstraction and then generate the visualization.

[0142] Figure 12 shows a process for determining the visualization type according to various embodiments. Process 1200 is performed by system 100 in Figure 1 and / or system 300 in Figure 3. In some embodiments, process 1200 corresponds to an expert system or service used to select the visualization type based at least partially on data abstraction. Steps 1225 and 1230 may be performed optionally. For example, steps 1205-1220 may be performed to select the visualization type, and steps 1225 and 1230 may be performed to generate the visualization and receive user feedback.

[0143] In step 1205, the system determines all possible ordered subsets of the data. In the illustrated example, the possible ordered subsets of the data include subsets: (i) A, (ii) B, (iii) C, (iv) AB, (v) BA, etc. The system retrieves the result data from the appropriate data source(s) based on the query intent and determines the possible ordered subsets within the result data.

[0144] In 1210, the system checks each rule against each subset of data within an ordered subset of data. The system stores a rulebook containing a given set of rules. A rule may be configured to return the value "true" if a subset of data matches the rule (for example, a visualization type may be implemented to display a subset of data) and the value "false" if a subset of data does not match the rule. In some implementations, rules return binary values ​​(e.g., 0 and 1) instead of "true" and "false". In the illustrated example, subsets A and B of the data are represented by bar rules (e.g., rules corresponding to horizontal bar chart visualization types), line rules (e.g., rules corresponding to line chart visualization types), donut rules (e.g., rules corresponding to donut visualization types), map rules (e.g., rules corresponding to map visualization types), sunburst rules (e.g., rules corresponding to sunburst visualization types), parallel set rules (e.g., rules corresponding to parallel set visualization types), pie chart rules (e.g., rules corresponding to pie chart visualization types), histograms (e.g., rules corresponding to histogram visualization types), and single-value (e.g., rules corresponding to single-value types). The visualization is checked against rules such as maps (for example, rules corresponding to map visualization types), bubble maps (for example, rules corresponding to bubble map visualization types), time bubble maps (for example, rules corresponding to time bubble map visualization types), pie maps (for example, rules corresponding to pie map visualization types), tree maps (for example, rules corresponding to tree visualization types), Sankey maps (for example, rules corresponding to Sankey visualization types), polar bar graphs (for example, rules corresponding to polar bar graph visualization types), scatter plots (for example, rules corresponding to scatter plot visualization types), and tables (for example, rules corresponding to table visualization types).Various other types of visualizations and corresponding rules for visualization types can be implemented.

[0145] In 1215, the system scores the rules on which a subset of data is checked / matched. In some embodiments, the system generates a score vector for the rules (e.g., for each rule). The score vector for each rule is determined based on scoring criteria for using a particular rule with respect to a particular ordered subset. Examples of scoring criteria include relevance (e.g., a measure of relevance to the use of the rule's visualization type on the resulting data), scope (e.g., a measure of scope to the use of the rule's visualization type on the resulting data), ornamentation (e.g., a measure of ornamentation or appeal to the use of the rule's visualization type on the resulting data), and user preference (e.g., a measure of user preference to the use of the rule's visualization type on the resulting data). Various other scoring criteria may be implemented. User preference may be determined / updated based on user feedback sought in connection with providing visualizations for particular natural language queries (e.g., feedback that may indicate the degree of preference for using a particular visualization type for data abstractions having certain characteristics).

[0146] In 1220, the system ranks the rules. In some embodiments, the system determines the ranking of the rules based on the score or score vector associated with each rule. For example, the system performs Pareto front optimization. The system selects rules corresponding to points along the Pareto front.

[0147] In some embodiments, the visualization type is at least partially based on the rule with the highest ranking or score. For example, the system determines the visualization type associated with a selected rule corresponding to a point on the Pareto front.

[0148] In 1225, the system generates visualizations based on selected rules. In some embodiments, the system provides multiple visualizations for a given natural language query. For example, different visualizations are provided that represent the resulting data in different ways.

[0149] In step 1230, the system obtains user feedback. The system may prompt the user to provide feedback for the visualization of the result data according to the selected visualization type. User feedback may be a binary like / dislike selection, such as for or against feedback. Alternatively, user feedback may include scoring of the visualization according to a given scale (for example, the user selects a value between 0 and 10 for their preference for using such visualization type for the result data). The system may iteratively / continuously update the user preference scores in the score vector based at least in part on the user feedback.

[0150] Figure 13 shows the process for generating visualizations according to various embodiments. Process 1300 is carried out by system 100 in Figure 1 and / or system 400 in Figure 4.

[0151] In 1305, a natural language query is received. In some embodiments, the system receives natural language queries based on user input to a user interface presented to a client system used by the user. The natural language query aims to request an answer to a specific question using operational data.

[0152] In 1310, a machine learning model is used to interpret a natural language query. In some embodiments, the system queries the LLM based on the natural language query. The LLM may interpret the query intent. Examples of query intents include query intents (e.g., instructions for data to respond to a natural language query), task intents, and / or design or context intents.

[0153] In 1315, the system obtains a query intent based on its interpretation of the natural language query using LLM. The system may determine one or more data requests (e.g., subqueries) to obtain result data from one or more data sources. In some embodiments, the system determines one or more data sources from which at least a subset of the result data will be obtained. The system may determine one or more data sources based at least partially on the query intent. The system generates one or more data requests based on the one or more data sources identified based on the query intent.

[0154] In 1320, the system retrieves result data from one or more data sources. For example, the system communicates one or more data requests and receives data from one or more data sources.

[0155] In 1325, the system obtains intents and context based on the interpretation of natural language queries using LLM. LLM can provide inferred intents and context. For example, inferred intents and context may include task intents and / or design intents.

[0156] In step 1330, the system determines the user preferences of the user associated with the natural language query (for example, the user who entered the natural language query, or the user to whom the requested visualization is provided).

[0157] In 1335, the system abstracts the resulting data in order to obtain a data abstraction. In some embodiments, obtaining a data abstraction involves determining one or more properties or statistics of the resulting data. Examples of properties / statistics include (i) the number of columns, (ii) the type of data in each column (e.g., category, numerical, date / time, etc.), and (iii) the distribution of numerical values.

[0158] In 1340, the system queries a machine learning model to provide a prediction. The prediction may include the predicted visualization type or the predicted visualization definition. The machine learning model provides the prediction based at least partially on the data abstraction. In some embodiments, the system queries the machine learning model based on the data abstraction, as well as inferred intents and contexts (e.g., task intent, design intent, etc.), and / or user preferences.

[0159] A machine learning model can be a Low-Level Machine Learning Model (LLM). A machine learning model may be the same as or different from an LLM used to interpret a natural language query. For example, a machine learning model that provides a predicted visualization type or visualization definition may be a lighter model compared to an LLM used to interpret a natural language query.

[0160] In step 1345, the system obtains a visualization definition. In some embodiments, the visualization definition is written in a predetermined data visualization language.

[0161] The visualization definition is obtained at least partially based on predictions from a machine learning model obtained in 1340. For example, the visualization definition is the predicted visualization definition output from the machine learning model. Alternatively, the system obtains the predicted visualization type as output from the machine learning model and generates a visualization definition according to a given data visualization language.

[0162] In step 1350, the system retrieves a visualization. The visualization is generated based at least partially on the visualization definition retrieved in step 1345. For example, the system queries another application or service to generate a visualization based at least partially on the visualization definition. As another example, the system generates a visualization based at least partially on the visualization definition. The system may translate the visualization definition from a given data visualization language to another language, such as a language determined based on the visualization type of the visualization to be generated.

[0163] Figure 14 is a flowchart of a method for generating visualizations based on natural language queries, according to various embodiments. Process 1400 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4.

[0164] In 1405, a natural language query is obtained. In 1410, the intent for the natural language query is determined. The intent may be a query intent that includes one or more of the data intent, task intent, and / or design intent. The intent for the natural language query is determined at least in part on analyzing the natural language query using a machine learning model (e.g., a large-scale language model such as ChatGPT). In 1415, one or more data requests are generated to one or more data sources. The system identifies the data source(s) to be queried based on the intent and, accordingly, generates requests to query those identified data source(s). In 1420, the resulting data is abstracted to obtain a data abstraction. In 1425, a visualization of the resulting data is generated at least in part on the data abstraction. In some embodiments, the system generates a visualization definition (e.g., in a data visualization language) based on the data abstraction. The system generates a visualization using the visualization definition. The system may directly generate a visualization based on the visualization definition, or the system may first convert the visualization definition from the data visualization language to another language (e.g., a high-level language which may be selected based on the type of visualization to be generated) and then generate an image using the converted visualization definition. At 1430, a decision is made as to whether process 1400 is complete. In some embodiments, process 1400 is determined to be complete in response to a decision that no further visualizations should be generated, no further queries are received regarding the data visualization, or an administrator indicates that process 1400 should be paused or stopped. In response to the decision that process 1400 is complete, process 1400 terminates. In response to a decision that process 1400 is not complete, process 1400 returns to 1405.

[0165] Figure 15 is a flowchart of a method for generating visualizations based on natural language queries, according to various embodiments. Process 1500 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4.

[0166] In 1505, a natural language query is obtained. In 1510, the intent of the natural language query is determined. 1510 may be the same as or identical to 1410 in process 1400 of Figure 14. In 1515, one or more data requests to one or more data sources are generated. 1515 may be the same as or identical to 1415 in process 1400 of Figure 14. In 1520, a predicted visualization definition is obtained, at least partially based on an abstraction of the resulting data. For example, the system first obtains a data abstraction for the resulting data, and then determines a predicted visualization definition based on the data abstraction. In some embodiments, determining a predicted visualization definition includes querying a prediction engine (e.g., a machine learning model) to predict the visualization definition of the visualization to be generated. In 1525, a visualization of the resulting data is generated, at least partially based on the predicted visualization definition. At 1530, a decision is made as to whether process 1500 is complete. In some embodiments, process 1500 is determined to be complete in response to a decision that no further visualizations should be generated, no further queries are received regarding the data visualization, or an administrator indicates that process 1500 should be paused or stopped. In response to the decision that process 1500 is complete, process 1500 terminates. In response to a decision that process 1500 is not complete, process 1500 returns to 1505.

[0167] Figure 16 is a flowchart of a method for determining the query intent of a natural language query according to various embodiments. Process 1600 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4. In some embodiments, process 1600 is invoked by process 1400 (e.g., in 1410) or by process 1500 (e.g., in 1510).

[0168] In 1605, the natural language query is retrieved. In some embodiments, the natural language query corresponds to a query entered by the user into the user interface. In 1610, the system queries the model for the query intent. The system queries the model based at least partially on the natural language query. For example, the system requests a machine learning model, such as a large language model, to interpret the natural language query. In 1615, the query intent for the natural language query is retrieved (e.g., from the model, or a service that provides the model being queried). In 1620, the query intent is provided. For example, the system provides the query intent to the system or service that called process 1600. In 1625, a decision is made as to whether process 1600 is complete. In some embodiments, process 1600 is determined to be complete in response to a decision that no further visualizations should be generated, no further queries will be received for the data visualization, no further queries should be analyzed, or an administrator indicates that process 1600 should be paused or stopped. In response to the decision that process 1600 is complete, process 1600 terminates. In response to the decision that process 1600 is not complete, process 1600 returns to 1605.

[0169] Figure 17 is a flowchart of a method for retrieving data in response to a query based on a query intent, according to various embodiments. Process 1700 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4. In some embodiments, process 1700 is invoked by process 1400 (e.g., at 1415) or by process 1500 (e.g., at 1515).

[0170] In 1705, a query intent is obtained. The query intent may be received in connection with a request to invoke process 1700 which should be executed. For example, the system may use process 1600 to determine the query intent and then invoke process 1700 based on the determined query intent. In 1710, a set of one or more data sources containing data that respond to the query intent is determined. In 1715, a data source is selected. The data source is selected from the set of one or more data sources determined to contain data that responds to the query intent. In 1720, the system generates a data request for the data stored in the selected data source. For example, the system generates a data request based at least partially on the stored data and the query intent. If the selected data source stores a subset of the data that responds to the query intent, the system determines the subset of data stored in the selected data source and generates a corresponding data request to retrieve the data. In 1725, the system determines whether another data request should be generated. For example, the system determines whether the set of data sources determined at 1710 includes another data source from which a request should be generated to obtain data in response to the query intent. In response to the determination that another data request should be generated (e.g., for a different data source), process 1700 returns to 1715 and repeats steps 1715-1725 until the system determines that no further data requests should be generated. In response to the determination that no further data requests should be generated, process 1700 proceeds to 1730. At 1730, the data request(s) are communicated. For example, the system communicates the data request(s) to their respective corresponding data sources (e.g., a data source in a set of one or more data sources from which at least a subset of the response data is retrieved). At 1735, the data in response to the data request(s) is retrieved.For example, the system retrieves the data in response to each data request generated to query the data source(s) in order to execute a query intent. In 1740, query data is provided in response to the query. For example, the system provides the query data to the system or service that called process 1700. In 1745, a decision is made as to whether process 1700 has completed. In some embodiments, process 1700 is determined to be complete in response to a decision that no further visualizations should be generated, no further data should be retrieved, no further queries are received regarding the data visualization, or an administrator indicates that process 1700 should be paused or stopped. In response to the decision that process 1700 has completed, process 1700 terminates. In response to a decision that process 1700 has not completed, process 1700 returns to 1705.

[0171] Figure 18 is a flowchart of a method for retrieving data in response to a query based on a query intent, according to various embodiments. Process 1800 is performed by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4. In some embodiments, process 1800 is invoked by process 1400 (e.g., in 1420) or by process 1500 (e.g., in connection with executing 1520).

[0172] In 1805, query data in response to the query is obtained. The query data in response to the query may be received in connection with a request to invoke process 1800 which should be executed. For example, the system may use process 1700 to determine the query data and then invoke process 1800 based on the determined query data. In 1810, an abstraction of the query data is performed in order to obtain a data abstraction. In some embodiments, obtaining a data abstraction includes determining one or more statistical properties of the resulting data. Determining one or more statistical properties of the resulting data includes analyzing the resulting data, which includes applying one or more predetermined rules to obtain one or more statistical properties. As an example, one or more statistical properties include one or more of columns, data within columns, outlier data, and numerical distributions. In 1815, the data abstraction is provided. For example, the system provides the data abstraction to the system or service that invoked process 1800. In 1820, a decision is made as to whether process 1800 has completed. In some embodiments, process 1800 is determined to be complete in response to a decision that no further visualizations should be generated, a decision that no further data should be abstracted, no further queries are received regarding the data visualization, or an administrator indicates that process 1800 should be paused or stopped. In response to the decision that process 1800 is complete, process 1800 terminates. In response to a decision that process 1800 is not complete, process 1800 returns to 1805.

[0173] Figure 19 is a flowchart of a method for determining visualization definitions for data abstraction according to various embodiments. Process 1900 is carried out by system 100 in Figure 1 and / or system 200 in Figure 2. In some embodiments, process 1900 is invoked by process 1400 (for example, in 1425 in relation to generating visualizations).

[0174] In 1905, the data abstraction is obtained. The data abstraction may be received in connection with a request to invoke process 1900, which should be executed. For example, the system may use process 1800 to determine the data abstraction and then invoke process 1900 based on the determined query data. In 1910, a lookup is performed in the rulebook for visualizations that match the data abstraction. The rulebook may contain a predetermined set of rules indicating the type of visualization to be generated based on the data abstraction (e.g., based on the characteristics / statistics of the data responding to the query). The rulebook may contain a hierarchy of predetermined sets of rules. The system may use the hierarchy to form a decision tree and walk through the rulebook to determine the visualization (e.g., the type of visualization to be generated). The system determines the type of visualization to be generated for the query or the data abstraction of the query, or the rules indicating the type of visualization. In 1915, the data abstraction is translated into a visualization definition in the data visualization language. The system determines that the visualization definition for the data visualization in response to the query is at least partially based on a rule (or visualization type) in a rulebook that matches the data abstraction. At 1920, the visualization definition is provided. For example, the system provides the visualization definition to the system or service that called process 1900. At 1925, a decision is made as to whether process 1900 has completed. In some embodiments, process 1900 is determined to be completed in response to a decision that no further visualization definitions should be determined, no further queries are received regarding the data visualization, or an administrator indicates that process 1900 should be paused or stopped. In response to the decision that process 1900 has completed, process 1900 terminates. In response to a decision that process 1900 has not completed, process 1900 returns to 1905.

[0175] Figure 20 is a flowchart of a method for determining visualization definitions for data abstraction according to various embodiments. Process 2000 is carried out by system 100 in Figure 1 and / or system 300 in Figure 3. In some embodiments, process 2000 is invoked by process 1400 (for example, in 1425 in relation to generating visualizations).

[0176] In 2005, a data abstraction is obtained. The data abstraction may be received in connection with a request to invoke a process 2000 to be executed. For example, the system may use process 1800 to determine the data abstraction and then invoke process 2000 based on the determined query data. In 2010, a type of visualization is selected based on a scoring of several types of visualizations for a particular data abstraction. In some embodiments, the type of visualization is selected at least in part on executing process 1200 (e.g., executing at least steps 1205-1220 of process 1200) or by invoking process 2100. In 2015, the data abstraction is translated into a visualization definition in the data visualization language, at least in part on the selected type of visualization. In 2020, the visualization definition is provided. For example, the system provides the visualization definition to the system or service that invoked process 2000. At 2025, a decision is made as to whether process 2000 is complete. In some embodiments, process 2000 is determined to be complete in response to a decision that no further visualization definitions should be determined, no further queries are received regarding the data visualization, or an administrator indicates that process 2000 should be paused or stopped. In response to the decision that process 2000 is complete, process 2000 terminates. In response to a decision that process 2000 is not complete, process 2000 returns to 2005.

[0177] Figure 21 is a flowchart of a method for determining the visualization type for data abstraction according to various embodiments. Process 2100 is carried out by system 100 in Figure 1 and / or system 300 in Figure 3. In some embodiments, process 2100 is invoked by process 2000 (e.g., in 2010).

[0178] In step 2105, all possible ordered subsets are obtained. The system obtains data in response to a query (e.g., data extracted from one or more data sources) and determines different possible subsets of the data. In step 2110, a set of rules is obtained. The system determines a set of rules in a given rulebook. The rulebook contains rules used to determine the type of visualization to be generated for a particular natural language query. In step 2115, each rule (e.g., a rule in the rulebook) is checked against each subset of the ordered subset. For example, checking a particular rule against the ordered subset involves analyzing the ordered subset to determine the result of the rule, for example, determining whether the rule is true or false (e.g., whether the ordered subset satisfies the rule). In step 2120, a score vector is created for each rule. The score vector can be obtained based on one or more scoring criteria. Examples of scoring criteria include relevance, scope, ornamentation, and user preference. Various other types of scoring criteria may be implemented. In step 2125, the type of visualization is selected based on the rule with the highest rank. The system may determine the ranking of a set of rules based on scoring criteria. For example, the rule with the highest aggregate score for a set of scoring criteria may be selected. In connection with determining the rule with the highest rank, the system may run a Pareto front optimization algorithm. In step 2130, the visualization type is provided. For example, the system provides the visualization type to the system or service that called process 2100 (e.g., to process 2000). In step 2135, a decision is made as to whether process 2100 has completed.In some embodiments, process 2100 is determined to be complete in response to a decision that no further visualization definitions should be determined, a decision that no further visualization types should be determined, no further queries are received regarding the data visualization, or an administrator indicates that process 2100 should be paused or stopped. In response to the decision that process 2100 is complete, process 2100 terminates. In response to a decision that process 2100 is not complete, process 2100 returns to 2105.

[0179] Figure 22 is a flowchart of a method for determining visualization definitions for data abstraction according to various embodiments. Process 2200 is carried out by system 100 in Figure 1 and / or system 400 in Figure 4. In some embodiments, process 2200 is invoked by process 1500 (for example, in 1520).

[0180] In 2205, a data abstraction is obtained. Obtaining a data abstraction may involve determining one or more properties / statistics associated with the query data (e.g., data responding to a query). The data abstraction may be received in connection with a request to invoke process 2200 to be executed. In 2210, the prediction engine is queried for the predicted visualization type, at least partially based on the data abstraction. In 2215, the data abstraction is translated into a visualization definition in a data visualization language (e.g., a given data visualization language), at least partially based on the predicted visualization type. In 2220, the visualization definition is provided. For example, the system provides the type of visualization to the system or service that invoked process 2200 (e.g., process 1500). In 2225, a decision is made as to whether process 2200 has completed. In some embodiments, process 2200 is determined to be complete in response to a decision that no further visualization definitions should be determined, no further queries are received regarding the data visualization, or an administrator indicates that process 2200 should be paused or stopped. In response to the decision that process 2200 is complete, process 2200 terminates. In response to a decision that process 2200 is not complete, process 2200 returns to 2205.

[0181] Process 2200 describes querying a prediction engine (e.g., a machine learning model) for a predicted visualization type, but various embodiments may implement querying a prediction engine for a predicted visualization definition. For example, a model may predict a specific predicted visualization for a particular query based on a data abstraction. The model may predict a visualization type and a specific visualization definition for visualizing the data in response to the query according to the predicted visualization type.

[0182] Figure 23 is a flowchart of a method for translating a visualization definition into another language, according to various embodiments. Process 2300 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4. In some embodiments, process 2300 is invoked by process 1400 (e.g., 1425) and / or process 1500 (e.g., 1525).

[0183] In step 2305, the visualization definition is obtained. The visualization definition is specified in a data visualization language. The visualization definition may be received in connection with a request to invoke process 2400 to be executed. For example, the visualization definition may be defined / generated in process 1900 (e.g., 1920), process 2000 (e.g., 2020), and / or process 2100 (e.g., 2130). In step 2310, the system decides whether to translate the visualization definition into a different language. For example, the system decides whether a different high-level language should be used to generate the visualization, and in response to deciding that a different high-level language should be used, the system decides to translate the visualization definition. In response to deciding in step 2310 to translate the visualization definition, process 2300 proceeds to step 2315. Conversely, in response to the decision in 2310 that the visualization definition should not be translated, process 2300 proceeds to 2335. In 2315, the target language for the visualization definition is selected (e.g., determined). The target language for the visualization definition may be determined at least in part on the type of visualization (e.g., certain types of visualizations may require a specific language). Examples of languages ​​that may be used in connection with generating various types of visualizations include D3, R, and Python. In 2320, the visualization definition is translated into the selected language. In 2325, the corresponding visualization definition is provided. In 2335, a decision is made as to whether process 2300 is complete. In some embodiments, process 2300 is determined to be complete in response to a decision that no further visualization definitions should be determined, an indication from the administrator that process 2300 should be paused or stopped, etc. In response to the decision that process 2300 is complete, process 2300 terminates.In response to the determination that process 2300 is not complete, process 2300 returns to 2305.

[0184] In some embodiments, the machine learning model used to interpret the intent of a natural language query (e.g., a large-scale language model) is different from the machine learning model used to predict the visualization definition. For example, the machine learning model used to predict the visualization definition may be smaller / lighter than the model used to predict the query intent.

[0185] Figure 24 is a flowchart of a method for training a model according to various embodiments. Process 2400 is carried out by system 100 in Figure 1, system 200 in Figure 2, system 300 in Figure 3, and / or system 400 in Figure 4. Process 2400 may be carried out in connection with generating a model for predicting visualization definitions. A similar process may be carried out to train a model for predicting query intents.

[0186] In 2405, a training set is acquired that includes samples of data abstractions and corresponding visualization definitions. In 2410, a model is trained to determine visualization definitions based on specific data abstractions. The model may be configured to predict visualization definitions. In some embodiments, the model is a machine learning model trained according to a machine learning process. For example, the model is a large-scale language model that interprets visualization definitions in the training set. In 2415, the model is deployed. In 2420, user feedback is received. The system may provide user feedback in response to visualizations being provided in response to natural language queries. User feedback may include information indicating alternative visualization types requested / desired by the user. In some embodiments, process 2400 proceeds to 2425 after a threshold number of user feedback records have been received. In 2425, the system decides whether to retrain the model. The system may decide to retrain the model based on the user feedback. For example, the system may decide to retrain the model after receiving a predetermined number of user feedback records for a corresponding visualization. In response to the decision to retrain the model, the process proceeds to 2430, and the model is retrained. Then, process 2400 returns to 2415, and process 2400 repeats steps 2415-2430 until it is decided that the model will not be retrained. Conversely, in response to the decision that the model will not be retrained, process 2400 proceeds to 2435. At 2435, a decision is made as to whether process 2400 is complete. In some embodiments, process 2400 is determined to be complete in response to a decision that no further models should be determined / trained (e.g., no further classification models should be created), or when an administrator indicates that process 2400 should be paused or stopped.In response to the decision that process 2400 is complete, process 2400 terminates. In response to the decision that process 2400 is not complete, process 2400 returns to 2405.

[0187] Various examples of embodiments described herein are illustrated with reference to flowcharts. While examples may include specific steps performed in a particular order, according to various embodiments, the steps may be performed in various orders, and / or the steps may be combined into a single step or in parallel.

[0188] While the embodiments described above are explained in some detail for the purpose of clarifying understanding, the present invention is not limited to the details provided. There are many alternative ways of carrying out the present invention. The disclosed embodiments are illustrative and not limiting.

Claims

1. It is a system for visualizing data. Obtaining natural language queries and, Determining the intent of the aforementioned natural language query, To generate one or more data requests to one or more selected data sources, wherein the one or more data requests are at least partially based on the intent. The process of abstracting result data in order to obtain data abstraction, wherein the result data responds to one or more data requests, and the abstraction is performed. To generate visualizations of the resulting data based at least partially on the aforementioned data abstraction. One or more processors configured to perform the following: A memory coupled to the one or more processors and configured to provide instructions to the one or more processors A system that includes these features.

2. The system according to claim 1, wherein determining the intent of the natural language query includes querying a large-scale language model based on the natural language query.

3. The system according to claim 1, wherein the intent is determined at least in part on a machine learning model.

4. The system according to claim 1, wherein abstracting the resulting data in order to obtain the data abstraction includes determining one or more statistical properties relating to the resulting data.

5. The system according to claim 4, wherein the one or more statistical properties include one or more of columns, data within the columns, outlier data, and numerical distributions.

6. The system according to claim 4, wherein the data abstraction is determined at least in part on one or more statistical properties.

7. The system according to claim 1, wherein generating the visualization of the resulting data based at least in part on the data abstraction includes determining the type of visualization based at least in part on the data abstraction and creating the visualization based at least in part on the type of visualization.

8. The system according to claim 1, wherein the data abstraction corresponds to the representation of the natural language query in accordance with a predetermined data visualization language.

9. The system according to claim 8, wherein the representation in accordance with the predetermined data visualization language includes an indication of a first dimension of the data to be visualized, a second dimension of the data to be visualized, and a type of visualization.

10. The system according to claim 1, wherein the data abstraction indicates the type of visualization to be generated.

11. The system according to claim 10, wherein the type of visualization is selected from a predetermined set of visualizations, the predetermined set of visualizations includes one or more of the following: bar graphs, line graphs, donut visualizations, map visualizations, bubble charts, sunburst representations, pie charts, histograms, single-value maps, bubble maps, time bubble maps, pie maps, tree maps, Sankey charts, polar bar graphs, scatter plots, tables, and parallel set representations.

12. In order to obtain the aforementioned data abstraction, abstracting the result data is Regarding the aforementioned result data, determine all possible ordered subsets of the data, For each rule within a given set of rules, analyze each subset of data using a specific rule to obtain the corresponding rule check, For each rule in the predetermined set of rules, create a score vector that includes the corresponding rule check on a subset of the data, Ranking the rules within the predetermined set of rules based on the corresponding score vectors, Selecting the data abstraction based on the ranking of the aforementioned rules The system according to claim 1, including the following:

13. The system according to claim 12, wherein creating the score vector includes scoring the fit of a type of visualization for a particular rule for a subset of the data, the fit is scored according to a predetermined scoring criterion.

14. A method for visualizing data, The steps include obtaining a natural language query using one or more processors, The steps include determining the intent of the aforementioned natural language query, A step of generating one or more data requests to one or more selected data sources, wherein the one or more data requests are at least partially based on the intent, A step of abstracting result data in order to obtain data abstraction, wherein the result data responds to one or more data requests, A step of generating a visualization of the resulting data based at least partially on the aforementioned data abstraction. A method that includes this.

15. A computer program product embodied in a non-temporary, computer-readable medium for visualizing data, One or more processors retrieve natural language queries, Determining the intent of the aforementioned natural language query, To generate one or more data requests to one or more selected data sources, wherein the one or more data requests are at least partially based on the intent. The process of abstracting result data in order to obtain data abstraction, wherein the result data responds to one or more data requests, and the abstraction is performed. To generate visualizations of the resulting data based at least partially on the aforementioned data abstraction. A computer program product that includes computer instructions for performing a specific task.

Images