Automatically generating data visualizations and information maps using large language models and diffusion models

Abstract

Systems and methods are provided for generating visualized data associated with raw data using a machine learning model. For example, a machine learning model may automatically generate candidate analyses and / or scene sets for visualizing raw data based on summary data. Given summary data and answers to cues used to visualize the data, the generated candidate analysis may reflect the context of the original data expected by the user. A visualization code scaffold according to a visualization specification may be used to generate program outputs corresponding to candidate analyses, which may thus be used to generate a visualization accordingly. In some examples, an information graph may also be generated using a diffusion model based on the visualizations and cues.

Description

Background Art

[0001] As the use and availability of data continue to increase, techniques involving visualizing data have become increasingly popular. In fact, such visualization increases the accessibility of data. As more data becomes available, creating visualizations of the data for viewers to effectively understand the data has become more complex. Some data is more substantial than other data in conveying data trends. Depending on the context of the information in the visualized data, different types of data become more useful than other data. Designing visualizations that are informative, interesting, and persuasive to viewers involves a great deal of effort and skill, which may limit the use of such visualizations and / or may result in visualizations being uninformative and / or unpersuasive, among other detriments.

[0002] Aspects disclosed herein have been formed with these and other general considerations in mind. Additionally, although relatively specific issues may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified elsewhere in the background art or in this disclosure. Summary of the Invention

[0003] Aspects of the present disclosure relate to a system for using a language model to generate visualizations of raw data. Examples can generate summary data of the raw data. The summary data can thus represent a compressed representation of the raw data and / or can add semantic types to the raw data. In some examples, the summary data also includes summary statistics associated with the raw data. The summary data can indicate fields of some of the data to be extracted for generating the visualization. Given the summary data, the disclosed techniques generate a visualization scenario for generating a visualization of the raw data. The visualization scenario represents a scenario that specifies the type of visual representation of the raw data, one or more fields to be extracted for rendering relevant portions of the raw data for visualization, and the context associated with the raw data. Specifically, embodiments use a language model to automatically generate a visualization scenario for visualizing the raw data. A user can have a level of control in the generation of the visualization scenario or can otherwise provide a visualization scenario used by aspects of the present disclosure. In an example, generating the visualization scenario also includes generating a prompt and receiving an answer to the prompt from the user for use in generating the visualization scenario. For example, the prompt includes requesting the user to identify one or more data fields that are more relevant when visualizing the raw data from the user's perspective. The language model thus incorporates the answer into generating the visualization scenario.

[0004] Given a visualization scenario, the disclosed techniques automatically generate a visualization code scaffold for generating a visualization specification. A generative machine learning model that can generate program output (e.g., which may be the case when the model is a multimodal generative model in addition to natural language output) is used to generate the visualization code scaffold. For example, a machine learning model can generate program output that specifies the type and / or color of a graphical representation as specified by the visualization scenario for the visualization of data. Examples can also include validating and / or filtering portions of the program output to remove errors when rendering the visualization. In some aspects, the disclosed techniques use a language model that is not limited to a generative machine learning model to generate the visualization code scaffold. Additionally or alternatively, the disclosed techniques can generate a visualization code scaffold based on existing visualization code scaffolds by retrieving existing visualization code scaffolds from various sources, including a database of example visualization code data, from user input, etc.

[0005] In some examples, the present disclosure uses a text-to-image generation model to generate an infographic from the visualization results of data. Examples of text-to-image generation models include, but are not limited to, diffusion models. A diffusion model can generate image data based on given image data (e.g., a visualization result in two or more dimensions) and a prompt (e.g., text data). In an example, the prompt also specifies the context and features for the generated infographic. Thus, the diffusion model can select or otherwise generate an infographic according to one or more artistic styles, where the one or more artistic styles have a color scheme or color palette, or any of various other artistic adaptations of the visualization.

[0006] The present invention content is provided to introduce a selection of concepts in a simplified form, which will be further described in the following detailed description. The present invention content is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be partly set forth in the following description, and partly will be obvious from the description, or can be learned through the practice of the present disclosure. Brief Description of the Drawings

[0007] Non-limiting and non-exhaustive examples are described with reference to the following drawings.

[0008] Figure 1 An overview of an example system for generating a visualization for an infographic according to aspects of the present disclosure is shown.

[0009] Figure 2 Examples of summary data and visualization scenario data for generating a visualization for an infographic according to aspects of the present disclosure are shown.

[0010] Figure 3Shows examples of a visualization code scaffolding and examples of visualization code for generating visualizations of infographics according to aspects of the present disclosure.

[0011] Figure 4 Shows an example of a method for generating visualizations of infographics according to aspects of the present disclosure.

[0012] Figure 5A And Figure 5B Shows an overview of an example generative machine learning model that can be used according to aspects of the present disclosure.

[0013] Figure 6 Shows an example of a computing device on which aspects of the present disclosure can be practiced.

[0014] Figure 7 Is another simplified block diagram of a computing device on which aspects of the present disclosure can be practiced. Detailed Description

[0015] There is an increasing number of cases of data visualization for various reasons. For example, devices (e.g., sensors as part of the Internet of Things) generate large amounts of data, and the semantics / statistics associated with the collected data will be communicated to users in a meaningful way. Visualization makes this and other data accessible to viewers by summarizing insights within large data collections. In particular, infographics have become a popular means of communicating information to users via graphics that represent data in a concise manner for intuitive understanding by viewers.

[0016] However, problems arise in generating visualizations of data and infographics when the user is not familiar with or does not have the skills to create visuals that are meaningful and persuasive to viewers based on a data set. For example, a data set can take any of a variety of forms, including but not limited to a table with rows and columns. It is often difficult and confusing for a user to understand the meaning of the columns and rows in such a table. As such, a user may have difficulty deriving meaning from the data, determining the context of the data, and / or determining which data is relatively more relevant for visualization (e.g., communicating data trends and / or semantics to a viewer).

[0017] Traditional systems aimed at helping users generate visualizations of data are based on some different methods. Some traditional systems automate data visualization based on predefined heuristics. However, developing a heuristic-based system typically involves manual work and is time-consuming.

[0018] The data analysis system disclosed below automatically generates visualizations of data in various fields and is robust, while enabling users to interactively confirm and control the generation process. For example, automatically generating various contexts that can be considered from a data set for visualization for the user to select can reduce the user's burden. Additionally, automatically summarizing the input data set and providing a summary of the data set for interactively receiving user feedback to determine the context associated with the data set can similarly improve the user experience when visualizing the data set. Based on the summary data, the system can automatically generate a visualization scenario (e.g., a scenario for selectively utilizing some parts of the data set relevant in that context) and use the visualization scenario to provide a prompt to the user. After receiving user confirmation and / or correction of the visualization scenario, the system can generate a visualization of the data accordingly.

[0019] As discussed in more detail below, the present disclosure relates to automatically generating visualizations of data. Specifically, examples can use generative machine learning models to process natural language input and perform various tasks accordingly. Various tasks include, but are not limited to, translation between languages, summarizing data, completing sentences, generating prompts, and predicting answers and / or generating instructions and code in a programming language (also referred to herein as "program output"). By adding more parameters and data to the model, the machine learning model is scalable in its performance. Additionally, the model can be used to predict reasonable results based on a limited or reduced number of examples. In some examples, the machine learning model can be fine-tuned to a target area of generating data. Examples of the target area include generating program output for one or more programming languages, answering questions, generating data based on a text-to-image model (e.g., a diffusion model), etc.

[0020] The automatic generation of visualizations according to the present disclosure is not limited to graphical representations of data. Additionally or alternatively, the disclosed techniques can generate infographics based on visualizations using declarative artificial intelligence (e.g., including diffusion models or other text-to-image models). Infographics can include specially designed images to persuasively convey information to viewers.

[0021] Aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which show specific example aspects as a part hereof. However, the different aspects of the present disclosure can be implemented in many different ways and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that the present disclosure will be thorough and complete and will fully convey the scope of these aspects to those skilled in the art. The aspects can be practiced as a method, system, or apparatus. Accordingly, the aspects can take the form of a hardware implementation, a fully software implementation, or an implementation combining software and hardware aspects. Thus, the following detailed description should not be taken in a limiting sense.

[0022] Figure 1 FIG. 0 shows an overview of an example system for generating visualizations of infographics according to aspects of the present disclosure. System 100 includes a client device 102 and a visualization generator 104 interconnected via a network 108. In an example, the client device 102 generates a collection of data (e.g., raw data) by collecting data and integrating the data into a collection. The client device 102 may include sensors for generating sensor data as the collection of raw data for visualization.

[0023] The visualization generator 104 generates visualizations of the raw data. Additionally or alternatively, the visualization generator 104 generates infographic data based on the visualization of the data. The visualization generator 104 includes a raw data receiver 110, a data profiler 112, a visualization scene generator 114, a visualization data generator 116, a visualization data transmitter 118, an infographic data generator 120, and an infographic data transmitter 122. In an example, the visualization generator 104 uses raw data 130, profile data 132, visualization scene data 134, visualization code scaffolding 136, visualization data 150, infographic data 152, a profile model 140, a visualization scene model 142, a code generation model 144, and / or a diffusion model 146.

[0024] The raw data receiver 110 receives the raw data 130 from the client device 102 via the network 108, for example. The raw data 130 can be in various forms that can be profiled for generating visualizations. Examples of the raw data 130 include, but are not limited to, one or more tables formed by one or more columns and one or more rows of data. In an example, the raw data 130 can be more than two dimensions. In aspects, the raw data 130 can include numerical data and / or text data. The raw data 130 can represent statistical data. The raw data 130 can represent data values measured by counters and sensors over time and / or at different locations.

[0025] The data profiler 112 generates profile data 132 based on the collection of the raw data 130. In aspects, the data profiler 112 uses the profile model 140 to automatically generate the profile data 132. In an example, the data profile model 140 receives the collection of the raw data as input and generates the profile data 132 in natural language. The profile model 140 determines the context associated with the collection of the raw data 130. In some aspects, the data profiler 112 transmits the profile data 132 to the client device 102 to interactively present the profile data 132 and receive feedback and / or corrections to the profile data 132.

[0026] In various aspects, the summary data 132 represents a compressed representation of the original data set. The compression can be based on various methods, including but not limited to multiple fields, semantic types associated with the original data set, and summary statistics including mean and / or median, maximum and / or minimum. In another example, for visualizing data, one or more data regions may be more useful than another data region. The data profiler 112 can use a neural network to infer one or more parts of the original data set. The data profiler 112 can output the summary data 132 and context data characterizing the summary data 132. In various aspects, the context data is used to generate a title for the visualization.

[0027] The visualization scene generator 114 generates a visualization scene based on the summary data 132 for generating a visualization of the set of original data 130. In various aspects, the visualization scene represents a strategy and / or scenario for generating a visualization of the original data set based on a combination of context and summary data associated with the original data set (e.g., as generated by the data profiler 112 and / or as has been received from a user of the client device 102). For example, the visualization scene indicates correlations, relationships, insights within the data, and / or visualization scenarios, such as which column or columns of tabular original data will be used to generate the visualization.

[0028] Accordingly, the visualization scene can specify data relevant to generating the visualization. Some data columns can be used to plot along the axes of a multi-dimensional graph. Some other data columns can be represented in different colors in the visualization. The visualization scene generator 114 uses the visualization scene model 142. The visualization scene model 142 generates a visualization scene based on a combination of the original data 130 and text summary data (e.g., the summary data 132). The visualization scene model 142 reduces the amount of data for subsequent processing and assigns different types to the columns of the original data set to simplify the visualization. The visualization scene model 142 can be based on a machine learning model (e.g., a large language model, a generative model, etc.), such that the visualization scene can be expressed in natural language (e.g., Docstring data type).

[0029] In various aspects, the visualization scene generator 114 generates a natural language output that includes a descriptive scene for visualizing data. The descriptive scene for visualizing data can include one or more hypotheses to be visualized by a visualization generated according to the aspects described herein. For example, a hypothesis can indicate a question related to the visualizing data and / or to the relationship between a first variable of the data and a second variable of the data, such that the visualization result will accordingly plot or otherwise display the data (e.g., based on the first variable and the second variable). It should be understood that in other examples, any number of variables can be used. Additionally, in other examples, a user can provide a natural language input that includes such a hypothesis. In some aspects, the visualization scene generator 114 can output a list of scene data that describes the visualizing data. In some examples, a machine learning model generates and poses questions such that the user can provide an input (e.g., a revision to the question and / or an answer to the question) to guide the system to generate a descriptive scene that is in line with the user's Figure 1 intent.

[0030] The visualization data generator 116 generates instructions for programming a visualization according to one or more visualization specifications. Specifically, the visualization data generator 116 uses a machine learning model (e.g., a multimodal generative model) to generate code scaffold stubs. In an example, the natural language text includes visualization scene data 134. In various aspects, the code scaffold specifies one or more graphics libraries to import into the programming code for rendering the visualization specified by the visualization scene.

[0031] In various aspects, the visualization scene generator 114 generates one or more prompting words and presents them to the user for confirming and / or correcting the visualization scene (e.g., via the client device 102). The prompting words can be expressed in natural language. The user can respond to the prompting words by inputting a natural language response. In other examples, instead of selecting or editing a proposed visualization scene from the visualization generator 104, the user can provide a new visualization scene. The code scaffold stubs can include example code for creating the visualization and a textual description of the scaffold (e.g., a Docstring), which describes the intent of the visualization and how the code integrates information from the data. For example, the Docstring can indicate "Show a scatter plot of horsepower and miles per gallon for various cars", followed by programming code that defines the scatter plot and specifies the data types for the corresponding axes and colors for plotting the data. In an example, the code scaffold includes the visualization scene confirmed by the user and / or code that the user can edit or further refine for rendering the visualization data 150.

[0032] In various aspects, the Visualization Data Generator 116 uses the Code Generation Model 144. The Code Generation Model 144 can generate programming code (e.g., using a multimodal generative model), which can be a declarative visualization language for rendering visualizations. As an example, the multimodal generative model uses a visualization scenario (e.g., described in natural language) to generate programming code for rendering visualizations. In various aspects, the Visualization Data Generator 116 can use one or more visualization libraries in various programming languages. The Visualization Data Generator 116 generates a visualization code scaffold as output.

[0033] In various aspects, the Visualization Data Generator 116 generates multiple visualization code scaffolds. The Visualization Data Transmitter 118 transmits the generated visualization data 150 to the client device 102 for the user to view. The generated visualization data 150 can include a rendered image of the visualization. For example, the client device 102 can display a scatter plot specified by the visualization code. The user can use the client device 102 to interactively select and / or modify the code based on user commands. In an example, the Visualization Data Generator 116 iteratively validates and filters the visualization code to remove compilation and other errors in the visualization code.

[0034] In additional aspects, the Visualization Data Generator 116 can iteratively generate code scaffolds with a high temperature until a predetermined criterion for generating a visualization is met. Examples of the predetermined criterion include a threshold (e.g., five) for extracting non-error visualizations. The Visualization Data Generator 116 can insert code and values into the visualization code scaffold such that the final visualization code complies with the specification for declarative visualizations. The Visualization Data Generator 116 accordingly outputs the visualization data 150.

[0035] In some examples, the Infographic Data Generator 120 generates an infographic based on a visualization of the original data set. An infographic is a graphical representation of data that persuasively conveys information to a viewer. Different from the visualization of the original data set, an infographic can more artistically convey specific aspects or characteristics to a viewer. In various aspects, generating an infographic includes converting the visualization into a form with a certain predetermined focus and / or combining the visualization with a description of the intent.

[0036] The infographic data generator 120 receives the visualization data 150 rendered by executing the visualization code and generates infographic data 152 representing the infographic. In various aspects, the infographic data generator 120 uses the diffusion model 146 to process the visualization data 150 and accordingly generate the infographic data 152. For example, the diffusion model 146 receives the rendered image data of the visualization data 150 and uses a prompt (e.g., as may be provided by a user of the client device 102), where the prompt indicates one or more artistic styles that the diffusion model 146 selects for presenting the infographic to convey information. For example, the prompt may state that the visualization chart should be in shades of green and purple. Accordingly, the diffusion model 146 generates the infographic data 152, which includes the shades of green and purple of the scatter plot in the visualization data 150. In some other examples, the prompt may indicate a visualization chart that attracts some specific viewing audience (e.g., teenagers, the elderly, etc.) for the diffusion model to adapt an artistic style that attracts the intended audience. The example system uses the diffusion model as an example of a text-to-image model and is not limited to the diffusion model. Other models that transform data types (e.g., text-to-image) may be used.

[0037] Figure 2 An example of summary data and visualization scenario data for generating visualizations of infographics according to aspects of the present disclosure is shown. The summary data 202 may indicate a summary of the data fields of the original data to be used for visualization. In various aspects, the summary data 202 includes the title of the data table (e.g., "Horsepower and Miles per Gallon of Cars"); the number of columns in the data table (e.g., three columns); and the number of rows in the data table (e.g., 120 rows). The summary data 202 may also include the title of the first column (e.g., "Index of Cars"); the title of the second column (e.g., "Horsepower") and the title of the third column (e.g., "Miles per Gallon"). In various aspects, the data summarizer (e.g., the data summarizer 112 as Figure 1 shown) generates the summary data based on the original data (e.g., the original data 130 as Figure 1 shown) by using a machine learning model (e.g., the summary model 140 as Figure 1 shown).

[0038] The visualization scenario data 204 includes text data describing how the data appears after being visualized. In some aspects, the visualization scenario data 204 may indicate "Show a scatter plot of the horsepower and miles per gallon of various cars". In various aspects, the visualization scenario generator (e.g., the visualization scenario generator 114 as Figure 1 shown) may use a machine learning model (e.g., the summary model 140 as Figure 1The visualization scenario model shown (142) outputs visualization scenario data 204 by receiving summary data 202 as input. Thus, the visualization scenario data 204 provides one or more suggested visualizations from which a user can select, according to aspects described herein, for generating a visualization. For example, the visualization scenario data 204 specifies that the visualization data take the form of a scatter plot, where horsepower and miles per gallon are the titles of the x-axis and y-axis, respectively. In some aspects, a machine learning model can generate a list of questions associated with the visualization data by receiving the summary data as input. By interactively inputting answers by the user to the list of questions, the machine learning model can also generate, accordingly, visualization scenario data based on the user's answers (which can thus ultimately be used to generate a corresponding visualization).

[0039] Figure 3 An example of a visualization code scaffold according to aspects of the present disclosure and an example of visualization code for generating an infographic visualization are shown. In aspects, the visualization code scaffold 302 includes instructions for importing one or more libraries and instructions for calling a function that generates visualization code to visualize data. A visualization data generator (e.g., the visualization data generator 116 shown as Figure 1 shown) can receive visualization scenario data (e.g., the visualization scenario data 134 shown as Figure 1 shown; the visualization scenario data 204 shown as Figure 2 shown) as input and generate the visualization code scaffold 302. For example, the visualization code scaffold 302 includes code for importing libraries, code for calling a method that generates visualization data by specifying the visualization scenario data and calling a command to generate visualization code using a machine learning model. In aspects, the visualization code scaffold 302 can be in the form of a programming language (e.g., the Python language), which integrates information from visualization scenario data associated with various fields and types of data charts (e.g., scatter plots).

[0040] The visualization code 304 includes a set of code for visualizing data. In aspects, the visualization code 304 includes instructions for importing libraries and instructions for generating a chart (e.g., a scatter plot) that plots the data. Examples of instructions for generating the chart can set values required for plotting the data (e.g., the title of the x-axis (e.g., "Horsepower"), the title of the y-axis (e.g., "MPG"), the color for plotting (e.g., "Source"), and tooltip data for the chart (e.g., "Name", "Source", "Horsepower", and "MPG"). In some aspects, a visualization data generator (e.g., as Figure 1The visualization data generator 116) shown generates visualization code 304 by compiling or otherwise processing a visualization code scaffold 302. In some other aspects, the visualization code can be edited interactively by a user.

[0041] Figure 4 An example of a method 400 for automatically generating visualizations by using a large language model is provided. The general order of operations of the method 400 in Figure 4 is provided. Generally, method 400 begins with a start operation 402. Method 400 can include more or fewer steps, or can arrange the order of steps differently from Figure 4 the order shown. Method 400 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. Additionally, method 400 can be executed by gates or circuits associated with a processor, ASIC, FPGA, SOC, or other hardware device. Hereinafter, method 400 will be explained with reference to Figures 1 to 3 and the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc. described in Figure 7 and FIGS. 5 to

[0042] After start operation 402, method 400 begins with a receive operation 404, wherein a collection of raw data is received. In various aspects, the collection of raw data includes a collection of data in a tabular structure. The values of the data can be based on data generated by sensors, survey data (e.g., raw answer data from a survey), historical log data, etc.

[0043] At a generate summary data operation 406, summary data (e.g., summary data 132 as Figure 1 shown) is generated based on the collection of raw data. The generate summary data operation 406 includes generating a compressed representation of the collection of raw data, for example, by replacing selected columns and fields of the data with summary statistical data. Examples of summary statistical data can include the mean / median, maximum, minimum, and / or selection of one or more regions of tabular data. The data types in the summary data can include numeric and / or text data. The summary data operation 406 can infer semantic types to add to columns. In some examples, a machine learning model can be used for inference. The generate summary data operation 406 determines the context associated with the collection of raw data and extracts data relevant in depicting the context. Additionally or alternatively, the generate summary data operation 406 includes interactively receiving from a user one or more corrections to summary candidate data and generating summary data by updating the summary candidate data.

[0044] At operation 408 of generating visualization scene data, visualization scene data is generated based on the contextualized summary data. As described above, the visualization scene data may describe a strategy for generating a visualization. In various aspects, operation 408 of generating visualization scene data uses a machine learning model to generate a list of questions that a user may be interested in, in order to generate a visualization from the original data set. For example, the machine learning model may use the summary data and context (e.g., as may have been generated at summary data operation 406) to generate a natural language output that includes a descriptive scenario for visualizing the data. The descriptive scenario for visualizing the data may include one or more hypotheses about how the data is predicted to behave when visualized. Thus, the user may receive a list of scene data that describes the scenarios for visualizing the data. In some examples, the machine learning model generates and poses questions such that the user can provide input (e.g., revisions to the questions and / or answers to the questions) to guide the system to generate a descriptive scenario that aligns with the user's Figure 1 intentions.

[0045] In various aspects, the visualization scene may specify the number of data columns to be used, the visualization technique to be used for generating the visualization (e.g., the type of graph), and / or which data to use to optimize the visualization of the original data set according to the context, among other examples.

[0046] In some aspects, operation 408 of generating visualization scene data may include generating a set of questions for the user to answer. In an example, the list of questions includes questions that ask about aspects of the data in the original data set that are more relevant compared to other data, such as which fields seem more useful to the user for visualization. Thus, operation 408 of generating visualization scene data may use the answers to the list of questions received from the user to determine whether one or more fields of the data are more relevant than another field based on the relevance to the answers. Such interactivity enables the user to confirm that the visualization scene conforms to the user's intentions, and thus enables the system to generate the visualization that the user expects.

[0047] At the code scaffolding stub generation operation 410, one or more visual code scaffolding stubs are generated based on the visual scene data. The code scaffolding is generated by using a multimodal machine learning model to generate program output based on a text description (e.g., such as the visual scene generated above). In various aspects, the multimodal machine learning model receives visual scene data (e.g., text data, image data, or other forms of data) as input data and outputs the code scaffolding in text form. In an example, the code scaffolding includes a set of instructions for importing one or more libraries for rendering the visualization of the data. It should be understood that in some examples, at least a portion of the code scaffolding is provided by the user and / or retrieved from a storage inventory of the code scaffolding, and other examples are also included. Thus, as used herein, the code scaffolding may not need to be generated by a generative model, or may be partially generated by a generative model, and other examples are also included. The code scaffolding may conform to a declarative visualization specification. In some examples, the code scaffolding includes the visual scene in the Docstring, which describes the context of the visualization to be generated.

[0048] At the visual code generation operation 412, visual code for rendering one or more visualizations (e.g., visual data) is generated. In an example, the multimodal generative model may also generate additional program code that conforms to a predetermined visualization specification (e.g., a user including one or more visual code libraries). The visual code generation operation 412 may include iteratively validating and filtering one or more portions of the visual code, e.g., to remove program code that causes compilation errors or results in unexpected behavior, etc. The iteration may continue until a predetermined condition for error removal is met.

[0049] At the visual data generation operation 414, visual data corresponding to the visualization of the original data set is generated. In an example, the visual data may be rendered as raster image data. For example, as described above, the visual data generation operation 414 may include executing the program code generated by the multimodal machine learning model at operation 412.

[0050] At the information graphic data generation operation 416, an information graphic is generated based on the visualization data generated at operation 414. In various aspects, the information graphic data generation operation 416 uses a diffusion model to generate the information graphic. For example, the diffusion model receives the rendered image data of the visualization and a prompt for generating the information graphic. The prompt can describe the type of the information graphic and / or the context associated with the information graphic. For example, the diffusion model can select or otherwise determine one or more artistic styles to process the visualization data to generate the information graphic accordingly. The method 400 ends at the end operation 418. The diffusion model is an example of a text-to-image model. Other types of text-to-image models or models that generate image data based on one or more types of data as input can be used to generate the information graphic.

[0051] It should be understood that operations 402-418 are described for the purpose of illustrating the method and system and are not intended to limit the present disclosure to a particular sequence of steps. For example, the steps can be performed in a different order, additional steps can be performed, and the disclosed steps can be excluded without departing from the present disclosure.

[0052] Figure 5A and 5B shows an overview of an example generative machine learning model that can be used in accordance with aspects described herein. First referring to Figure 5A , the conceptual diagram 500 depicts an overview of a pre-trained generative model package 504 in accordance with aspects described herein, which processes an input and a prompt 502 to generate a model output for a visualization hypothesis 506. Examples of the pre-trained generative model package 504 include but are not limited to the Megatron-Turing Natural Language Generation model (MT-NLG), Generative Pretrained Transformer 3 (GPT-3), Generative Pretrained Transformer 4 (GPT-4), BigScience BLOOM (Big Open Science Open Access Multilingual Language Model), DALL-E, DALL-E2, Stable Diffusion, or Jukebox.

[0053] In an example, the generative model package 504 is pre-trained based on various inputs (e.g., various human languages, various programming languages, and / or various content types), and thus does not need to be fine-tuned or trained for a specific scenario. Instead, the generative model package 504 can be pre-trained more generally such that the input 502 includes a prompt word that is generated, selected, or otherwise designed to induce the generative model package 504 to produce certain generative model outputs 506. For example, the prompt word includes context and / or one or more completion prefixes and thus pre-loads the generative model package 504 accordingly. Thus, the generative model package 504 is induced to generate an output based on the prompt word, which includes a predicted sequence of tokens related to the prompt word (e.g., up to the token limit of the generative model package 504). In an example, the predicted token sequence is further processed (e.g., via output decoding 516) to produce the output 506. For example, each token is processed to identify the corresponding word, word fragment, or other content that forms at least a part of the output 506. It should be understood that the input 502 and the generative model output 506 can each include any one of various content types, including but not limited to text output, image output, audio output, video output, program output, and / or binary output, etc. In an example, the input 502 and the generative model output 506 can have different content types, such as may be the case when the generative model package 504 includes a generative multimodal machine learning model).

[0054] Thus, the generative model package 504 can be used in any one of a variety of scenarios, and further, different generative model packages can be used in place of the generative model package 504 with substantially no modification to other associated aspects (e.g., similar to those aspects described herein with respect to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 6 , Figure 7 and Figure 8). Thus, the generative model package 504 operates as a tool for performing machine learning processing, where certain inputs 502 to the generative model package 504 are programmatically generated or otherwise determined such that the generative model package 504 produces a model output 506, which can then be used for further processing.

[0055] The generative model package 504 can be provided or otherwise used according to any one of a variety of paradigms. For example, the generative model package 504 can be on a computing device (e.g., Figure 1It can be used locally at the visualization generator 104) in, or can be remotely accessed from a machine learning service (e.g., client device 102). In other examples, aspects of the generative model package 504 are distributed across multiple computing devices. In some instances, the generative model package 504 can be accessed via an application programming interface (API), such as provided by the operating system of a computing device and / or by a machine learning service and other examples.

[0056] Now referring to the illustrated aspects of the generative model package 504, the generative model package 504 includes input tokenization 508, input embedding 510, model layer 512, output layer 514, and output decoding 516. In an example, the input tokenization 508 processes the input 502 to generate the input embedding 510, which includes a sequence of symbolic representations corresponding to the input 502. Correspondingly, the input embedding 510 is processed by the model layer 512, the output layer 514, and the output decoding 516 to produce the model output 506. Figure 5B An example architecture corresponding to the generative model package 504 is depicted in, which is further discussed in detail below. Even so, it should be understood that the architectures shown and described herein should not be considered restrictive, and in other examples, any of a variety of other architectures can be used.

[0057] Figure 5B is a conceptual diagram depicting an example architecture 550 of a pre-trained generative machine learning model that can be used according to the aspects described herein. As described above, without departing from the aspects described herein, any of a variety of alternative architectures and corresponding ML models can be used in other examples.

[0058] As shown, the architecture 550 processes the input 502 to produce a generative model output 506, aspects of which were discussed above with respect to Figure 5A The architecture 550 is depicted as a transformer model including an encoder 552 and a decoder 554. The encoder 552 processes the input embedding 558 (aspects of which can be similar to Figure 5A the input embedding 510 in), which includes a sequence of symbolic representations corresponding to the input 556. In an example, the input 556 includes the input for the visualization problem and the prompt words 502, aspects of which can be similar to the raw data 130 received by the raw data receiver 110, and / or the prompt words generated based on the raw data 130 according to the aspects described herein.

[0059] In addition, the positional encoding 560 can introduce information about the relative and / or absolute positions of the tokens of the input embedding 558. Similarly, the output embedding 574 includes a sequence of symbolic representations corresponding to the output 572, and the positional encoding 576 can similarly introduce information about the relative and / or absolute positions of the tokens of the output embedding 574.

[0060] As shown in the figure, the encoder 552 includes an example layer 570. It should be understood that any number of such layers can be used, and it should be understood that the depicted architecture is simplified for illustrative purposes. The example layer 570 includes two sub-layers: a multi-head attention layer 562 and a feed-forward layer 566. In the example, residual connections are included around each layer 562, 566, followed by normalization layers 564 and 568 respectively.

[0061] The decoder 554 includes an example layer 590. Similar to the encoder 552, any number of such layers can be used in other instances, and the depicted architecture of the decoder 554 is simplified for illustrative purposes. As shown in the figure, the example layer 590 includes three sub-layers: a masked multi-head attention layer 578, a multi-head attention layer 582, and a feed-forward layer 586. Aspects of the multi-head attention layer 582 and the feed-forward layer 586 can be similar to those discussed above with respect to the multi-head attention layer 562 and the feed-forward layer 566 respectively. Additionally, the masked multi-head attention layer 578 performs multi-head attention on the output of the encoder 552 (e.g., output 572). In the example, the masked multi-head attention layer 578 prevents certain positions from attending to subsequent positions. This masking combined with the offset embedding (e.g., offset by one position, as shown in the multi-head attention layer 582) can ensure that the prediction for a given position depends on the known outputs for one or more positions less than the given position. As shown in the figure, residual connections are also included around the layers 578, 582, and 586, followed by normalization layers 580, 584, and 588 respectively.

[0062] The multi-head attention layers 562, 578, and 582 can each linearly project queries, keys, and values to corresponding dimensions using a set of linear projections. An attention function (e.g., dot product or additive attention) can be used to process each linear projection, thereby producing an n-dimensional output value for each linear projection. The resulting values can be concatenated and projected again such that the values are then processed as Figure 5B shown (e.g., by the corresponding normalization layers 564, 580, or 584).

[0063] The feed-forward layers 566 and 586 can each be fully-connected feed-forward networks applied to each position. In the example, the feed-forward layers 566 and 586 each include a plurality of linear transformations with rectified linear unit activations therebetween. In the example, each linear transformation is the same at different positions, while each linear transformation can use different parameters compared to other linear transformations of the feed-forward network.

[0064] Additionally, aspects of the linear transformation 592 can be similar to the linear transformations discussed above with respect to the multi-head attention layers 562, 578, and 582, and the feed-forward layers 566 and 586. The Softmax 594 can also convert the output of the linear transformation 592 into the probabilities of the predicted next token, as shown by the output probabilities 596. It should be understood that the illustrated architecture is provided as an example, and in other examples, any of a variety of other model architectures can be used in accordance with the disclosed aspects. In some cases, multiple iterations of processing are performed in accordance with the above aspects (e.g., using the generative model package 504 in Figure 5A or the encoder 552 and decoder 554 in Figure 5B ) to generate a series of output tokens (e.g., words), which, for example, will be combined later to produce a complete sentence (and / or any of a variety of other things). It should be understood that other generative models can generate multiple output tokens in a single iteration, and thus iterations with a reduced number of iterations or a single iteration can be used.

[0065] Thus, the output probabilities 596 can, in accordance with the aspects described herein, shape the output 506 into summary data for visualization, such that the output of the generative ML model (e.g., which can include structured output) is used as input for subsequent steps of generating visualization scene data and code scaffolding stubs for further generating visualization code in accordance with the aspects described herein (e.g., similar to the generate visualization scene data operation 408 and the generate code scaffolding stub operation 410 in Figure 3 ).

[0066] Figure 6 is a block diagram showing the physical components (e.g., hardware) of a computing device 600 that can practice the aspects of the present disclosure. The computing device components described below can be applicable to the computing device described above. In a basic configuration, the computing device 600 can include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of the computing device, the system memory 604 can include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of these memories. The system memory 604 can include an operating system 605 and one or more program tools 606 suitable for executing the various aspects disclosed herein. The operating system 605 can, for example, be suitable for controlling the operation of the computing device 600. Additionally, the aspects of the present disclosure can be practiced in conjunction with a graphics library, other operating systems, or any other application, and are not limited to any particular application or system. This basic configuration is shown in Figure 6Those components within the dashed line 608 are shown. The computing device 600 may have additional features or functionality. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or magnetic tapes. Such additional storage is Figure 6 shown by the removable storage device 609 and the non-removable storage device 610 in

[0067] As described above, a plurality of program tools and data files may be stored in the system memory 604. When executed on at least one processing unit 602, the program tool 606 (e.g., the application 620) may perform processes including but not limited to the aspects described herein. The application 620 includes, as Figure 1 more detailedly described in, a data profiler 630, a scene generator 632, a visual data generator 634, and an infographic data generator 636. Other program tools that may be used in accordance with aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, and the like.

[0068] In addition, aspects of the present disclosure may be practiced in a circuit including discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. For example, aspects of the present disclosure may be practiced via a system-on-chip (SOC), where Figure 6 each or many of the components shown in are integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functionalities, all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality regarding the client switching protocol described herein may be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 600. Aspects of the present disclosure may also be practiced using other technologies capable of performing logical operations, such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. Additionally, aspects of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0069] The computing device 600 may also have one or more input devices 612, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. Output devices 614 may also be included, such as a display, speakers, printer, etc. The above devices are examples, and other devices may be used. The computing device 600 may include one or more communication connections 616 that allow communication with other computing devices 650. Examples of communication connections 616 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0070] As used herein, the term computer-readable medium may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and that can be accessed by the computing device 600. Any such computer storage media may be part of the computing device 600. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0071] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal having one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0072] Figure 7 Shown is a computing device (e.g., a client device 102 as Figure 1 shown) and a server (e.g., as Figure 1An example block diagram of an aspect of the architecture of a visualization generator 104), mobile computing device, etc. That is, the computing device 700 can be combined with the system 702 (e.g., system architecture) to implement some aspects. The system 702 can be implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 702 is integrated as a computing device, such as an integrated digital assistant (PDA) and wireless phone.

[0073] One or more application programs 766 can be loaded into the memory 762 and run on or in association with the operating system 764. Examples of application programs include a phone dialer, email program, personal information management (PIM) program, word processing program, spreadsheet program, Internet browser program, messaging program, etc. The system 702 also includes a non-volatile storage area 768 within the memory 762. The non-volatile storage area 768 can be used to store persistent information that should not be lost if the system 702 loses power. The application programs 766 can use and store information in the non-volatile storage area 768, such as emails or other messages used by an email application. A synchronization application (not shown) also resides on the system 702 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 768 synchronized with the corresponding information stored at the host computer. It should be understood that other applications can be loaded into the memory 762 and run on the computing device 700 described herein.

[0074] The system 702 has a power supply 770, which can be implemented as one or more batteries. The power supply 770 can also include an external power supply, such as an AC adapter or a charging docking cradle that supplements or recharges the battery.

[0075] The system 702 can also include a radio interface layer 772 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 772 supports a wireless connection between the system 702 and the "outside world" via a communication carrier or service provider. Transmissions to and from the radio interface layer 772 are under the control of the operating system 764. In other words, communications received by the radio interface layer 772 can be propagated to the application programs 766 via the operating system 764, and vice versa.

[0076] A visual indicator 720 (e.g., an LED) can be used to provide visual notifications, and / or an audio interface 774 can be used to generate audible notifications via an audio transducer 725. In the illustrated configuration, the visual indicator 720 is a light-emitting diode (LED), and the audio transducer 725 is a speaker. These devices can be directly coupled to a power source 770 such that when activated, they remain on for the duration indicated by the notification mechanism even if the processor 760 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes an action to indicate the powered-on state of the device. The audio interface 774 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 774 can also be coupled to a microphone to receive audible input, such as to support a phone conversation. According to aspects of the present disclosure, the microphone can also be used as an audio sensor to support control of notifications, as will be described below. The system 702 can also include a video interface 776, which enables operations of devices connected to the peripheral device port 730 to record still images, video streams, etc.

[0077] The computing device 700 implementing the system 702 can have additional features or functionality. For example, the computing device 700 can also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or magnetic tapes. Such additional storage is Figure 7 shown by the non-volatile storage area 768.

[0078] As described above, the data / information generated or captured by the computing device 700 and stored via the system 702 can be stored locally on the computing device 700, or the data can be stored on any number of storage media that can be accessed by the device via the radio interface layer 772 or via a wired connection between the computing device 700 and a separate computing device associated with the computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via the computing device 700 via the radio interface layer 772 or via a distributed computing network. Similarly, such data / information can be easily transmitted between computing devices for storage and use according to well-known data / information transmission and storage means, including email and collaborative data / information sharing systems.

[0079] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure in any way. The claimed disclosure should not be construed as limited to any aspect or detail provided in this application, for example. Various features (structures and methods), whether shown and described in combination or separately, are intended to be selectively included or omitted to produce embodiments with a particular set of features. Given the description and illustration of this application, those skilled in the art can envision variations, modifications, and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application and that do not depart from the broader scope of the claimed disclosure.

[0080] The present disclosure relates to systems and methods for generating visualizations of data using large language models at least according to the examples provided in the following sections. The method includes: receiving a collection of raw data; generating summary data based on the collection of raw data using a first machine learning model; generating one or more candidate analyses for the collection of raw data using a second machine learning model, the one or more candidate analyses specifying contexts associated with the visualization of the collection of raw data; using a third machine learning model to generate a visualization code scaffold based on the one or more candidate analyses and the collection of raw data, wherein the visualization code scaffold includes at least a portion of the candidate analysis as text data and programming code for rendering at least a portion of the collection of raw data according to the summary data; generating visualization data associated with the collection of raw data using the visualization programming code generated from the visualization code scaffold; and providing the visualization data for display to a user. The method further includes generating infographic data using a diffusion model based on the visualization data and a prompt. The prompt is received from the user's computing device. One or more candidate analyses are generated based on the summary data. The method further includes receiving a user selection of a candidate analysis from among the one or more candidate analyses generated for the collection of raw data. The third machine learning model is a multimodal generative machine learning model. The summary data includes a compressed representation of the collection of raw data.

[0081] In another aspect, a system for generating visual data based on a data set is provided. The system includes a memory; and a processor configured to perform steps including: receiving an original data set; generating summary data based on the original data set using a first machine learning model; generating one or more candidate analyses for the original data set using a second machine learning model, the one or more candidate analyses specifying contexts associated with the visualization of the original data set; generating visualization program code based on a visualization code scaffold, the one or more candidate analyses, and the original data set, and using a third machine learning model, the visualization program code being for rendering at least a portion of the original data set according to the summary data; generating visual data associated with the original data set using the visualization program code; and providing the visual data for display to a user. The processor is further configured to perform steps including: generating infographic data using a diffusion model based on the visual data and a prompt. Receiving the prompt from the user's computing device. Generating one or more candidate analyses based on the summary data. The processor is further configured to perform steps including: receiving a user selection of a candidate analysis from among the one or more candidate analyses generated for the original data set. The third machine learning model is a multimodal generative machine learning model. The summary data includes a compressed representation of the original data set.

[0082] In yet another aspect, a device for generating visual data based on a data set is provided. The device includes a memory; a processor configured to perform a method including: receiving an original data set; generating summary data based on the original data set using a first machine learning model; generating one or more candidate analyses for the original data set using a second machine learning model, the one or more candidate analyses specifying contexts associated with the visualization of the original data set; generating visualization program code based on a visualization code scaffold, one or more candidate analyses, and the original data set, and using a third machine learning model, the visualization program code being for rendering at least a portion of the original data set according to the summary data; generating visual data associated with the original data set using the visualization program code; and providing the visual data for display to a user. The processor is further configured to perform a method including: generating infographic data using a diffusion model based on the visual data and a prompt. Receiving the prompt from the user's computing device. Generating one or more candidate analyses based on the summary data. The processor is further configured to perform a method including: receiving a user selection of a candidate analysis from among the one or more candidate analyses generated for the original data set. The third machine learning model is a multimodal generative machine learning model, and wherein the summary data includes a compressed representation of the original data set.

[0083] Any aspect of the one or more aspects described above is combined with any other aspect of the one or more aspects. Any aspect of the one or more aspects as described herein.

Claims

1. A method for generating visualization data based on a data set, comprising: Receiving an original data set; Generating summary data using a first machine learning model based on the original data set; Using a second machine learning model to generate one or more candidate analyses for the original data set, the one or more candidate analyses specifying contexts associated with the visualization of the original data set; Based on a visualization code scaffold, the one or more candidate analyses, and the original data set, and using a third machine learning model, generating visualization program code for rendering at least a portion of the original data set according to the summary data; Using the visualization program code to generate visualization data associated with the original data set; and Providing the visualization data for display to a user.

2. The method according to claim 1, further comprising: Generating infographic data using a diffusion model based on the visualization data and a prompt.

3. The method according to claim 1, wherein the third machine learning model is a multimodal generative machine learning model.

4. A system for generating visualization data based on a data set, the system comprising: A memory; And A processor configured to perform steps including: Receiving an original data set; Generating summary data using a first machine learning model based on the original data set; Using a second machine learning model to generate one or more candidate analyses for the original data set, the one or more candidate analyses specifying contexts associated with the visualization of the original data set; Based on a visualization code scaffold, the one or more candidate analyses, and the original data set, and using a third machine learning model, generating visualization program code for rendering at least a portion of the original data set according to the summary data; Using the visualization program code to generate visualization data associated with the original data set; and Providing the visualization data for display to a user.

5. The system according to claim 4, the processor is further configured to perform steps including: Generating infographic data using a diffusion model based on the visualization data and a prompt.

6. The system according to claim 4, the processor is further configured to perform steps including: Receiving a user selection of a candidate analysis among the one or more candidate analyses generated for the original data set.

7. A device for generating visualization data based on a data set, the device comprising: A memory; A processor configured to execute a method, the method comprising: Receiving an original data set; Generating summary data using a first machine learning model based on the original data set; Using a second machine learning model to generate one or more candidate analyses for the original data set, the one or more candidate analyses specifying contexts associated with the visualization of the original data set; Based on the visual code scaffolding, the one or more candidate analyses, and the original data set, and using a third machine learning model, generate visual program code for rendering at least a portion of the original data set according to the summary data; Using the visual program code, generate visual data associated with the original data set; and Provide the visual data for display to a user.

8. The apparatus according to claim 7, wherein the processor is further configured to execute a method comprising: Based on the visual data and a prompt, generate infographic data using a diffusion model.

9. The apparatus according to claim 8, wherein the prompt is received from the user's computing device.

10. The apparatus according to claim 7, wherein the one or more candidate analyses are generated based on the summary data.

11. The method according to claim 2, wherein the prompt is received from the user's computing device.

12. The method according to claim 1, wherein the summary data includes a compressed representation of the original data set.

13. The system according to claim 4, wherein the one or more candidate analyses are generated based on the summary data.

14. The system according to claim 4, wherein the third machine learning model is a multimodal generative machine learning model.

15. The apparatus according to claim 7, wherein the processor is further configured to execute a method comprising: Receive a user selection of a candidate analysis from among the one or more candidate analyses generated for the original data set.

Images