COMPUTER SYSTEM AND METHOD FOR PREPARING DATA FOR SUBSEQUENT ANALYSIS, AND COMPUTER-READABLE NON-TRANSITORY STORAGE MEDIUM

Abstract

This is a computer system that displays a user interface including a flow panel, a tool panel, a profile panel, and a data panel. The flow panel displays a flowchart of nodes / links that identifies data sources, operations, and output datasets. The tool panel includes a data source selector that allows users to add data sources to the flowchart, and includes an operations palette that allows users to insert nodes into the flowchart to perform specific transformation operations. The profile panel displays schemas corresponding to selected nodes in the flowchart, including information about data fields and statistical information about data values ​​for the data fields, and allows users to modify the flowchart by interacting with individual data elements.The data panel displays rows of data corresponding to the nodes selected in the flowchart, and allows users to modify the flowchart by interacting with individual data values.

Description

1 / 88 “COMPUTER SYSTEM AND METHOD FOR PREPARING DATA FOR SUBSEQUENT ANALYSIS, AND COMPUTER-READABLE NON-TRANSITORY STORAGE MEDIUM” TECHNICAL FIELD

[001] The implementations disclosed relate, in general, to data visualization, and more specifically, to systems, methods and user interfaces for preparing and curating data for use by a data visualization application. BACKGROUND OF THE INVENTION

[002] Data visualization applications allow a user to visually understand a dataset, including distribution, trends, discrepancies, and other factors that are important in business decision-making. Some datasets are very large or complex, and include many data fields. Various tools can be used to help understand and analyze the data, including dashboards that can have multiple data visualizations. However, data often needs to be manipulated or processed to put it into a format that can be easily used by data visualization applications. Sometimes, multiple ETL (Extract / Transform / Load) tools are used to build usable data sources.

[003] There are two dominant models in the ETL and data preparation space today. Data flow-style systems focus the user on operations and the flow of data through the system, which helps provide clarity on the overall task structure and makes it simpler for the user to control these steps. However, these systems generally do not perform well in the function of showing the user their actual data, which can make it difficult for users to actually understand what their data is or what needs to be done with it. These systems Petition 870250077578, dated 01 / 09 / 2025, page 8 / 202 2 / 88 can also suffer from a knot explosion. When every small operation gets its own node in a diagram, even a moderately complex flow can become a confusing tangle of nodes and edges.

[004] On the other hand, “potter’s wheel” style systems present the user with a concrete spreadsheet-style interface to their actual data, and allow the user to sculpt their data through direct actions. Although users are actually authorizing a data flow in these systems, this flow is usually occluded, making it difficult for the user to understand and control the overall structure of their task. SUMMARY

[005] The disclosed implementations have features that offer the benefits of both data flow-style systems and "potter's wheel"-style systems, and go even further, making it even easier for the user to build a data flow. The disclosed data preparation applications describe data flows, but gather the nodes into larger groups that better represent the high-level actions that users want to perform. The design of these nodes uses direct action on the actual data, guided by relevant statistics and visualizations at each step.

[006] According to some implementations, a computer system prepares data for analysis. The computer system includes one or more processors, memory, and one or more programs stored in memory. The programs are configured for execution by one or more processors. The programs display a user interface for a data preparation application. The user interface includes a data flow panel, a tools panel, a profile panel, and a data panel. The data flow panel displays a flowchart of nodes / links that identifies data sources, operations, and output datasets. The tools panel includes a data source selector. Petition 870250077578, dated 01 / 09 / 2025, page 9 / 202 3 / 88 data that allows users to add data sources to the flowchart, includes an operations palette that allows users to insert nodes into the flowchart to perform specific transformation operations, and a palette of other flowcharts that a user can incorporate into the flowchart. The profile panel displays schemas corresponding to selected nodes in the flowchart, including information about data fields and statistical information about data values ​​for the data fields, and allows users to modify the flowchart by interacting with individual data elements. The data panel displays rows of data corresponding to the selected nodes in the flowchart, and allows users to modify the flowchart by interacting with individual data values.

[007] In some implementations, the information about the data fields displayed in the profile panel includes data ranges for a first data field.

[008] In some implementations, in response to an initial user action on a first data range for the first data field in the profile panel, a new node is added to the flowchart, which filters the data for the first data range.

[009] In some implementations, the profile panel allows users to map data ranges to the first data field to specified values, thereby adding a new node to the flowchart that performs the user-specified mapping.

[010] In some implementations, in response to a user's first interaction with a first data value in the data panel, a node is added to the flowchart, which filters the data for the first data value.

[011] In some implementations, in response to a user-made modification of a first data value in a first data field in Petition 870250077578, dated 01 / 09 / 2025, page 10 / 202 4 / 88 data panel, a new node is added to the flowchart, which performs the modification in each row of data whose data value for the first data field equals the first data value.

[012] In some implementations, in response to an initial user action on a first data field in the data panel, a node is added to the flowchart, which splits the first data field into two or more separate data fields.

[013] In some implementations, in response to a first user action in the data flow panel of dragging a first node to the tool panel, a new operation is added to the operations palette, the new operation corresponding to the first node.

[014] In some implementations, the profile panel and the data panel are configured to update asynchronously as selections are made in the data flow panel.

[015] In some implementations, the information about the data fields displayed in the profile panel includes one or more histograms that display distributions of data values ​​for data fields.

[016] According to some implementations, a process transforms data. The process is performed on a computing system equipped with a display medium, one or more processors, and memory storing one or more programs configured for execution by the one or more processors. The process displays a user interface that includes a data flow panel and a data panel. The process receives initial user input to construct a data transformation flowchart of nodes / links in the data flow panel. Each node in the flowchart specifies a respective operation to retrieve data from a respective data source, specifies a respective operation to transform data, or specifies a respective operation to create a respective Petition 870250077578, dated 01 / 09 / 2025, p. 11 / 202 5 / 88 Output dataset. The flowchart includes a subtree having one or more data source nodes that retrieve data from a first data source and one or more transformation operation nodes. The process receives a second user input to execute at least the subtree. Based on the second user input and a determination that the nodes in the subtree are configured to execute imperatively, the process performs the operations on the nodes in the subtree sequentially as specified by the links in the subtree, thereby retrieving data from the first data source, transforming the retrieved data, forming a first intermediate dataset, and displaying the first intermediate dataset in the data pane. The process receives a third user input to configure the nodes in the subtree to execute declaratively and receives a fourth user input to execute at least the subtree.Based on the user's fourth input and a determination that the nodes in the subtree are configured to execute declaratively, the process constructs a database query that is logically equivalent to the operations specified by the nodes in the subtree, transmits the database query to the first data source to retrieve a second intermediate dataset from the first data source according to the database query, and displays the second intermediate dataset in the data pane.

[017] According to some implementations, the process includes storing the first intermediate dataset and the second intermediate dataset.

[018] In some examples, the subtree is the entire flowchart.

[019] According to some implementations, one of the transformation operation nodes specifies a filter transformation operation to filter data rows received by the node, perform node operations on the subtree Petition 870250077578, dated 01 / 09 / 2025, page 12 / 202 6 / 88 sequentially includes performing the filter transformation operation on the computer system to filter the received data rows, and retrieving the second intermediate data set from the first data source according to the database query includes applying the filter operation on a remote server hosting the first data source.

[020] According to some implementations, one of the transformation operation nodes specifies a union transformation operation that joins two datasets from the first data source, performing the node operations in the subtree sequentially includes performing the union transformation operation to combine the two datasets in the computer system, and retrieving the second intermediate dataset from the first data source according to the database query includes applying the union operation on a remote server hosting the first data source.

[021] According to some implementations, the database query is written in SQL.

[022] According to some implementations, the flowchart includes a part not included in the subtree, the part is configured to execute imperatively, the fourth user input specifies the execution of the entire flowchart, and the execution of the flowchart includes performing the node operations in the part sequentially as specified by the links in the part, thus accessing the second intermediate dataset, transforming the second intermediate dataset, and forming a final dataset.

[023] According to some implementations, a process refactors a flowchart. The process is performed on a computer system equipped with a display medium, one or more processors, and memory storing one or more programs configured for execution by the one or more processors. The process includes displaying a user interface that includes a plurality of panels, including Petition 870250077578, dated 01 / 09 / 2025, page 13 / 202 7 / 88 a data flow panel and a palette panel. The data flow panel includes a flowchart with multiple existing nodes, each node specifying a respective operation to retrieve data from a respective data source, specifying a respective operation to transform data, or specifying a respective operation to create a respective output dataset. Additionally, the palette panel includes multiple flow element templates.The process further includes receiving initial user input to select an existing node from the flowchart or a flow element template from the palette panel, and in response to the first user input: (i) displaying a movable icon representing a new node for placement in the flowchart, where the new node specifies a data flow operation corresponding to the selected existing node or the selected flow element template, and (iii) displaying one or more drop points in the flowchart according to the dependencies between the new node's data flow operation and the operations of the plurality of existing nodes. The process further includes receiving second user input to place the movable icon over a first drop point of the drop points, and suspending the detection of the second user input. In response to suspending the detection of the second user input, the process inserts the new node into the flowchart at the first drop point.The new node performs the specified data flow operation.

[024] According to some implementations, each of the existing nodes has a respective intermediate dataset calculated according to the respective operation specified, and inserting the new node into the flowchart at the first drop point includes calculating an intermediate dataset for the new node according to the specified data flow operation.

[025] According to some implementations, the new node is placed in the flowchart after an existing first node has a first set of data. Petition 870250077578, dated 01 / 09 / 2025, page 14 / 202 8 / 88 intermediate datasets, and calculating the intermediate dataset for the new node includes applying the data flow operation to the first intermediate dataset.

[026] According to some implementations, the new node has no predecessor in the flowchart, and calculating the intermediate dataset for the new node involves retrieving data from a data source to form the intermediate dataset.

[027] According to some implementations, the process additionally includes, in response to the suspension of the detection of the second user input, displaying a sample of data from the intermediate dataset in a data panel of the user interface. The data panel is one of a plurality of panels.

[028] According to some implementations, the data flow operation filters rows of data based on values ​​of a first data field, and displaying one or more drop points includes displaying one or more drop points immediately after existing nodes whose intermediate datasets include the first data field.

[029] According to some implementations, the user's first input selects an existing node from the flowchart, and inserting the new node into the flowchart at the first drop point creates a copy of the existing node.

[030] According to some implementations, inserting the new node in the flowchart at the first drop point additionally includes removing the existing node from the flowchart.

[031] According to some implementations, the data flow operation includes a plurality of operations that are executed in a specified sequence.

[032] According to some implementations, a method is executed on an electronic device with a display medium. For example, the electronic device could be a smartphone, a tablet, a notebook, a computer, or a Petition 870250077578, dated 01 / 09 / 2025, page 15 / 202 9 / 88 desktop computer. The method implements any of the computer systems described here.

[033] In some implementations, a non-temporary, computer-readable storage medium stores one or more programs configured for execution by a computer system containing one or more processors, memory, and a display medium. The one or more programs include instructions for implementing a system that prepares data for analysis, as described herein.

[034] In this way, methods, systems and graphical user interfaces are revealed that allow users to analyze, prepare and curate data, as well as refactor existing flows and data. BRIEF DESCRIPTION OF THE DRAWINGS

[035] For a better understanding of the aforementioned systems, methods and graphical user interfaces, as well as additional systems, methods and graphical user interfaces that offer data visualization analytics and data preparation, reference should be made to the following Implementation Description, together with the following drawings, in which similar reference numerals refer to corresponding parts throughout the figures.

[036] Figure 1 illustrates a graphical user interface used in some implementations.

[037] Figure 2 is a block diagram of a computing device according to some implementations.

[038] Figures 3A and 3B illustrate user interfaces for a data preparation application according to some implementations.

[039] Figure 3C describes some aspects of the user interfaces illustrated in Figures 3A and 3B.

[040] Figure 3D illustrates a sample flow diagram according to some implementations. Petition 870250077578, dated 01 / 09 / 2025, page 16 / 202 10 / 88

[041] Figure 3E illustrates a pair of streams that work together but are executed at different frequencies, according to some implementations.

[042] Figures 4A to 4V illustrate the use of a data preparation application to construct a union according to some implementations.

[043] Figure 5A illustrates a part of a log file according to some implementations.

[044] Figure 5B illustrates a part of a lookup table according to some implementations.

[045] Figures 6A to 6C illustrate some operations, inputs and outputs for a flow, according to some implementations.

[046] Figures 7A and 7B illustrate some components of a data preparation system, according to some implementations.

[047] Figure 7C illustrates the evaluation of a flow, both for analysis and execution, according to some implementations.

[048] Figure 7D schematically represents an asynchronous subsystem used in some data preparation implementations.

[049] Figure 8A illustrates a sequence of flow operations according to some implementations.

[050] Figure 8B illustrates three aspects of a type system according to some implementations.

[051] Figure 8C illustrates properties of a type environment according to some implementations.

[052] Figure 8D illustrates simple type checking based on a stream with all known data types, according to some implementations.

[053] Figure 8E illustrates a simple type fault with fully known types, according to some implementations.

[054] Figure 8F illustrates simple type environment calculations for a flow Petition 870250077578, dated 01 / 09 / 2025, page 17 / 202 11 / 88 partial, according to some implementations.

[055] Figure 8G illustrates types of a packaged container node, according to some implementations.

[056] Figure 8H illustrates a more complicated type environment scenario, according to some implementations.

[057] Figure 8I illustrates the reuse of a more complex type environment scenario, according to some implementations.

[058] Figures 8J-1, 8J-2, and 8J-3 show the properties for many of the most frequently used operators, according to some implementations.

[059] Figures 8K and 8L illustrate a corresponding flow and execution process, according to some implementations.

[060] Figure 8M illustrates that the execution of an entire flow begins with implicit physical models in the input and output nodes, according to some implementations.

[061] Figure 8N illustrates that the execution of a partial flow materializes a physical model with the results, according to some implementations.

[062] Figure 8O illustrates the execution of part of a flow based on previous results, according to some implementations.

[063] Figures 8P and 8Q illustrate the evaluation of a flow with an affixed node 860, according to some implementations.

[064] Figure 9 illustrates a part of a flowchart according to some implementations.

[065] Reference will now be made to the implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are presented in order to allow a thorough understanding of the present invention. However, it will be clear to those with Petition 870250077578, dated 01 / 09 / 2025, page 18 / 202 12 / 88 general knowledge in the art that the present invention can be practiced without needing these specific details. DESCRIPTION OF THE IMPLEMENTATIONS

[066] Figure 1 illustrates a graphical user interface 100 for interactive data analysis. The user interface 100 includes a Data tab 114, and an Analytics tab 116 according to some implementations. When the Data tab 114 is selected, the user interface 100 displays a schema information region 110, which is also called a data panel. The schema information region 110 provides named data elements (e.g., field names) that can be selected and used to construct a data visualization. In some implementations, the list of field names is separated into a dimension group (e.g., categorical data) and a measure group (e.g., numeric quantities). Some implementations also include a list of parameters. When the Analytics tab 116 is selected, the user interface displays a list of analytical functions instead of the data elements (not illustrated).

[067] The graphical user interface 100 also includes a data visualization region 112. The data visualization region 112 includes a plurality of shelf regions, such as a column shelf region 120 and a row shelf region 122. These are also called column shelf 120 and row shelf 122. As illustrated here, the data visualization region 112 also has a large space to display a visual graphic. Since no data element has been selected so far, the space initially has no visual graphic. In some implementations, the data visualization region 112 has multiple layers that are called sheets.

[068] Figure 2 is a block diagram illustrating a device of Petition 870250077578, dated 01 / 09 / 2025, page 19 / 202 13 / 88 computing 200 that can display the graphical user interface 100 according to some implementations. The computing device can also be used by a data preparation application 250. Several examples of the computing device 200 include a desktop computer, a laptop computer, a tablet computer, and other computing devices that have a display medium and a processor capable of running a data visualization application 222. The computing device 200 typically includes one or more processing units / cores (CPUs) 202 for executing modules, programs and / or instructions stored in memory 214 and thus performing processing operations; one or more networks or other communication interfaces 204; memory 214; and one or more communication buses 212 to interconnect these components.Communication buses 212 may include a system of circuits that interconnects and controls communications between system components.

[069] The computing device 200 includes a user interface 206 comprising a display device 208 and one or more input devices or mechanisms 210. In some implementations, the input device / mechanism includes a keyboard. In some implementations, the input device / mechanism includes a “virtual” (soft) keyboard, which is displayed as needed on the display device 208, allowing a user to “press keys” that appear on the display medium 208. In some implementations, the display medium 208 and the input device / mechanism 210 comprise a touch-screen display (also called a touch-sensitive display).

[070] In some implementations, memory 214 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, memory 214 includes non-volatile memory, such as one or more Petition 870250077578, dated 01 / 09 / 2025, p. 20 / 202 14 / 88 magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 214 includes one or more storage devices located remotely from the CPU(s) 202. Memory 214, or alternatively, the non-volatile memory device(s) within memory 214, comprise a non-temporary, computer-readable storage medium. In some implementations, memory 214, or the computer-readable storage medium of memory 214, stores the following programs, modules, and data structures, or a subset thereof: • an operating system 216, which includes procedures for handling various basic system services and for performing hardware-dependent tasks; • a communications module 218 that is used to connect the computing device 200 to other computers and devices by means of one or more communication network interfaces 204 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so forth; • a web browser (or other application capable of displaying web pages) that allows a user to communicate over a network with remote computers or devices; • a data visualization application 222, which provides a graphical user interface 100 for a user to construct visual charts. For example, a user selects one or more data sources 240 (which may be stored on the computing device 200 or stored remotely), selects data fields from the data source(s), and uses the selected fields to define a visual chart. In some implementations, the information that the user provides is stored as a visual specification 228. The application of Petition 870250077578, dated 01 / 09 / 2025, page 21 / 202 15 / 88 Data visualization 222 includes a data visualization generation module 226, which takes user input (e.g., visual specification 228) and generates a corresponding visual graphic (also called a “data visualization” or “data viz”). The data visualization application 222 then displays the generated visual graphic in the user interface 100. In some implementations, the data visualization application 222 runs as a standalone application (e.g., a desktop application). In some implementations, the data visualization application 222 runs within the Network browser 220 or in another application using Network pages provided by a Network server; and • zero or more databases or data sources 240 (e.g., a first data source 240-1 and a second data source 240-2), which are used by the data visualization application 222.In some implementations, data sources are stored as spreadsheet files, CSV files, XML files, or plain text files, or stored in a relational database.

[071] In some cases, the computing device 200 stores a data preparation application 250, which can be used to analyze and process data for subsequent analysis (e.g., by a data visualization application 222). Figure 3B illustrates an example of a user interface 251 used by a data preparation application 250. The data preparation application 250 allows the user to construct flows 323, as described in more detail below.

[072] Each of the executable modules, applications, or sets of procedures identified above can be stored in one or more of the memory devices mentioned above, and corresponds to a set of instructions to perform a function described above. The modules or programs identified above (i.e., sets of instructions) do not need to be Petition 870250077578, dated 01 / 09 / 2025, page 22 / 202 16 / 88 implemented as separate program software, procedures, or modules, and thus, various subsets of these modules can be combined or otherwise rearranged in the various implementations. In some implementations, memory 214 stores a subset of the modules and data structures identified above. Additionally, memory 214 may store additional modules or data structures not described above.

[073] Although Figure 2 illustrates a computing device 200, Figure 2 is conceived more as a functional description of the various aspects that may be present rather than as a structural scheme of the implementations described herein. In practice, and as recognized by those skilled in the art, the items presented separately could be combined and some items could be separated.

[074] Figures 3A and 3B illustrate a user interface for preparing data according to some implementations. In these implementations, there are at least five regions, which have distinct functionality. Figure 3A shows this conceptually as a menu bar region 301, a left panel 302, a flow panel 303, a profile panel 304, and a data panel 305. In some implementations, the profile panel 304 is also called the schema panel. In some implementations, the functionality of the “left panel” 302 is in an alternative location, such as below the menu panel 301 or below the data panel 305.

[075] This interface provides a user with multiple coordinated and streamlined views that help the user see and understand what they need to do. This new user interface presents users with multiple views of their flow and data to help them not only perform actions but also discover what actions they need to take. The flowchart in the flow panel 303 combines and summarizes actions, making the flow more readable, and is coordinated with data views. Petition 870250077578, dated 01 / 09 / 2025, page 23 / 202 17 / 88 reais in profile panel 304 and data panel 305. Data panel 305 provides representative samples of the data at each point in the logical flow, and the profile panel provides histograms of the data domains.

[076] In some implementations, Menu Bar 301 has a “File” menu with options to create new data flow specifications, save data flow specifications, and load previously created data flow specifications. In some cases, a flow specification is called a flow. A flow specification describes how to manipulate input data from one or more data sources to create a target dataset. Target datasets are typically used in subsequent data analysis using a data visualization application.

[077] In some implementations, Left Panel 302 includes a list of recent data source connections, as well as a button to connect to a new data source.

[078] In some implementations, the Flow Panel 303 includes a visual representation (flowchart or flow) of the flow specification. In some implementations, the flow is a diagram of nodes / links showing the data sources, the operations that are performed, and the target outputs of the flow.

[079] Some implementations provide flexible execution of a flow by treating parts of the flow as declarative queries. That is, instead of having a user specify every computational detail, a user specifies the goal (e.g., input and output). The process executing the flow optimizes the plans to choose execution strategies that improve performance. The implementations also allow users to selectively inhibit this behavior to control execution.

[080] In some implementations, the Profile Panel 304 displays the relevant schema and statistics and / or visualizations for the nodes selected in the Panel. Petition 870250077578, dated 01 / 09 / 2025, page 24 / 202 18 / 88 Flow 303. Some implementations support selecting multiple nodes simultaneously, but other implementations support selecting only a single node at a time.

[081] In some implementations, Data Panel 305 displays row-level data for the nodes selected in Flow Panel 303.

[082] In some implementations, a user creates a new flow using a “File -> New Flow” option in the Menu Bar. Users can also add data sources to a flow. In some examples, a data source is a relational database. In some cases, one or more data sources are file-based, such as CSV files or spreadsheet files. In some implementations, a user adds a file-based source to the flow using a visual file connection indicator in the left pane 302. This opens a file dialog that prompts the user to choose a file. In some implementations, the left pane 302 also includes a visual database connection indicator, which allows the user to connect to a database (e.g., an SQL database).

[083] When a user selects a node (for example, a table) in the Flow Pane 303, the schema for the node is displayed in the Profile Pane 304. In some implementations, the profile pane 304 includes statistics or visualizations, such as distributions of data values ​​for the fields (for example, as histograms or pie charts). In implementations that allow the selection of multiple nodes in the flow pane 303, the schemas for each of the selected nodes are displayed in the profile pane 304.

[084] Additionally, when a node is selected in the Flow Pane 303, the data for the node is displayed in the Data Pane 305. The Data Pane 305 typically displays the data as rows and columns.

[085] The implementations make it easy to edit the flow using the panel. Petition 870250077578, dated 01 / 09 / 2025, page 25 / 202 19 / 88 Flow 303, Profile Pane 304, or Data Pane 305. For example, some implementations allow a right-click operation on a node / table in any of these three panes to add a new column based on a scalar calculation on the existing columns in that table. For example, the scalar operation could be a mathematical operation to calculate the sum of three numeric columns, a string operation to concatenate string data into two columns that are strings, or a conversion operation to convert a string column into a date column (when a date has been encoded as a string in the data source). In some implementations, a context menu (accessed from a table / node in Flow Pane 303, Profile Pane 304, or Data Pane 305) offers a “Create calculated field...” option.Selecting this option brings up a dialog box to create a calculation. In some implementations, calculations are limited to scalar calculations (e.g., excluding aggregations, custom Level of Detail calculations, and table calculations). When a new column is created, the user interface adds a calculated node in the Flow Pane 303, connects the new node to its antecedent, and selects this new node. In some implementations, as the number of nodes in the flowchart increases, the Flow Pane 303 adds scroll boxes. In some implementations, nodes in the flowchart can be grouped together and labeled, which is displayed hierarchically (e.g., showing a high-level flow initially, expanding to see the details of the selected nodes).

[086] A user can also remove a column by interacting with the Flux Panel 303, the Profile Panel 304, or the Data Panel 305 (for example, by right-clicking on the column and choosing the “Remove Column” option). Removing a column results in the addition of a Petition 870250077578, dated 01 / 09 / 2025, page 26 / 202 Connect the new node to the Flow Panel 303 (20 / 88), ensuring it is properly connected, and selecting the new node.

[087] In the Flow Pane 303, a user can select a node and choose “Output As” to create a new output dataset. In some implementations, this is done with a right-click. This evokes a file dialog box that allows the user to select a target filename and directory (or a database and table name). Doing so adds a new node to the Flow Pane 303, but does not actually create the target datasets. In some implementations, a target dataset has two components, including a first file (a Tableau Data Extract or TDE) that contains the data, and a corresponding index or pointer record (a Tableau Data Source or TDS) that points to the data file.

[088] The actual output data is created when the flow is executed. In some implementations, a user runs a flow via the “File -> Run Flow” option in Menu Bar 301. Note that a single flow can produce multiple output data files. In some implementations, the flowchart provides visual feedback as it runs.

[089] In some implementations, Menu Bar 301 includes a “File” menu option for Save or Save As, which allows the user to save the stream. In some implementations, a stream is saved as a “.loom” file. This file contains everything needed to recreate the stream during loading. When a stream is saved, it can be reloaded later using a “Load” menu option in the “File” menu. This causes a file selection dialog to appear so that the user can load a previous stream.

[090] Figure 3B illustrates a user interface for preparing Petition 870250077578, dated 01 / 09 / 2025, page 27 / 202 21 / 88 data, showing the user interface elements in each of the panels. The menu bar 311 includes one or more menus, such as a File menu and an Edit menu. Although the edit menu is available, further changes to the flow are made by interacting with the flow panel 313, the profile panel 314, or the data panel 315.

[091] In some implementations, the left pane 312 includes a data source palette / selector, which includes visual cues for locating and connecting to data. The set of connectors includes extract-only connectors, including cubes. Implementations can emit custom SQL expressions for any data source that supports them.

[092] The left panel 312 also includes an operations palette, which displays operations that can be placed in the flow. This includes arbitrary joins (of arbitrary type and with various predicates), union, row-to-column transformation, renaming and restricting columns, scalar calculation projection, filtering, aggregation, data type conversion, data analysis, merging, combining, splitting, aggregation, value replacement, and sampling. Some implementations also support operators for creating sets (e.g., partitioning data values ​​for a data field into sets), binning (e.g., grouping numeric data values ​​for a data field into a set of ranges), and table calculations (e.g., calculating data values ​​(e.g., percentage of total) for each row that depends not only on the data values ​​in the row but also on other data values ​​in the table).

[093] The left panel 312 also includes a palette of other flows that can be incorporated wholly or partially into the current flow. This allows a user to reuse components of a flow to create new flows. For example, if a part of a flow has been created that debugs a certain type of input using a combination of 10 steps, that 10-step flow part can be Petition 870250077578, dated 01 / 09 / 2025, page 28 / 202 22 / 88 is saved and reused, either in the same stream or in completely separate streams.

[094] The Flow Panel 313 displays a visual representation (e.g., node / link flowchart) 323 for the current flow. The Flow Panel 313 provides an overview of the flow, which serves to document the process. In many existing products, a flow is overly complex, which hinders understanding. The revealed implementations facilitate understanding by coalescing nodes, keeping the overall flow simpler and more concise. As noted earlier, as the number of nodes increases, implementations typically add scrollbars. The need for scrollbars is reduced by combining multiple related nodes into supernodes, which are also called container nodes. This allows a user to view the entire flow more conceptually, and allows a user to explore the details only when necessary.In some implementations, when a "supernode" is expanded, the 313 flow pane shows only the nodes within the supernode, and the 313 flow pane has a header that identifies which part of the flow is being displayed. Implementations typically allow multiple hierarchical levels. A complex flow is likely to include multiple levels of node nesting.

[095] As described above, the profile panel 314 includes schema information about the data in the node (or nodes) currently selected in the flow panel 313. As illustrated here, the schema information provides statistical information about the data, such as a histogram 324 of the data distribution for each of the fields. A user can interact directly with the profile panel to modify the flow 323 (for example, by selecting a data field to filter the data rows based on the values ​​of that data field). The profile panel 314 also provides users with relevant data about the currently selected node (or nodes) and visualizations that guide a user's work. For example, the histograms 324 show the distributions of the domains of Petition 870250077578, dated 01 / 09 / 2025, page 29 / 202 23 / 88 each column. Some implementations use cleanup to show how these domains interact with each other.

[096] An example here illustrates how the process differs from typical implementations by allowing a user to directly manipulate data in a stream. Consider two alternative ways to filter specific data rows. In this case, a user wants to disregard “California”. Using a typical tool, a user selects a “filter” node, places the filter in the stream at a certain location, and then opens a dialog box to enter the calculation formula, such as “state_name <> 'CA'”. In the implementations revealed here, the user can see the data value in profile pane 314 (e.g., illustrating the field value 'CA' and how many rows have that field value) and in data pane 315 (e.g., individual rows with 'CA' as the value for state_name).In some implementations, the user can right-click on “CA” in the list of state names in Profile Panel 314 (or Data Panel 315), and choose “Delete” from a drop-down list. The user interacts with the data itself, not with a flow element that interacts with the data. The implementations offer similar functionality for calculations, joins, unions, aggregates, and so on. Another benefit of the approach is that the results are immediate. When “CA” is filtered, the filter is applied immediately. If the operation takes some time to complete, the operation is performed asynchronously, and the user can continue with their work while the task runs in the background.

[097] Data panel 315 displays the rows of data corresponding to the node or nodes selected in flow panel 313. Each of the columns in 315 corresponds to one of the data fields. A user can interact directly with the data in the data panel to modify flow 323 in flow panel 313. A user can also interact directly with the data panel to Petition 870250077578, dated 01 / 09 / 2025, page 30 / 202 24 / 88 Modify individual field values. In some implementations, when a user makes a change to a field value, the user interface applies the same change to all other values ​​in the same column whose values ​​(or default) match the value the user just changed. For example, if a user has changed “WA” to “Washington” for a field value in a “State” data column, some implementations update all other “WA” values ​​to “Washington” in the same column. Some implementations go further and update the column to replace any state abbreviations in the column with the full state names (for example, replacing “OR” with “Oregon”). In some implementations, the user is prompted to confirm before applying a global change to an entire column.In some implementations, a change to a value in one column can be applied (automatically or pseudo-automatically) equally to the other columns. For example, a data source might include both a state for residence and a state for billing. A change in formatting for the states could then be applied to both.

[098] The sampling of data in data pane 315 is selected to provide valuable information to the user. For example, some implementations select rows that display the entire range of values ​​for a data field (including outliers). As another example, when a user has selected nodes that have two or more data tables, some implementations select rows to assist in joining the two tables. The rows displayed in data pane 315 are selected to show both matching rows between the two tables and non-matching rows. This can be useful when determining which fields will be used in the join and / or when determining which type of join to use (e.g., inner, left outer, right outer, or full outer). Petition 870250077578, dated 01 / 09 / 2025, page 31 / 202 25 / 88

[099] Figure 3C illustrates some of the aspects shown in the user interface, and what is shown by the aspects. As illustrated above in Figure 3B, flowchart 323 is always displayed in flow panel 313. The profile panel 314 and the data panel 315 are also always shown, but the content of these panels changes based on which node or nodes are selected in flow panel 313. In some cases, selecting a node in flow panel 313 causes one or more node-specific panels to appear (not shown in Figure 3A or Figure 3B). When displayed, a node-specific panel is in addition to the other panels. In some implementations, node-specific panels are displayed as floating “popups” that can be moved. In some implementations, node-specific panels are displayed in fixed locations within the user interface.As noted above, the left panel 312 includes a data source palette / selector for selecting or opening data sources, as well as an operations palette for selecting operations that can be applied to the flowchart 323. Some implementations also include an “other flows palette,” which allows the user to import all or part of another flow into the current flow 323.

[0100] The different nodes within the 323 flowchart perform different tasks, and therefore the internal information of the node is different. Furthermore, some implementations display different information depending on whether a node is selected or not. For example, an unselected node includes a simple description or label, whereas a selected node displays more detailed information. Some implementations also display the status of operations. For example, some implementations display nodes within the 323 flowchart differently depending on whether the node's operations have been executed or not. Additionally, within the operations palette, some implementations display operations differently depending on whether they are available for use with the currently selected node. Petition 870250077578, dated 01 / 09 / 2025, page 32 / 202 26 / 88

[0101] A flowchart 323 provides a simple visual way to understand how data is being processed, and keeps the process organized in a way that is logical for a user. Although a user can edit a flowchart 323 directly in the flow pane 313, changes to operations are typically made more immediately by operating directly on the data or schemas in the profile pane 314 or the data pane 315 (for example, by right-clicking on the statistics for a data field in the profile pane to add or remove a column from the flow).

[0102] Instead of displaying a node for each tiny operation, users can group operations together into a smaller number of more meaningful nodes. For example, a join followed by the removal of two columns can be implemented in one node instead of three separate nodes.

[0103] Within the flow panel 313, a user can perform various tasks, such as: • Swap node selection. This determines which data is displayed in the rest of the user interface. • Prefix flow operations. This allows the user to specify that a certain part of the flow must happen first and cannot be reordered. • Division and Combination of Operations. Users can easily reorganize the operation to fit a logical model of what is happening. For example, a user might want to create a node called “Standardize Hospital Codes,” which contains several operations and special cases. A user can initially create the individual operations, and then join the nodes representing individual operations into the supernode “Standardize Hospital Codes.” Conversely, having created a node containing many individual operations, a user can choose to split off one or more of them. Petition 870250077578, dated 01 / 09 / 2025, p. 33 / 202 27 / 88 operations (for example, to create a node that can be reused more generally).

[0104] The 314 profile pane offers a quick way for users to discover if the results of the transformations are what they expected. Outliers and incorrect values ​​often visually “jump out” based on comparisons with other values ​​in the node as well as comparisons of values ​​in other nodes. The profile pane helps users uncover data problems, regardless of whether the problems are caused by incorrect transformations or dirty data. In addition to helping users find bad data, the profile pane also allows direct interactions to fix the discovered problems. In some implementations, the 314 profile pane is updated asynchronously. When a node is selected in the flow pane, the user interface begins to populate partial values ​​(e.g., histograms of data value distribution) that improve over time.In some implementations, the profile panel includes an indicator to alert the user whether the profile is complete or not. With very large datasets, some implementations build a profile based only on sample data.

[0105] Within profile panel 314, a user can perform various tasks, including: • Investigate data ranges and correlations. Users can use the 314 profile panel to focus on specific relationships between columns or data using direct navigation. • Filtering data or data ranges. Users can add filter operations to flow 323 through direct interactions. This results in the creation of new nodes in the flow 313 panel. • Data transformation. Users can interact directly with the Petition 870250077578, dated 01 / 09 / 2025, page 34 / 202 28 / 88 profile panel 314 in order to map values ​​from one range to another value. This creates new nodes in flow panel 313.

[0106] Data panel 315 provides a way for users to view and modify rows resulting from flows. Typically, the data panel selects a sample of rows corresponding to the selected node (e.g., a sample of 10, 50, or 100 rows instead of a million rows). In some implementations, rows are sampled in order to display a variety of aspects. In some implementations, rows are sampled statistically, such as every nth row.

[0107] Data pane 315 is typically where a user performs data cleansing (e.g., when the source data is not clean). Like the profile pane, the data pane updates asynchronously. When a node is first selected, rows in data pane 315 begin to appear, and sampling improves over time. Most datasets will only have a subset of the data available here (unless the dataset is small).

[0108] Within data panel 315, a user can perform various tasks, including: • Sort by navigation. A user can sort the data in the data panel based on a column, which has no effect on the flow. The goal is to assist in navigating the data in the data panel. • Filter by navigation. A user can filter the data that is in the view, which does not add a filter to the flow. • Add a filter to the flow. A user can also create a filter that applies to the flow. For example, a user can select an individual data value for a specific data field, and then perform an action to filter the data according to that value (for example, delete that value or include it). Petition 870250077578, dated 01 / 09 / 2025, page 35 / 202 29 / 88 only this value). In this case, the user interaction creates a new node in the data flow 323. Some implementations allow a user to select multiple data values ​​in a single column, and then build a filter based on the set of selected values ​​(e.g., exclude the set or limit to only this set). • Modify row data. A user can directly modify a row. For example, change a data value for a specific field in a specific row from 3 to 4. • Mapping one value to another. A user can modify a data value for a specific column and propagate that change to all rows that contain that value for that specific column. For example, replacing “NY” with “NY” for an entire column representing states. • Split columns. For example, if a user sees that the dates have been formatted as “14-Nov-2015”, the user can split this field into three separate fields for day, month, and year. • Merge columns. A user can merge two or more columns to create a single combined column.

[0109] A node-specific panel displays information that is specific to a selected node in the flow. Since a node-specific panel is not needed most of the time, the user interface generally does not designate a region with user interface that serves only this purpose. Instead, a node-specific panel is displayed as needed, typically using a “popup” window that floats over other regions of the user interface. For example, some implementations use a node-specific panel to provide specific user interfaces for joins, unions, row-to-column transformations, column-to-row transformations, execution of Python scripts, parsing of log files, or transformation of JSON objects to the Petition 870250077578, dated 01 / 09 / 2025, page 36 / 202 30 / 88 tabular form.

[0110] The Data Source Palette / Selector allows a user to display data from multiple data sources. In some implementations, the data source palette / selector is located in the left panel 312. A user can perform various tasks with the data source palette / selector, including: • Establish a data source connection. This allows a user to retrieve data from a data source, which can be an SQL database, a data file such as a CSV or spreadsheet, a non-relational database, a network service, or another data source. • Define connection properties. A user can specify credentials and other properties required to connect to data sources. For some data sources, the properties include selecting specific data (for example, a specific table in a database or a specific sheet from a workbook file).

[0111] In many cases, users invoke operations on nodes in the flow based on user interactions with the profile panel 314 and the data panel 315, as illustrated above. Additionally, the left panel 312 offers an operations palette, which allows a user to invoke certain operations. For example, some implementations include an option to “Call a Python Script” in the operations palette. Furthermore, when users create nodes that they wish to reuse, they can save them as available operations in the operations palette. The operations palette provides a list of known operations (including user-defined operations), and allows a user to incorporate operations into the flow using user interface gestures (e.g., drag and drop).

[0112] Some implementations offer an Other Flows Palette / Selector, which allows users to easily reuse flows they have built or flows that other people have built. The other flows palette provides a Petition 870250077578, dated 01 / 09 / 2025, page 37 / 202 31 / 88 list of other flows from which the user can branch out or incorporate. Some implementations support selecting parts of other flows in addition to selecting entire flows. A user can incorporate other flows using user interface gestures, such as drag and drop.

[0113] The node's internal mechanisms specify exactly which operations are occurring within a node. There is sufficient information to allow a user to "refactor" a flow or understand a flow in more detail. A user can see exactly what is in the node (e.g., what operations are performed), and can move operations out of the node, to another node.

[0114] Some implementations include a project template, which allows the user to group multiple flows together into a “project” or “workbook”. For complex flows, a user can break down the overall flow into more understandable components.

[0115] In some implementations, the status of operations is displayed in the left panel 312. Because many operations are executed asynchronously in the background, the operation status region indicates to the user which operations are in progress, as well as the progress status (e.g., 1% complete, 50% complete, or 100% complete). The operation status shows which operations are happening in the background, allows the user to cancel operations, allows the user to update data, and allows the user to run partial results until they are completed.

[0116] A stream, such as stream 323 in Figure 3B, represents a pipeline (flow) of rows that flow from the original data sources through transformations to target datasets. For example, Figure 3D illustrates a simple example stream 338. This stream is based on traffic accidents involving vehicles. The relevant data is stored in a database. Petition 870250077578, dated 01 / 09 / 2025, page 38 / 202 32 / 88 SQL querying a data set for an accident table and a vehicle table. In this flow, a first node (340) reads data from the accident table, and a second node (344) reads data from the vehicle table. In this example, the accident table is normalized (342) and one or more key fields are identified (342). Similarly, one or more key fields are identified (346) for the vehicle data. The two tables are joined (348) using a shared key, and the results are written (350) to a target dataset. If both the accident table and the vehicle table are in the same SQL database, an alternative is to create a single node that reads data from both tables in a query. The query can specify which data fields to select and whether the data should be limited by one or more filters (e.g., WHERE clauses).In some examples, the data is retrieved and joined locally as indicated in flow 338, because the data used to join the tables needs to be modified. For example, the primary key of the vehicles table might have an integer data type, while the accidents table might specify the vehicles involved using a character field padded with zeros.

[0117] A flow abstraction like the one illustrated in Figure 3D is common to most ETL and data preparation products. This flow model offers users logical control over their transformations. Such a flow is generally interpreted as an imperative program and executed with little or no modification by the platform. That is, the user has provided the specific details to define physical control over the execution. For example, a typical ETL system operating on this flow will open the two database tables exactly as specified, model the data as specified, join the tables in the ETL engine, and then write the result to the target dataset. Total control over the physical plane can be useful, but it impairs the system's ability to modify or Petition 870250077578, dated 01 / 09 / 2025, page 39 / 202 33 / 88 Optimize the plan to improve performance (e.g., execute the preceding plan on the SQL server). Most clients do not need control over execution details, so the implementations here allow operations to be expressed declaratively.

[0118] Some implementations here cover the entire range from fully declarative queries to imperative programs. Some implementations utilize an internal Analytical Query Language (AQL) and a Federated Evaluator. By default, a flow is interpreted as a single declarative query specification whenever possible. This declarative query is converted to AQL and delivered to a Query Evaluator, which ultimately splits the operators, distributes them, and executes them. In the example above in Figure 3D, the entire flow can be launched as a single query. If both tables come from the same server, this entire operation would likely be sent to the remote database, achieving a significant performance benefit.Flexibility not only allows for optimization and distribution of workflow execution, but also for the execution of queries against real-time data sources (for example, from a transactional database, and not just a data warehouse).

[0119] When a user wants to control the actual execution order of the flow (for example, for performance reasons), the user can pin an operation. Pinning tells the flow execution module not to move operations beyond that point in the plan. In some cases, a user may want to exert extreme control over the order temporarily (for example, during flow creation or debugging). In this case, all operators can be pinned, and the flow is executed exactly in the order the user specified.

[0120] Note that not all flows can be decomposed into a single AQL query, as illustrated in Figure 3E. In this flow, there is an elimination of Petition 870250077578, dated 01 / 09 / 2025, page 40 / 202 34 / 88 hourly 352 which is executed hourly (362), and the data are normalized (354) before being appended (356) to a temporary database. Then, on a daily basis (364), the data from the temporary database are aggregated (358) and written (360) as a target dataset. In this case, the hourly schedule and the daily schedule need to remain as separate units.

[0121] Figures 4A to 4V illustrate some aspects of adding a join to a flow according to some implementations. As illustrated in Figure 4A, the user interface includes a left panel 312, a flow area 313, a profile area 314, and a data grid 315. In the example of Figures 4A to 4V, the user first connects to an SQL database using the connection palette in the left panel 312. In this case, the database includes data from the Fatality Analysis Reporting System (FARS) provided by the National Highway Traffic Safety Administration. As illustrated in Figure 4B, a user selects the “Accidents” table 404 from the list 402 of available tables. In Figure 4C, the user drags the Accidents table icon 406 to the flow area 313. Once the table icon 406 is dropped into the flow area 313, a node 408 is created to represent the table, as shown in Figure 4D.At this point, the data for the Accident table is loaded, and profile information for the accident table is displayed in profile panel 314.

[0122] Profile panel 314 provides distribution data for each of the columns, including the state column 410, as illustrated in Figure 4E. In some implementations, each data column in the profile panel displays a histogram to show the data distribution. For example, California, Florida, and Georgia have a high number of accidents, while Delaware has a low number of accidents. The profile panel makes it easy to identify columns that are keys or partial keys using key icons 412 at the top of each column. Petition 870250077578, dated 01 / 09 / 2025, page 41 / 202 35 / 88 As shown in Figure 4F, some implementations use three different icons to specify whether a column is a database key, a 414 system key, or a “practically” 416 system key. In some implementations, a column is practically a system key when the column, together with one or more other columns, is a system key. In some implementations, a column is practically a system key if the column would be a system key if rows with null values ​​were excluded. In this example, both “ST Case” and “Case Number” are practically system keys.

[0123] In Figure 4G, a user selected the “Persons” table 418 in the left panel 312. In Figure 4H, the user drags the People table 418 to the flow area 313, which is displayed as a movable icon 419 while being dragged. After dropping the People table icon 419 into the flow area 313, a People node 422 is created in the flow area, as illustrated in Figure 4I. At this stage, there is no connection between the Accidents node 408 and the People node 422. In this example, both nodes are selected, therefore the profile panel 314 is divided into two parts: the first part 420 shows the profile information for the Accidents node 408 and the second part 421 shows the profile information for the People node 422.

[0124] Figure 4J provides an enlarged view of the flow pane 313 and the profile pane 314. The profile pane 314 includes an option 424 to show join column candidates (i.e., possibilities for joining the data from the two nodes). After selecting this option, the data fields that are join candidates are displayed in the profile pane 314, as illustrated in Figure 4K. Once the join candidates are displayed, the profile pane 314 displays an option 426 to hide join column candidates. In this example, the profile pane 314 indicates (430) that the “ST case” column in the People table could be joined with the “ST case” field in the Accidents table. The profile pane also indicates (428) Petition 870250077578, dated 01 / 09 / 2025, page 42 / 202 36 / 88 that there are three additional join candidates in the Accidents table and indicates (432) that there are two additional join candidates in the People table. In Figure 4L, the user clicks (433) on the tooltip icon, and in response, the profile panel places the two candidate columns adjacent to each other as illustrated in Figure 4M. The header 434 for the “ST Case” column of the Accidents table now indicates that it can be joined with the “ST case” column of the People table.

[0125] Figure 4N illustrates an alternative method for joining data for multiple nodes. In this example, a user loaded data from the Accidents table 408 and data from the Populations table 441 into flow area 313. By simply dragging the Populations node 441 onto the Accidents node 408, a join is automatically created and a Join Experience panel 442 is displayed, which allows the user to analyze and / or modify the join. In some implementations, the Join Experience is placed in the profile panel 314; in other implementations, the Join Experience temporarily replaces the profile panel 314. When the join is created, a new node 440 is added to the flow, which graphically displays the creation of a connection between the two nodes 408 and 441.

[0126] The Join Experience 442 includes a toolbar area 448 with various icons, as illustrated in Figure 4O. When the join candidate icon 450 is selected, the interface identifies which fields in each table are join candidates. Some implementations include a favorites icon 452, which displays highlighted “favorite” data fields (e.g., either previously selected by the user, previously identified as important by the user, or previously selected by users in general). In some implementations, the favorites icon 452 is used to designate certain data fields as favorites. Since there is limited space to display the columns in the profile pane 314 and data pane 315, some implementations use information about favorite data fields to select which columns are Petition 870250077578, dated 01 / 09 / 2025, page 43 / 202 37 / 88 displayed by default.

[0127] In some implementations, selecting the show keys icon 454 causes the interface to identify which data columns are keys or parts of a key consisting of multiple data fields. Some implementations include a toggle data / metadata icon 456, which toggles the display between showing data information and showing metadata information. In some implementations, the data is always displayed, and the metadata icon 456 toggles whether or not metadata is displayed in addition to the data. Some implementations include a data grid icon 458, which toggles the display of the data grid 315. In Figure 4O, the data grid is currently displayed, so the data grid icon 458 would cause the data grid not to be displayed. Implementations typically also include a search icon 460, which triggers a search window.By default, a search applies to both data and metadata (for example, both the names of data fields and the data values ​​within the fields). Some implementations include the option for an advanced search to more precisely specify what is being searched.

[0128] To the left of the join experiment 442 is a join set, including a join type specification 464. As is known in the art, a join is typically a left outer join, an inner join, a right outer join, or a fully outer join. These are graphically illustrated by the join icons 464. The current join type is highlighted, but the user can change the join type by selecting a different icon.

[0129] Some implementations offer an overview of the 466 union clause, which displays both the field names on both sides of the union, as well as histograms of the data values ​​for the data fields on both sides of the union. When there are multiple data fields in the union, some Petition 870250077578, dated 01 / 09 / 2025, page 44 / 202 38 / 88 implementations display all relevant data fields; other implementations include a user interface control (not shown) to iterate through the data fields in the join. Some implementations also include an overview control 468, which illustrates how many rows from each of the tables are joined based on the join condition type. Selecting parts within this control determines what is displayed in the profile pane 314 and the data grid 315.

[0130] Figures 4P, 4Q, and 4R illustrate alternative user interfaces for the join control area 462. In each case, the join type appears at the top. In each case, there is a visual representation of the data fields included in the join. Here, there are two data fields in the join, which are “ST case” and “Year”. In each of these alternatives, there is also a section that graphically illustrates the row fractions of each table that are joined. The top part of Figure 4Q appears in Figure 4U below.

[0131] Figure 4R includes a bottom section showing how the two tables are related. Split bar 472 represents rows in the Accidents table, and split bar 474 represents the Populations table. Large bar 477 in the middle represents rows that are connected by an inner join between the two tables. Since the currently selected join type is a left outer join, the join result set 476 also includes a part 478 representing rows from the Accidents table that are not linked to any row in the Populations table. At the bottom is another rectangle 480, which represents rows from the Populations tables that are not linked to any row in the Accidents table. Since the current join type is a left outer join, part 480 is not included in result set 476 (the rows in the bottom rectangle 480 would be included in a right outer join or a full outer join).A user can select any part of this diagram, and the selected part is displayed in the profile panel and the data panel. For example, Petition 870250077578, dated 01 / 09 / 2025, page 45 / 202 39 / 88 a user can select the “outer left” rectangle 478 and then look at the rows in the data panel to see if those rows are relevant to the user's analysis.

[0132] Figure 4S shows a Union Experiment using the union control interface elements illustrated in Figure 4R, including the union control selector 464. Here, the outer left union icon 482 is highlighted, as illustrated more clearly in the enlarged view of Figure 4T. In this example, the first table is the Accidents table, and the second table is the Factor table. As illustrated in Figure 4U, the interface shows both the number of rows that are joined 486 and the number of rows that are not joined 488. This example has a large number of rows that are not joined. The user can select the unjoined bar 488 to display the screen in Figure 4V. By clearing the profile and filtering the data grid, the user can see that the nulls are a result of an outer left union and mismatched values ​​due to the fact that the Factor table has no entries prior to 2010.

[0133] The revealed implementations support several aspects that help in a variety of situations. Many of the aspects have been described above, but some of the following scenarios illustrate the aspects. Scenario 1: Event Log Collection

[0134] Alex works in IT, and one of his tasks is to collect and prepare logs from the machines in his infrastructure to produce a shared dataset that is used for various debugging and analysis in the IT organization.

[0135] The machines run Windows, and Alex needs to collect Application logs. There is already an agent that runs nightly and deposits exported CSV files of the logs into a shared directory; daily data is placed in a separate directory and is generated in a format that Petition 870250077578, dated 01 / 09 / 2025, page 46 / 202 40 / 88 indicates the machine name. An excerpt from the Applications registry is illustrated in Figure 5A.

[0136] This has some interesting features: • Line 1 contains header information. This may or may not be the case in all cases. Each row of data has six columns, but the header has five. • The delimiter here is clearly “,”. • The final column may have used multi-line character sequences enclosed in quotation marks. Note that all lines 3 through 9 here are part of a row. Also note that this field uses double-double quotation marks to indicate quotation marks that should be interpreted literally.

[0137] Alex creates a flow that reads all CSV files in a given directory and performs an irregular join on them (for example, it creates a data field if it exists in at least one of the CSV files, but when the same data field exists in two or more of the CSV files, it creates only one instance of that data field). The CSV input routine performs incredibly well in reading five columns, but clashes with the quotation marks in the sixth column, interpreting them as multiple columns.

[0138] Alex, then: • Select the columns in the data panel and combine them again. • Add the machine name from which the data came, taken from the filename. To do this, select the machine name from a data example and choose “Extract as new column”. The system infers a pattern from this action. • Generates a unique identifier for each row by right-clicking and choosing “add identifier”. • Edit column names and types directly in the data panel. Petition 870250077578, dated 01 / 09 / 2025, page 47 / 202 41 / 88

[0139] All of this is done through direct action on the data in data panel 315, but results in the logic being inserted into the flow in flow panel 313.

[0140] Alex then drags his target data repository to the flow pane, and prepares the output to append those records to a cache that will contain a complete record of his records.

[0141] Finally, Alex's flow queries this target dataset to find the set of machines that reported the previous day, compares this with the machines of the current day, and issues an alert to Alex with a list of expected machines that did not report.

[0142] Note that Alex could have achieved the same result in different ways. For example: • Alex could create two separate streams: one that performs the ingestion; and another that compares the machines from each day with the machines from the previous day, and then alerts Alex with the results. • Alex could create a flow that performs the ingestion in one stage. When this is complete, Alex could execute a second flow that queries the database and compares each day with the previous day and alerts Alex. • Alex could create a flow that would have the target as both input and output. This flow would perform data ingestion, record it in the database, and additionally aggregate it to find the machines of the day. It would also query the target to obtain the results from the previous day, perform the comparison, and trigger the alert.

[0143] Alex knows that the machines are supposed to produce the report overnight, so the first thing Alex does every morning is run his workflow. He then has the rest of the morning to check which machines haven't produced the report. Scenario 2: FARS Collection and Integration Petition 870250077578, dated 01 / 09 / 2025, page 48 / 202 42 / 88

[0144] Bonnie works for an insurance company, and would like to obtain Fatality Analysis Reporting System (FARS) data as a component of his analysis. The FARS data is available through an FRP, and Bonnie needs to figure out how to obtain and reconstruct it. He decides to do this using the Data Preparation Application 250.

[0145] Bonnie takes a look at the set of formats in which FARS publishes, and decides to use the DBF file. These DBF files are scattered across the site's FTP and are only available as compressed ZIP files. Bonnie explores a tree view and selects the files she wants to download. While the data is downloading, Bonnie initiates the next step in her workflow. She selects the set of files and chooses “Extract,” which adds a step of unpacking the files into separate directories labeled by year. [ 0146] As the data starts to come in, Bonnie begins to see problems: • The initial years have three files, corresponding to three tables: accident, person, and vehicle. These are present in the later years, but there are many other tables as well. • The files do not have uniform names. For example, the accident file is named “accident.dbf” for the years 1975 to 1982 and 1994 to 2014, but is named “accYYYY.dbf” (where YYYY is the four-digit year) for the middle years. • Even when the table names are the same, their structure changes considerably over time. Later tables include additional columns not present in the earlier data.

[0147] Bonne starts with the “accident” table, which is present in all years. She selects the files, right-clicks, and chooses “Union,” which performs an irregular union and preserves the columns. She repeats this with the other three tables. Petition 870250077578, dated 01 / 09 / 2025, page 49 / 202 43 / 88 present in all years, and then for the remaining tables. When it has finished, the final stage of its flow produces 19 separate tables.

[0148] Once she has this, she will try to reconstruct the data. It seems that the common union key should be a column called ST_CASE, but just by looking at the profile panel for the accident table, she can see that this is not a key column anywhere. ST_CASE is not a key, but by clicking on years, she can easily verify that there is only one ST_CASE per year. Together, year and ST_CASE seem like a satisfactory union key.

[0149] It starts with the “person” table. Before it can perform the join on this table, it needs the year in each of its tables, which is not present. However, since the file paths include the year, it can select this data in the data pane and choose “Extract as a new column”. The system infers the correct pattern for this, and extracts the year for each row. It then selects both tables in its flow, selects the year and ST_CASE columns as one, and drags them to the other table, creating a join.

[0150] Now that she has the keys, she continues to create joins to flatten the FARS data. When she is finished, she publishes the data as a TDE (Tableau Data Extract) to her Tableau Server so that her team can use it. Scenario 3: FARS Cleanup

[0151] Colin is another employee in the same department as Bonnie. Some people are trying to use the data that Bonnie's stream produces, but it includes many encrypted values. Upon discovering that Bonnie has gone to another company, they turn to Colin.

[0152] Looking at the flow, Colin can easily see its overall logic and also sees the encrypted encoded data. When he finds the 200-page PDF manual containing the lookup tables (LUTs) for the encrypted codes, Petition 870250077578, dated 01 / 09 / 2025, page 50 / 202 44 / 88 The process seems intimidating. An example of a lookup table in the PDF is illustrated in Figure 5B. Some are simpler and others are considerably more complex.

[0153] Colin starts with some of the most important tables. He discovers that it is possible to select the table in the PDF file and paste it into the flow pane 313. In some cases, the data in the table is not entirely correct, but he manages to provide a reasonable result, and Colin can then manually patch the results in the data pane 315, saving considerable time. As he works, he sees his results immediately. If the tables do not align, he can notice it immediately.

[0154] Finally, Colin produces several LUTs that seem particularly relevant to the analysis his team is conducting, and publishes the results so his team can use the data. As people ask for more information about specific columns, Colin can further increase his workflow to generate additional LUTs. Scenario 4: Discovering Data Errors

[0155] Danielle, a developer at a large software firm, is investigating data representing compilation times. Danielle has considerable control over the data format, and has produced it in a good consumable CSV format, but simply wants to load and append it to a database she has created.

[0156] While loading the data, she checks profile view 314. Something immediately seems strange to her: there are a few compilations with negative times. There must be something wrong here, and she wants to investigate the problem, but she also wants to gather the data for analysis.

[0157] She selects the negative times in the profile view, and clicks “Keep only” to keep only the erroneous rows. She adds a target. Petition 870250077578, dated 01 / 09 / 2025, page 51 / 202 45 / 88 to transfer them to a file. She will use these unprocessed rows to guide her in debugging.

[0158] Returning to her flow, she adds another branch just before the filter. Again, she selects the negative values ​​(for example, in profile panel 314 or data panel 315) and then simply presses “delete”. This replaces the values ​​with null, which is a good indicator that the actual value is simply unknown. She proceeds with the rest of her simple flow, appending the compilation data to her database, and will look into the negative values ​​later. Scenario 5: Vehicle parts tracking

[0159] Earl works for a car manufacturer and is responsible for maintaining a dataset that shows the current condition of each vehicle and major part in the factory. The data is reported to several operational repositories, but these operational repositories are quite large. There are hundreds and thousands of parts, and as an automated facility, many thousands of records are mechanically created for each vehicle or part as it moves through the factory. These operational repositories also contain many records that have nothing to do with the condition of the part, but other operational information (e.g., “the pressure at valve 134 is 500 kPa”). There is a business need for a quick and concise record for each part.

[0160] Earl drags the tables for each of the three relational operational repositories to the flow pane 313. Two of them store data as individual tables containing log records. The third has a small star schema that Earl quickly flattens by dragging and dropping to create a union.

[0161] Next, through an additional drag-and-drop operation, Earl is able to quickly perform an irregular join of the tables. As a result, Petition 870250077578, dated 01 / 09 / 2025, page 52 / 202 46 / 88 he can drag and drop columns together and the interface combines the results for him.

[0162] The part identification number is a bit problematic: one system has a hyphen in the value. Earl takes one of the values ​​in data pane 315, selects the hyphen, and presses delete. The interface infers a rule to remove the hyphens from this column, and inserts a rule into the flow that removes the hyphen for all data in that column.

[0163] Earl doesn't want most of the condition codes, as they aren't relevant to this current project. He only wants the condition codes that are related to the parts. He opens a table containing information about condition codes and drops it onto the last node of his flow, resulting in a new join on the condition code. Now he selects only the rows with "target type" equal to "part" and chooses "Keep only" to filter out the other values. This filtering is done in the profile panel 314 or the data panel 315.

[0164] Finally, Earl wants only the last value for each piece. Through a direct gesture, he sorts the data in the data panel by date, groups by piece number, and adds a calculation of “n tables on top” to get only the final update for each piece.

[0165] Earl runs his flow and discovers that it takes four hours to run. However, he knows how he could speed it up. He could record the last time he ran his flow and only incorporate new records in each subsequent run. To do this, however, he needs to update the existing rows in his accumulated set and only add rows if they represent new pieces. He needs a "merge" operation.

[0166] Earl uses the part number to identify matches, and provides actions for when a match occurs or not. With the update logic, Earl's workflow takes only 15 minutes to execute. The savings of Petition 870250077578, dated 01 / 09 / 2025, page 53 / 202 47 / 88 time allows the company to track much more efficiently where the parts are located in its warehouse and what their condition is.

[0167] Earl then sends this task to a server so that it can be scheduled and executed centrally. He could also create a scheduled task on his desktop computer that executes the flow using a command-line interface. Scenario 6: An Investment Broker

[0168] Gaston works at an investment brokerage firm on a team responsible for taking data produced by the IT department and digesting it so that it can be used by various teams working with clients. The IT department produces several datasets that show parts of a client's portfolio—bond positions, equity positions, etc.—but each of them, separately, is not what Gaston's clients need.

[0169] A team, led by Hernine, needs all the extracted customer location data so that their team can answer questions their customers may have when they ask for help. Preparing the data isn't that complex.

[0170] Gaston performs some pre-processing of the nightly database deletions that the IT department produces, merges them together, and performs some simple checks to make sure the data looks okay. He then filters them to exactly what Hermine's team needs and creates a TDE for his team to use.

[0171] With his previous tools, Gaston needed to remember to go and run the workflow every morning. However, with the new 250 data preparation application, this workflow can be handled declaratively. It sends a TDS to Hermine that his team uses, so all data visualizations performed by Hermine's team are executed directly on the database. This means that Petition 870250077578, dated 01 / 09 / 2025, page 54 / 202 48 / 88 Gaston doesn't need to worry about updating the data, and it runs quickly.

[0172] Another team, led by Ian, uses similar data to perform performance analyses of their clients' accounts. To produce this data, Gaston reuses the work he did for Hermine, but filters the data for Ian's team's accounts, and then performs an additional flow to combine the data with various indexes and performance indicators so that Ian's team can perform their analysis. This work is expensive and doesn't seem to perform well in real time. If he runs the flow, it will take several hours to complete – but Ian's team only needs it once a month. Gaston sets up a recurring calendar item on the server to run it once a month. Scenario 7: Debugging Customer Data

[0173] Karl is a strategic account manager for a large software company. He is trying to use Tableau to visualize information about the participants at an industry conference, who they work for, who their representatives are, whether they are active or prospective clients, whether their companies are small or large, and so on.

[0174] Karl has a list of conference attendees, but he’s done this before. The last time he was in this position, it took him 8 hours to clear the list—and 15 minutes to build the visualization after he was finished. That time he’s using the Data Preparation 250 app to speed up and automate the process.

[0175] Karl first wants to clean up the company names. Scanning the data, he sees what he expected: the same company is frequently listed in multiple different formats, and some of them contain spelling errors. He invokes a fuzzy deduplication routine provided in the operations palette to identify potential duplicates. He analyzes the results and Petition 870250077578, dated 01 / 09 / 2025, page 55 / 202 49 / 88 corrects some cases where the algorithm overshot. It also discovers a few cases that the algorithm missed, and then groups them together. This produces a list of customers with canonical company names.

[0176] He then tries to merge his data with a list of companies maintained in a data source on his Tableau Server. He discovers that each company has multiple listings. Multiple different companies may have the same name, and a single company may have multiple accounts based on region.

[0177] To solve this, Karl uses a REST connector for LinkedIn® that he discovered, and passes it to each of the email addresses in his data to retrieve the country and state for each person. This procedure takes the information he has (e.g., the person's name, the person's company, and the person's job title) and uses LinkedIn's search functionality to suggest the best result for each entry. He then combines the company and location data with the data on his server to find the correct account.

[0178] Karl discovers that his union doesn't always work. The canonical company name he chose doesn't always match what's in his account database. He converts his union to a fuzzy union, analyzes the fuzzy matches, and additionally corrects the result manually.

[0179] Now that he has his data cleaned, he opens it in Tableau to create a data visualization.

[0180] Aspects commonly used in flows include: • Multiple levels and unions, joins and aggregations that require the user to have precise control over the logical order of operations. • A layout that was created and annotated by the user to improve understanding. • The need for clarity in the data structure as it progresses through the flow. Petition 870250077578, dated 01 / 09 / 2025, page 56 / 202 50 / 88 • Reusing parts of a stream to produce two different outputs. • An author preparing data for two or more other users, sometimes in separate teams. • Scheduling workflows to be executed automatically. [ 0181] Data preparation applications are sometimes classified as ETL (extract, transform, and load) systems. Each of the three phases performs different types of tasks. [ 0182] In the extraction phase, users extract data from one or more available data sources. Typically, users perform these tasks: • Simply move files. For example, a user can retrieve a file from an FTP source before further processing. • Ingesting data that varies enormously in terms of its structure (e.g., relational, semi-structured, unstructured), format (e.g., structured repository, CSV files, or JSON files), and source (e.g., from a file system or from a formal database). • Read an entire source, or select a portion of it. Partial reads are common, both for extracting data that is more recent or has changed since the last intake, or for sampling or extracting chunks for performance reasons.

[0183] In the transformation phase, users transform data in a wide variety of ways. Transformations generally include these tasks: • Clean the data to correct errors, handle missing or duplicate values, reconcile variant values ​​that should be the same, conform values ​​to standards, and so on. • Augment or enrich data through scalar and table calculations, aggregation, filtering of rows and columns, and transforming rows into columns. Petition 870250077578, dated 01 / 09 / 2025, page 57 / 202 51 / 88 (and vice versa), or incorporation of external data (for example, through geocoding). • Combining multiple sources through joining or junctions (including fuzzy junctions). • Deinterlace multiple data types that have been grouped together (either in rows or columns) for separate processing. • Extracting profiles from the data or metrics about the data to better understand them. [ 0184] During the loading phase, a user stores the results so that they can be analyzed. This includes: • Save data to a Tableau Data Extract (TDE), formatted files (e.g., CSV or Excel), or an external database. • Create snapshots in a schedule. • Attach or update data with new or modified results.

[0185] Once a user has built a flow to prepare data, the user typically needs to: • Schedule the flow to execute at specified times, or in accordance with other flows. • Share the results of a workflow with others. • Share your workflow with others so that they can examine, modify, clone, or manage it. This includes sharing the workflow or data with IT so that IT can improve and manage it.

[0186] The 250 revealed systems offer control to users. In many cases, the data preparation application makes intelligent choices for the user, but the user is always able to reclaim control. Control generally has two different facets: control over the logical ordering of operations, which is used to ensure that the results are correct and Petition 870250077578, dated 01 / 09 / 2025, page 58 / 202 52 / 88 correspond to the user's desired semantics; and the physical control, which is mostly used to ensure performance.

[0187] The revealed data preparation application 250 also offers freedom. However, users can assemble and reassemble their data production components as they wish in order to obtain the data format they need.

[0188] The revealed 250 data preparation applications offer incremental interaction and immediate feedback. When a user performs actions, the system provides feedback through immediate results on the user's data samples, as well as through visual feedback.

[0189] Typically, ETL tools use imperative semantics. That is, a user specifies the details of each operation and the order in which the operations will be performed. This gives the user complete control. In contrast, an SQL database engine evaluates declarative queries and is able to select an optional execution plan based on the data requested by the query.

[0190] Revealed implementations support both imperative and declarative operations, and a user can choose between these two execution options at various levels of granularity. For example, a user might want to exercise complete control of a flow initially while learning about a new dataset. Later, when the user is comfortable with the results, the user can relinquish all or part of the control to the data preparation application in order to optimize execution speed. In some implementations, a user can specify a default behavior for each flow (imperative or declarative) and override the default behavior on individual nodes.

[0191] The revealed implementations can record data in many databases Petition 870250077578, dated 01 / 09 / 2025, page 59 / 202 53 / 88 of different target data, including a TDE, SQL Server, Oracle, Redshift, plain text files, and so on. In some cases, a flow creates a new dataset in the target system. In other cases, the flow modifies an existing dataset by appending new rows, updating existing rows, inserting rows, or deleting rows.

[0192] Errors can occur during the execution of a flow. Errors can include temporary system problems, a known potential error condition in the data for which the user can code corrective action, and implicit constraints that the author did not consider. The revealed implementations generally handle these error conditions automatically when possible. For example, if the same error condition had been encountered in the past, some implementations reapply a known solution.

[0193] Although a flow is essentially a data transformation, implementations allow users to annotate their outputs with declarative modeling information to explain how the outputs can be used, visualized, validated, or combined. Examples include: • Annotations that affect how values ​​are displayed in Tableau, such as default coloring or formatting. • Annotations in a field to indicate units or lineage. • The creation of alternative names (aliases) and groups. • Functional constraints, such as primary and foreign keys between tables. • Domain restrictions, such as requiring a field to be positive.

[0194] The revealed implementations generally include these components: • A front-end area with which users interact to view, build, edit, and execute flows. Petition 870250077578, dated 01 / 09 / 2025, page 60 / 202 54 / 88 • An Abstract Flow Language (AFL). This is an internal language that expresses all the logic in a flow, including connections to sources, calculations and other transformations, modeling operations, and what is done with the rows that are the result of the flow. • An execution mechanism. The mechanism interprets and executes AFL programs. In some implementations, the mechanism runs locally. Queries can be sent to remote servers, but the results and further processing will be done using local resources. In a server environment, the server provides a shared, distributed execution environment for flows. This server can schedule and execute flows from many users, and can automatically parse and schedule AFL flows. • A catalog server, which allows streams to be published to third parties.

[0195] Some data visualization applications are capable of running data preparation flows and can use TDEs or other created datasets to build data visualizations.

[0196] The revealed implementations can also import some data streams created by other applications (for example, created in an ETL tool).

[0197] The implementations allow users to: • Connect and read from a data source, as shown in Figure 6B. • Construct a flow that combines supported operations (see Figure 6A) in arbitrary orders and combinations. • See a reasonable sample of how the data will be transformed at each stage of your flow (for example, in the profile dashboard and the data dashboard). • Create visualizations of the data at each stage of a flow. Petition 870250077578, dated 01 / 09 / 2025, page 61 / 202 55 / 88 • Execute a complete flow locally to produce outputs, such as a TDE or CSV output (see Figure 6C). • Publish a pipeline or TDE result to a Catalog Server. • Import a Tableau Data Source (TDS) created in the Data Preparation Application as an explicit flow.

[0198] With access to a configured Server, a user can: • Sharing a TDE with third parties. • Share a data preparation pipeline (flow) with other users using appropriate security. • Executes a data preparation pipeline in a server environment to produce a TDE manually or on a schedule.

[0199] The output of a node can be directed to more than one of the following nodes. There are two basic cases here. In the first case, the flows diverge and do not converge. When the flows do not converge, there are multiple outputs from the flow. In this case, each branch is effectively a separate query consisting of all predecessors in the tree. When possible, implementations optimize this so that the shared part of the flow is not executed more than once.

[0200] In the second case, the flow converges. Semantically, this means that the rows flow through both paths. Again, stream execution generally does not execute ancestors twice. Note that a single stream can have both of these cases.

[0201] The user interface: • Allows users to create forks in a flow. When a new node is added, a user can specify whether the new node creates a fork in the selected node or is inserted as an intermediate node in the existing sequence of operations. For example, if there is currently a path from node A to node B, and the user chooses to insert a new node at A, the user can select either to create a fork. Petition 870250077578, dated 01 / 09 / 2025, page 62 / 202 56 / 88 second path to the new node when inserting the new node between A and B. • Allows a user to execute individual outputs of a stream instead of the entire stream.

[0202] Users can add filters to a flow of arbitrary complexity. For example, a user can click to add a filter at a point in the flow, and then enter a calculation that acts as a predicate. In some implementations, the calculation expressions are limited to scalar functions. However, some implementations allow more complex expressions, such as aggregations, table calculations, or Level of Detail expressions.

[0203] A user can edit any filter, even if it has been inferred by the system. More specifically, all filters are represented as expressions.

[0204] The profile panel 314 and the data panel 315 offer simple ways to create filters. For example, some implementations allow a user to select one or more data values ​​for a column in the data panel, and then right-click and choose “keep only” or “delete”. This inserts a filter into the flow at the currently selected node. The system infers an expression to implement the filter, and the expression is saved. If the user needs to modify the filter later, it is easier to do so, regardless of whether the later time is just around the corner or a year later.

[0205] In profile panel 314, a user can select a “bucket” that specifies a range of values ​​for a data field. For example, with a categorical field, the range is typically specified as a list of values. For a numeric field, the range is typically specified as a continuous range with an upper or lower limit. A user can select a “bucket” and easily create a filter that selects (or excludes) all rows whose value for the field falls within the range. Petition 870250077578, dated 01 / 09 / 2025, page 63 / 202 57 / 88

[0206] When a user creates a filter based on multiple values ​​in a column or multiple “buckets” for a column, the filter expression uses the OR operator. That is, a row matches the expression if it matches any of the values ​​or ranges.

[0207] A user can also create a filter based on multiple data values ​​in a single row in the data panel. In this case, the filter expression uses the AND operator. That is, only the rows that match all the specified values ​​match the expression. This can be similarly applied to the “buckets” in the profile panel. In this case, a row must match in each of the selected “bucket” ranges.

[0208] Some implementations also allow the creation of a filter based on a plurality of data values ​​that include two or more rows and include two or more columns. In this case, the expression created is in disjoint normal form, with each disjoint corresponding to one of the rows with a selected data value. Some implementations also apply the same technique to range selections in the profile window.

[0209] Note that in each of these cases, a user visually selects data values ​​or “buckets,” and then, with a simple gesture (e.g., right-click plus a menu selection), creates a filter that limits the rows to only the selected values ​​or excludes the selected values. The user does not need to figure out how to write a correct Boolean logic expression.

[0210] As illustrated above with respect to Figures 4A to 4V, a user can create joins. Depending on whether declarative execution is enabled, the join can be sent to a remote server for execution, as illustrated in Figure 9 below.

[0211] Some implementations provide simplified versions or Petition 870250077578, dated 01 / 09 / 2025, page 64 / 202 58 / 88 condensed versions of the flows are presented as nodes and annotations. In some implementations, a user can switch between a full view or a condensed view, or toggle individual nodes to hide or expose the details within the node. For example, a single node might include a series of operations to perform cleanup on certain source files. After several iterations of experimentation with the cleanup steps, they are working well, and the user generally doesn't want to see the details. The detail is still there, but the user is able to hide the clump and see only the condensed version of the node.

[0212] In some implementations, operational nodes that do not expand are grouped together in annotations on the node. Operations such as unions and splits will break the flow with additional nodes. In some implementations, the layout for the condensed view is automatic. In some implementations, a user can reorder the nodes in the condensed view.

[0213] Both the profile pane and the data pane provide useful information about the set of rows associated with the node currently selected in the flow pane. For example, the profile pane shows the cardinalities for various data values ​​in the data (e.g., a histogram showing how many rows each data value has). The distributions of the values ​​are illustrated for multiple data fields. Because of the amount of data illustrated in the profile pane, data retrieval is usually performed asynchronously.

[0214] In some implementations, a user can click on a data value in the profile panel and see the proportional clearing of other items. When a user selects a specific data value, the user interface: • Indicates the selection. • Use proportional cleaning to indicate the correlation with other columns in this table. Petition 870250077578, dated 01 / 09 / 2025, page 65 / 202 59 / 88 • Filters or highlights the associated data pane to show only rows whose values ​​match the selection. (This filters the data displayed in the data pane, and does not create a filter node in the flow pane). • When multiple values ​​are selected in the profile panel, all selected values ​​are displayed and the data panel is filtered accordingly (i.e., filtered for rows that match any of the values).

[0215] In some implementations, rows are not displayed in the data pane unless specifically requested by the user. In some implementations, the data pane is always automatically populated, with the process proceeding asynchronously. Some implementations apply different patterns based on the cardinality of the rows for the selected node. For example, some implementations display the rows when the cardinality is below a threshold and do not display the rows or proceed asynchronously if the cardinality is above the threshold. Some implementations specify two thresholds, designating a set of rows as small, large, or very large. In some implementations, the interface displays the rows for small cardinalities, proceeds to display rows asynchronously for large cardinalities, and does not display the results for very large cardinalities.Obviously, the data pane can only display a small number of rows, which is usually selected by sampling (for example, every nth row). In some implementations, the data pane implements infinite scrolling to accommodate an unknown amount of data.

[0216] The revealed data preparation applications provide a document template that the User Interface natively reads, modifies, and operates on. This template describes the flow for users while also providing a formalism for the UI. The template can be translated into Tableau templates that utilize AQL and the Federated Evaluator to execute. The template also allows Petition 870250077578, dated 01 / 09 / 2025, page 66 / 202 60 / 88 reliable caching and reuse of intermediate results.

[0217] As illustrated in Figure 7A, the data model includes three submodels, each of which describes a flow in its appropriate evaluation stages. The first submodel is a “Loom Doc” 702. (Some implementations refer to the data preparation application as “Loom”).

[0218] A Loom 702 document is the model that describes the flow a user sees and interacts with directly. A Loom 702 document contains all the information necessary to perform all ETL operations and type checking. Typically, the Loom 702 document does not include information that is purely necessary for rendering or editing the flow. A Loom 702 document is constructed as a flow. Each operation has: • A set of properties that describe how it will perform its operations. • Zero or more entries describing which data will be operated on. • Zero or more outputs that describe the data resulting from this operation.

[0219] There are four main types of operations: inbound operations, transformation operations, outbound operations, and container operations.

[0220] Input operations perform the "Extraction" part of the ETL. They connect the flow to a data source and are configured to extract data from that source and expose that data to the flow. Input operations include loading a CSV file or connecting to an SQL database. A node for an input operation typically has zero inputs and at least one output.

[0221] Transformation operations perform the "Transformation" part of ETL. They offer "functional" operations through continuous data flows. Petition 870250077578, dated 01 / 09 / 2025, page 67 / 202 61 / 88 and transform them. Examples of transformation operations include “Create calculation as [HospitalName]-[Year]”, “Filter rows that include hospitalId-HarbourView”, and so on. Transformation nodes have at least one input and at least one output.

[0222] Output operations provide the “Loading” part of ETL. They operate with the side effect of actually updating downstream data sources with the incoming data stream. These nodes have an input, and no output (there are no outputs to subsequent nodes in the stream).

[0223] The container operations group groups other operations into logical groups. These are used to help make flows easier to document. Container operations are exposed to the user as “Nodes” in the flow pane. Each container node contains other flow elements (e.g., a sequence of regular nodes) as well as fields for documentation. Container nodes can have any number of inputs and any number of outputs.

[0224] A data stream represents the actual rows of data that move through the stream from one node to another. Logically these can be viewed as rows, but operationally, a data stream can be implemented in any of several ways. For example, some streams are simply compiled to AQL (Analytical Query Language).

[0225] Extensible operations are operations that the data preparation application does not know how to evaluate directly, so it calls an external process or code. These are operations that are not executed as part of the federated evaluator.

[0226] The 704 logical model is a model that contains all entities, fields, relationships, and constraints. It is built by traversing the flow, and defines the model that is built anywhere in the flow. The fields in the logical model are columns in the results. The entities in the logical model represent tables in the Petition 870250077578, dated 01 / 09 / 2025, page 68 / 202 62 / 88 results, although some entities are composed of other entities. For example, a union has an entity that is a result of other entities. The constraints in the logical model represent additional constraints, such as filters. The relationships in the logical model represent the relationships between the entities, providing sufficient information to combine them.

[0227] The physical model 706 is the third submodel. The physical model includes metadata for caching, including information that identifies whether a flow needs to be re-executed, and also how to directly query the results database to find a flow. The metadata includes: • A hash of the logical model at this point. • A timestamp for each root data source, and when it was last queried. • A path or URI describing where the results data is located.

[0228] This data is used to optimize flows, as well as to allow faster navigation of results.

[0229] The physical model includes a reference to the logical model used to create this physical model (for example, a pointer to a file or data repository). The 706 physical model also includes a Tableau Data Source (TDS), which identifies the data source that will be used to evaluate the model. Typically, this is generated from the 704 logical model.

[0230] The physical model also includes an AQL (Analytical Query Language) query that will be used to extract data from the specified data source.

[0231] As illustrated in Figure 7A, the loom document 702 is compiled (722) to form the logical model 704, and the logical model 704 is evaluated (724) to form the physical model. Petition 870250077578, dated 01 / 09 / 2025, page 69 / 202 63 / 88

[0232] Figure 7B illustrates a 710 file format that is used by some implementations. The 710 file format is used in both local and remote execution. Note that the file format contains both data and streams. In some cases, a stream can create data through copy / paste. In these cases, the data becomes part of the stream. The file format maintains a UI state, separate from the underlying stream. Part of the display is saved with the application. Other parts of the layout are user-specific and are stored outside the application. The file format can be versioned.

[0233] The file format has a multi-document format. In some implementations, the file format has three main parts, as illustrated in Figure 7B. In some implementations, the file format 710 includes editing information 712. This section is responsible for making the editing experience continuous across devices and editing sessions. This section stores any data segments that are not needed to evaluate a flow, but are needed to reconstruct the UI for the user. Editing information 712 includes Undo History, which contains a persistent temporary undo storage that allows the user to undo operations after an editing session has been closed and reopened. Editing information also includes a UI State, such as which panels are visible, x / y coordinates of flow nodes, which are not reflected in how a flow is executed.When a user reopens the UI, they see what was there before, making it easier to resume work.

[0234] The 710 file format includes a 702 Document Loom, as described above with respect to Figure 7A. This is the only section of the file format that is required. This section contains the stream.

[0235] The 710 file format also includes 714 local data, which contains any tables or local data needed to run a flow. These Petition 870250077578, dated 01 / 09 / 2025, page 70 / 202 64 / 88 data can be created through user interactions, such as pasting an HTML table into the data preparation application, or when a flow uses a local CSV file that needs to be sent to a server for evaluation.

[0236] The Evaluation Subsystem is illustrated in Figure 7C. The evaluation subsystem provides a reliable way to evaluate a flow. The evaluation subsystem also provides an easy way to operate on the results of a previous execution or to layer operations on the operation of a flow. In addition, the evaluation subsystem provides a natural way to reuse the results of one part of the flow while executing later parts of the flow. The evaluation subsystem also provides a fast way to execute compared to cached results.

[0237] There are two basic contexts for evaluating a flow, as illustrated in Figure 7C. When executing (740) a flow, the process evaluates the flow and dumps the results to the output nodes. If running in debug mode, the process writes the results to temporary databases that can be used for navigation, analysis, and execution of partial flows more quickly.

[0238] In navigation and analysis (730), a user is investigating a dataset. This may include closely examining data distributions, searching for dirty data, and so on. In these scenarios, the evaluator usually avoids running the entire flow, and instead runs faster queries directly against temporary databases created from previously running the flows.

[0239] These processes take advantage of good metadata around caching in order to ensure that intelligent caching decisions are possible.

[0240] Some implementations include an asynchronous subsystem, as illustrated in Figure 7D. The asynchronous subsystem offers non-asynchronous behavior. Petition 870250077578, dated 01 / 09 / 2025, page 71 / 202 65 / 88 blocking the user. If the user is performing a series of operations that do not require retrieving the rows, the user is not prevented from obtaining them. The asynchronous subsystem provides incremental results. Generally, a user would not need the complete dataset to begin validating or trying to understand the results. In these cases, the asynchronous subsystem delivers the best results as they arrive. The asynchronous subsystem also provides a reliable "cancel" operation for ongoing queries.

[0241] In some implementations, the asynchronous model includes four main components: • A browser layer. This layer obtains a UUID and an updated version from the asynchronous tasks it initiates. It then uses the UUID to obtain updates. • A REST API. This layer initiates tasks in a set of execution threads (thread pool). The tasks in the thread pool update the Condition Services as they receive updates. When the browser layer wants to know if there are updates, it calls a REST API procedure to obtain the latest condition. • An AqlAPI. This layer is called as if it were a synchronous call that had callbacks. The call will only end when the underlying request has finished. However, the callbacks allow updates to the Condition Service with the already processed rows. This allows providing incremental progress to the user. • A federated evaluator. The AqlApi makes a call to the federated evaluator, which provides another layer of asynchronicity, as it runs as a new process.

[0242] The implementation of cancellation operations depends on where the cancellation occurs. At the browser layer, it is easy to send a request to Petition 870250077578, dated 01 / 09 / 2025, page 72 / 202 66 / 88 cancellation, and then interrupt the query for results. In the REST API, it's easy to send a cancellation event to a thread that is running.

[0243] Some implementations make it safe and easy to “refactor” a flow after it has already been created. Currently, ETL tools allow people to create flows that initially seem quite simple, but become impossible to change as they grow larger. This is because it is difficult for people to understand how their changes will affect the flow and because it is difficult to decompose blocks of behavior into fragments that are related to business requirements. Much of this is caused by the user interface, but the underlying language needs to provide the information that the UI requires.

[0244] The revealed implementations allow users to create flows that can be easily refactored. What this means is that users can take operations or nodes and easily: • Move operations freely, logically reordering them. The implementations provide direct feedback on whether these operations create errors. For example, suppose a user has a flow with ADD_COLUMN -> FILTER. The user can drag the FILTER node before the ADD_COLUMN node, unless the FILTER uses the column that was added. If the FILTER uses the new column, the interface immediately generates an error, informing the user of the problem. • Gather a series of operations and nodes into a new node (which can be reused). This new node will have a “type” that it accepts and a “type” that it returns. For example, suppose a user has a section of a flow that includes JOIN_TABLES -> ALTER_COLUMN -> ALTER_COLUMN -> ALTER_COLUMN. The implementations allow a user to combine these four steps into one node and assign the node a meaningful name, such as “FIXUP_CODES”. The new node takes Petition 870250077578, dated 01 / 09 / 2025, page 73 / 202 67 / 88 takes two tables as inputs and returns one table. The types of the input tables would include the columns on which they were joined, and any columns that ended up being used in ALTER_COLUMNS. The type of the output table is the type that results from the operations. • Splitting operations from a node. This is where a user can reorganize operations that were organically added to a node during immediate operations. For example, suppose a user has a giant node containing 20 operations, and the user wants to split the 10 operations related to correcting hospital codes into their own node. The user can select these nodes and extract them. If there are other operations in the node that depend on the operations being removed, the system displays the error and suggests a correction or the creation of a new node after the FixupHospitalCodes node. • Alignment of operations on an existing node. After a user has performed a certain cleanup, there may be some work that belongs to another part of the flow. For example, as a user cleans up Insurance Codes, they find some problems in Hospital Codes and perform the cleanup. Then, they want to move the Hospital Code cleanup to the FixupHospitalCodes node. This is done using a simple drag-and-drop operation. If the user tries to drop the operation to a location in the flow before an operation it depends on, the interface provides immediate visual feedback that the proposed drop location does not work. • Change a type and immediately find out if it breaks parts of the flow. A user can use a flow and then decide to change the type of one or more of the columns. The implementations immediately inform the user about any problems, even before running the flow.

[0245] In some implementations, when a user is refactoring a flow, the system assists with identifying drop points. For example, Petition 870250077578, dated 01 / 09 / 2025, page 74 / 202 68 / 88 If a user selects a node and starts dragging it within the flow pane, some implementations display locations (e.g., by highlighting) where the node can be moved.

[0246] The revealed data preparation applications use a language that has three aspects: • An expressive language. This is how users define calculations. • A data flow language. This is how users define the inputs, transformations, relationships, and outputs of a flow. These operations directly alter the data model. The types in this language are entities (tables) and relationships instead of mere individual columns. Users don't see this language directly, but use it indirectly through node creation and UI operations. Examples include table joins and column removal. • A control flow language. These are operations that can happen around the data flow, but are not actually the data flow. Examples include copying a zip file from a file share and then unzipping it, taking a recorded TDE and then copying it to a share, or executing a data flow through an arbitrary list of data sources.

[0247] These languages ​​are distinct, but they fit together in layers on top of each other. The expression language is used by the flow language, which, in turn, can be used by the control flow language.

[0248] The language describes a flow of operations that logically goes from left to right, as illustrated in Figure 8A. However, because of the way the flow is evaluated, the actual implementation may reorder the operations for better performance. For example, moving filters to remote databases as data is extracted can greatly improve the Petition 870250077578, dated 01 / 09 / 2025, page 75 / 202 69 / 88 overall execution speed.

[0249] Data flow language is the language most people associate with data preparation applications, as it describes the flow and relationships that directly affect ETL. This part of the language has two main components: models and nodes / operations. This is different from standard ETL tools. Instead of a flow operating directly on the data (e.g., making the actual rows go from a “filter” operation to an “add field” operation), revealed flows define a logical model that specifies what it wants to create and a physical model that defines how it wants to materialize the logical model. This abstraction offers greater leeway in terms of optimization.

[0250] Models are the basic nouns. They describe the schema and relationships of the data being targeted by operations. As noted above, there is a logical model and a separate physical model. A Logical Model provides the basic “type” for a flow at a given point. It describes the fields, entities, and relationships that describe the data being transformed. This model includes things like sets and groups. The logical model specifies what is desired, but not any materialization. The core parts of this model are: • Fields: These are the actual fields that will be transformed into data fields in the output (or that assist the calculations that do this). Each field is associated with an entity and an expression. Not all fields necessarily need to be visible. There are 3 types of fields: physical fields, calculated fields, and temporary fields. Physical fields are materialized in the resulting dataset. These can be either proper fields or calculations. Calculated fields are written to the resulting TDS as calculated fields, so they will never be materialized. Temporary fields are written to better factor calculations for a physical field. They are not written in any way. If a temporary field is referenced by a Petition 870250077578, dated 01 / 09 / 2025, page 76 / 202 70 / 88 calculated field, the language will issue an alert and treat this field as a calculated field. • Entities: These are the objects that describe the namespace for the logical model. Entities are created either by the schema of a received table, or they can be composed of a set of entities that are associated with each other by relationships. • Relationships: These are objects that describe how different entities relate to each other. They can be used to combine multiple entities into a new composite entity. • Restrictions: These describe restrictions added to an entity. Restrictions include filters that actually limit the results for an entity. Some restrictions are enforced. Enforced restrictions are guaranteed by an upstream source, such as a unique restriction, or a non-null restriction. Some restrictions are claimed. These are restrictions that are believed to be true. Whenever data that violates such a restriction is found, the user is notified in some way.

[0251] A flow can include one or more branches in the logical model. A branch of a flow uses the same Logical Model for each branch. However, there are new hidden entities for each side of the branch. These entities basically pass directly to the original entities, unless a column is projected or removed from them.

[0252] One reason for creating new entities is to keep track of any relationships between entities. These relationships will continue to be valid when none of the fields change. However, if a field is modified, it will be a new field in the new entity, causing the known relationship to no longer work.

[0253] Some implementations allow attaching a node or operation. The Petition 870250077578, dated 01 / 09 / 2025, page 77 / 202 71 / 88 flows describe the logical ordering for a set of operations, but the system is free to optimize processing by making the physical ordering different. However, a user may wish to ensure that the logical and physical orderings are exactly the same. In these cases, a user can "pin" a node. When a node is pinned, the system ensures that operations before pinning physically occur before operations after pinning. In some cases, this will result in some form of materialization. However, the system transmits this in a flow whenever possible.

[0254] The physical model describes a materialization of the logical model at a specific point. Each physical model has a backreference to the logical model that was used to generate it. Physical models are important for caching, incremental flow executions, and load operations. A physical model includes a reference to any file containing results from a flow, which is a unique hash describing the logical model up to that point. The physical model also specifies the TDS (Tableau Data Source) and AQL (Analytical Query Language) generated for an execution.

[0255] Nodes and operations are the basic verbs. Nodes in the model include operations that define how data is modeled, calculated, and filtered. In order to remain consistent with UI language, the term “operations” refers to one of the “nodes” in a flow that does something. Nodes are used to refer to containers that hold operations, and map to what the user sees in the flow panel in the UI. Each specialized node / operation has properties associated with it that describe how it will operate.

[0256] There are four basic types of nodes: input operations, transformation operations, output operations, and container nodes. Input operations create a logical model from some external source. Examples include an operation that imports a CSV. Input operations represent the AND in ETL. Petition 870250077578, dated 01 / 09 / 2025, page 78 / 202 72 / 88 (Extraction). Transformation operations transform a logical model into a new logical model. A transformation operation receives a logical model and returns a new logical model. Transformation nodes represent the T in ETL (Transformation). An example is a project operation that adds a column to an existing logical model. Output operations receive a logical model and materialize it into some other data repository. For example, an operation that receives a logical model and materializes its results into a TDE. These operations represent the L in ETL (Loading). Container nodes are the base abstraction around how composition is done between flows, and also provide an abstraction for what should be illustrated as the nodes are illustrated in the UI.

[0257] As illustrated in Figure 8B, the type system consists of three main concepts: Operations are atomic actions, each having inputs and outputs, as well as a set of required fields. • Required fields are fields that are necessary for an operation. Required fields can be determined by evaluating the operation with an empty environment, then gathering any of the fields that are "assumed". • Type Environments are the constructs that determine how to query types for a given point in a flow. Each "edge" in the flow diagram represents a type environment.

[0258] Type checking is performed in two phases. In the type environment creation phase, the system executes the flow in the direction of the flow. The system determines which types are needed for each node, and which type environments they generate. If the flow is abstract (for example, if it does not actually connect to any input node), the type environment is used. Petition 870250077578, dated 01 / 09 / 2025, page 79 / 202 73 / 88 empty. Type refinement is the second phase. In this phase, the system takes the type environments from the first phase and "reverses the flow" of them to check if any of the type reductions that occurred during the creation of the type environment created type conflicts. In this phase, the system also creates a set of required fields for the entire subflow.

[0259] Each operation has a type environment associated with it. This environment contains all the accessible fields and their types. As illustrated in Figure 8C, a type environment has five properties.

[0260] An environment can be either “Open” or “Closed”. When an environment is Open, it is assumed that there may be fields of which it is unaware. In this case, any unknown field will be assumed to be of any type. These fields will be added to the AssumedTypes field. When an environment is Closed, it is assumed that it knows all fields, so fields that it does not know are considered a failure.

[0261] All known types are in the Types member. This is a mapping of field names to their types. The type can be either another type environment or it can be a Field. A field is the most basic type.

[0262] Each field is composed of two parts. basicTypes is a set of types that describes the possible set of types for the field. If this set has only one element, then we know what type it has. If the set is empty, then there was a type error. If the set has more than one element, then there are multiple possible types. The system can resolve and perform further type reduction if necessary. derivedFrom is a reference to the fields from which this was derived.

[0263] Each field in a scope has a set of potential types. Each type can be any combination of Boolean, String, Integer, Decimal, Petition 870250077578, dated 01 / 09 / 2025, p. 80 / 202 74 / 88 Date, DateTime, Double, Geometry, and Duration. There is also an “Any” type, which is shorthand for a type that can be anything.

[0264] In the case of open Type Environments, there may be instances where fields are known not to exist. For example, after a “removeField” operation, the system may not know all the fields in a Type Environment (since it is open), but the system knows that the newly removed field is not there. The Type Environment property “NotPresent” is used to identify such fields.

[0265] The AssumedTypes property is a list of types that have been added, as they were referenced rather than defined. For example, if there is an expression [A] + [B] that is evaluated in an open type environment, the system assumes there were two fields: A and B. The AssumedTypes property allows the system to keep track of what has been added in this way. These fields can be accumulated for further type filtering, as well as to determine the required fields for a container.

[0266] The “Previous” type environment property is a reference to the type environment from which this was derived. It is used for type refinement stages, during the reverse traversal through the flow searching for type inconsistencies.

[0267] Type environments can also be composite. This occurs in operations that take multiple inputs. When a type environment is merged, it will map each type environment to a value in its type set. Additional type resolution is then delegated to the individual type environments. It will then be up to the operator whether to transform that type environment into the output type environment, usually by “flattening” the type environment in some way to create a new type environment that only has fields as types.

[0268] This is used by the Join and Union operators in order to accurately utilize all fields from different environments in their own Petition 870250077578, dated 01 / 09 / 2025, page 81 / 202 75 / 88 expressions, and having a way to map the environment to an output type environment.

[0269] The type environment created by an input node is the schema returned by the data source it is reading. For an SQL database, this will be the schema of the table, query, stored procedure, or view it is extracting. For a CSV file, this will be the schema that is extracted from the file, with any types that a user has associated with the columns. Each column and its type is transformed into a field / type mapping. Additionally, the type environment is marked as closed.

[0270] The type environment for a transformation node is the environment for its input. If it has multiple inputs, they will be merged to create the type environment for the operation. The output is a single type environment based on the operator. The table in Figures 8J-1 to 8J-3 lists many of the operations.

[0271] A container node can have multiple entries, so its type environment is a composite type environment that directs the appropriate child type environments to the appropriate exit nodes. When a container is extracted for reuse, it resolves with empty type environments for each entry to determine its dependencies.

[0272] In some implementations, a container node is the only node type that can have more than one output. In this case, it can have multiple output type environments. This should not be confused with output branching, which can occur with any node. However, in the case of output branching, each of the output edges has the same type environment.

[0273] There are a few instances where type errors are flagged when the system discovers conflicting requirements for a field. Unresolved fields are not treated as errors at this stage, as this stage can occur along flows with unbound inputs. However, if a user Petition 870250077578, dated 01 / 09 / 2025, page 82 / 202 76 / 88 attempts to execute a flow, unresolved variables would be a reported problem.

[0274] Many of the entries have specific type definitions. For example, specific definitions include using CHAR(10) instead of VARCHAR(2000), which collation a field uses, or what scale and precision a decimal type has. Some implementations do not track these as part of the type system, but track them as part of the runtime information.

[0275] The UI and the middle layer are able to obtain them from the types at runtime. This information can flow through the regular callback, as well as be incorporated into the types for tempdb (for example, if the system is performing the population from a cached execution). The UI shows users the most specific known types, but does not perform type checking based on these. This allows the creation of OutputNodes that use more specific types, while allowing the rest of the system to use the more simplified types.

[0276] Figures 8D illustrate simple type checking based on a flow with all known data types. Figures 8E illustrate a simple type failure with fully known types. Figure 8F illustrates simple type environment calculations for a partial flow. Figure 8G illustrates types of a packed container node. Figure 8H illustrates a more complicated type environment scenario. Figure 8I illustrates the reuse of a more complicated type environment scenario.

[0277] Some implementations infer data types and use the inferred data types to optimize or validate a data flow. This is particularly useful for text-based data sources, such as XLS or CSV files. Based on how a data element is subsequently used in a flow, a type of Petition 870250077578, dated 01 / 09 / 2025, page 83 / 202 77 / 88 data can sometimes be inferred, and the inferred data type can be used earlier in the flow. In some implementations, a data element received as a text string can be cast as the appropriate data type immediately after retrieval from the data source. In some cases, data type inference is recursive. That is, by inferring the data type for a data element, the system is able to infer the data types of one or more additional data elements. In some cases, a data type inference is able to discard one or more data types without determining an exact data type (for example, determining that a data element is numeric, but not being able to determine whether it is an integer or a floating-point number).

[0278] Most type errors are found in the type checking stage. This arises right after calculating the initial type environments, and refines the scopes based on what is known about each type.

[0279] This phase begins with all environments of the terminal type. For each environment type, the system walks backward to its previous environments. The process walks backward until it reaches a closed environment or an environment with no previous environments. The process then checks the types in each environment to determine if any fields differ in type. In this case, and if their intersection is null, the process generates a type error. If any of the fields diverge in type and the intersection is not null, the process sets the type to be the intersection and any affected nodes have their type environments recalculated. In addition, any types that are "assumed" are added to the previous type environment and the type environments are recalculated.

[0280] There are a few subtleties that are tracked. Firstly, field names, by themselves, are not necessarily unique, as a user can overwrite a field with something of a different type. As a result, Petition 870250077578, dated 01 / 09 / 2025, page 84 / 202 78 / 88 The process uses a type pointer to backtrack to the types used to generate it, thus avoiding being misled by unrelated things that resolve to the same name in different parts of the graph. For example, suppose a field A has a type [int, decimal], but then there is a node that performs a project that converts A to a string. It would be an error to go back to previous versions of A and say that the type doesn't work. Instead, backtracking at this point will not backtrack A beyond the addField operation.

[0281] Type checking reduces one variable at a time. In the steps above, type checking is applied to only one variable before recalculating the known variables. This is to be safe in case there is an overloaded function with multiple signatures, such as Function1(string, int) and Function1(int, string). Suppose this is called Function1([A], [B]). The process determines that the types are A: [String, int] and B: [String, int]. However, it would be invalid for the types to resolve to A:[String] and B:[String], because if A is a String, B must be an int. Some implementations manage this type dependency by re-performing the type environment calculation after each type reduction.

[0282] Some implementations optimize which work will be done by performing the work only on the nodes that actually have a necessary field that includes the reduced variable. There is a slight subtlety here, since reducing A may end up causing B to be reduced as well. Let's take the example of Function1 above. In these cases, the system needs to know when B was changed and check its reduction as well.

[0283] When we look at how operators will act, it is best to think of them in terms of four main properties, identified here as “Is Open”, “Multi-Input”, “Input Type” and “Resulting Type”.

[0284] An operation is designated as Open when it flows through the Petition 870250077578, dated 01 / 09 / 2025, page 85 / 202 79 / 88 columns. For example, "filter" is an open operation, since any column that is in the input will also be in the output. Grouping by is not open, since any column that is not aggregated or grouped will not be in the resulting type.

[0285] The “Multi-Input” property specifies whether this operation takes multiple input entities. For example, a join is multi-input because it takes two entities and combines them into one. A union is another operation that is multi-input.

[0286] The “Input Type” property specifies the type that the node requires. For a multi-input operation, this is a composite type where each input contains its own type.

[0287] The “Resulting Type” property specifies the type of output that results from this operation.

[0288] The tables in Figures 8J-1, 8J-2, and 8J-3 indicate the properties for many of the most frequently used operators.

[0289] In many cases, a flow is created over time as needs change. When a flow grows through organic evolution, it can become large and complex. Sometimes, a user needs to modify a flow, either to address a changing need or to reorganize a flow so that it is easier to understand. Such refactoring of a flow is difficult or impossible in many ETL tools.

[0290] The implementations here not only enable refactoring, but help the user to do it. At a technical level, the system can obtain the RequireFields for any node (or sequence of nodes), and then illuminate drop points at any point that has a type environment that can accommodate it.

[0291] Another scenario involves reusing existing nodes in a flow. For example, suppose a user wants to take a string of operations and create a custom node. The custom node operates to “normalize Petition 870250077578, dated 01 / 09 / 2025, page 86 / 202 80 / 88 insurance codes. The user can create a container node with a series of operations on it. The system can then calculate the necessary fields for it. The user can save the node for future use, either using a save command or dragging the container node to the left panel 312. Now, when a person selects the node from the palette in the left panel, the system highlights the drop points in the flow, and the user can drop the node at one of the drop points (for example, just like the refactoring example above).

[0292] ETL can become complicated, therefore the implementations here allow for various system extensions. Extensions include • User-Defined Flow Operations. Users can extend a data flow with Input, Output, and Transformation operations. These operations can use custom logic or analytical data to modify the contents of a row. • Control Flow Scripts. Users can embed scripts that perform non-data flow operations, such as downloading a file from a shared location, unpacking a file, running a flow for each file in a directory, and so on. • Execution of Command-Line Scripts. Users can run their workflows from a command line.

[0293] The implementations here use an approach that is language-indifferent in terms of how people use the extensibility provided.

[0294] A first extension allows users to build custom nodes that fit into a flow. There are two parts to creating an extension node: • Define the output type. For example, “everything that came plus the new column 'foo'”. Petition 870250077578, dated 01 / 09 / 2025, page 87 / 202 81 / 88 • Provide the script or executable to actually perform the transformation.

[0295] Some implementations define two types of nodes that allow user-defined extensions. A “ScriptNode” is a node where the user can write the script to manipulate rows and pass them back. The system offers API functions. The user can then write a transformation node (either input or output) as a script (e.g., in Python or Javascript). A “ShellNode” is a node where the user can define an executable program to run and pipe the rows to the executable. The executable program will then write the results to stdout, write errors to stderr, and exit when finished.

[0296] When users create extensions to the flows, the internal processing is more complex. Instead of compiling everything into an AQL statement, the process splits the evaluation into two segments around the custom node, and routes the results of the first segment to the node. This is illustrated in Figures 8K and 8L, where the user-defined node 850 splits the flow into two parts. During the flow evaluation, the user-defined script node 852 receives data from the first part of the flow, and provides the output to the second part of the flow.

[0297] In addition to customizations that modify the data stream in certain ways, users can write scripts that control how a stream is executed. For example, suppose a user needs to retrieve data from a shared folder that has spreadsheets published to it each day. A defined stream already knows how to handle CSV or Excel files. A user can write a control script that iterates over a remote shared folder, extracts the relevant files, and then performs execution on those files.

[0298] There are many common operations, such as pattern joining, that a user can add to a data flow node. However, as technology continues to evolve, there will always be ways to obtain or store data. Petition 870250077578, dated 01 / 09 / 2025, page 88 / 202 82 / 88 that are not accommodated by the data flow nodes defined by the system. These are the cases where control flow scripts are applicable. These scripts are executed as part of the flow.

[0299] As noted above, streams can also be invoked from the command line. This will allow staff to incorporate the scripts into other processes or nightly tasks.

[0300] The implementations have a flow evaluation process that offers many useful features. These features include: • Execute a workflow from start to finish. • Decompose a flow in order to ensure the order or "fixing" operations. • Decompose a flow to allow the execution of third-party code. • Execute a flow, but instead of traversing the entire path back to the upstream data sources, have it escape the output of a previously executed flow. • Pre-execute some parts of the flow to populate local cache memories.

[0301] The evaluation process works based on the interaction between logical models and physical models. Any materialized physical model can be the starting point of a flow. However, the language runtime provides the abstractions to define which subsections of the flows will be executed. In general, the runtime does not determine when it will execute subflows or complete flows. This is determined by other components.

[0302] Figure 8M illustrates that the execution of an entire flow begins with implicit physical models at the input and output nodes. Figure 8N illustrates that the execution of a partial flow materializes a physical model with the results. Figure 8O illustrates the execution of part of a flow based on the results. Petition 870250077578, dated 01 / 09 / 2025, page 89 / 202 83 / 88 previous.

[0303] Although physical models can be reordered to optimize processing, logical models hide these details from the user, as they are generally not relevant. The flow evaluator makes it appear as if the nodes are evaluated in the order in which they are presented in the flow. If a node is affixed, it will actually cause the flow to materialize there, ensuring that the segment on the left is evaluated before the one on the right. In a forked flow, the common pre-flow is executed only once. The process is idempotent, meaning that input operators can be called again due to a failure and not fail. Note that there is no requirement that the returned data be exactly the same as it would have been the first time (i.e., when the data in the upstream data source has changed between the first and second attempts).

[0304] The execution of transform operators has no side effects. On the other hand, extraction operators generally have no side effects. Any operation that modifies the data sources before entering the flow is not seen until the next execution of the flow. Load operators generally have no side effects, but there are exceptions. In fact, some load operators require side effects. For example, extracting files from a shared folder and unpacking them are considered side effects.

[0305] Some implementations are case-sensitive with respect to column names, but other implementations are not. Some implementations provide a user-configurable parameter to specify whether column names are case-sensitive.

[0306] In general, views of cached objects always go “forward” in time. Petition 870250077578, dated 01 / 09 / 2025, page 90 / 202 84 / 88

[0307] Figures 8P and 8Q illustrate the evaluation of a flow with an affixed node 860. During flow evaluation, nodes before the pin are executed first to create the user node results 862, and the user node results 862 are used in the last part of the flow. Note that affixing does not prevent reordering of execution within each of the parts. An affixed node is effectively a logical checkpoint.

[0308] In addition to nodes that are affixed by a user, some nodes are inherently affixed based on the operations they perform. For example, if a node makes a call to custom code (e.g., a Java process), logical operations cannot be moved through the node. The custom node is a “black box,” therefore its inputs and outputs must be well defined.

[0309] In some cases, moving operations can improve performance, but creates a side effect of reducing consistency. In some cases, a user may use pinning as a way to ensure consistency, but at the cost of performance.

[0310] As noted above, a user can edit data values ​​directly in data grid 315. In some cases, the system infers a general rule based on the user's edit. For example, a user might add the sequence “19” to the data value “75” to create “1975”. Based on the data and the user's edit, the system might infer a rule that the user wants to fill in the character sequence to form 4-character years for 2-character years where the century information is missing. In some cases, the inference is based solely on the change itself (e.g., adding “19” at the beginning), but in other cases, the system also bases the inference on the data in the column (e.g., that the column has values ​​in the range of “74” to “99”). In some implementations, the user is prompted to confirm the rule before applying it. Petition 870250077578, dated 01 / 09 / 2025, page 91 / 202 85 / 88 applies the rule to other data values ​​in the column. In some implementations, the user can also choose to apply the same rule to other columns.

[0311] User edits to a value can include adding an existing data value as just described, removing a portion of a character string, replacing a certain subsequence with another subsequence, or any combination thereof. For example, phone numbers can be specified in a variety of formats, such as (XXX)YYY-ZZZZ. A user can edit a specific data value to remove the parentheses and the hyphen and add periods to create XXX.YYY.ZZZZ. The system can infer the rule based on a single instance of editing a data value and apply the rule to the entire column.

[0312] As another example, numeric fields can also have inferred rules. For example, if a user replaces a negative value with zero, the system can infer that all negative values ​​should be zeroed.

[0313] In some implementations, a rule is inferred when two or more data values ​​are edited in a single column of the data grid 315 according to a shared rule.

[0314] Figure 9 illustrates how a logical flow 323 can be executed in different ways, depending on whether the operations are designated as imperative or declarative. In this flow, there are two input datasets, dataset A 902 and dataset B 904. In this flow, these datasets are retrieved directly from the data sources. According to the flow, the two datasets 902 and 904 are combined using a join operation 906 to produce an intermediate dataset. After the join operation, flow 323 applies a filter 908, which creates another intermediate dataset, with fewer rows than the first intermediate dataset created by the join operation 906. Petition 870250077578, dated 01 / 09 / 2025, page 92 / 202 86 / 88

[0315] If all nodes in this flow are designated as imperative, executing the flow does exactly what the nodes specify: datasets 902 and 904 are retrieved from their data sources, these datasets are combined locally, and then the number of rows is reduced by the filter.

[0316] If the nodes in this flow are designated to have declarative execution (which is usually the default), the execution optimizer can reorganize the physical flow. In a first scenario, assume that datasets 902 and 904 come from distinct data sources and that filter 908 applies only to the fields in dataset A 902. In this case, the filter can be passed back to the query that retrieved dataset A 902, thus reducing the amount of data retrieved and processed. This can be particularly useful when dataset A 902 is retrieved from a remote server and / or the filter eliminates a substantial number of rows.

[0317] In a second scenario, declarative execution is again assumed, but it is assumed that both dataset A 902 and dataset B 902 are executed from the same data source (for example, each of these datasets corresponds to a table in the same database on the same database server). In this case, the stream optimizer can send the entire execution back to the remote server, constructing a single SQL query that joins the two tables and includes a WHERE clause that applies the filter operation specified by filter node 908. This execution flexibility can greatly reduce the overall execution time.

[0318] A user builds and modifies a data stream over time, so some implementations offer incremental stream execution. Intermediate results for each node are saved and recalculated only when necessary.

[0319] To determine if a node needs to be recalculated, some Petition 870250077578, dated 01 / 09 / 2025, page 93 / 202 87 / 88 implementations use a stream hash and a vector clock. Each node in stream 323 has its own stream hash and vector clock.

[0320] A stream hash for a given node is a hash value that identifies all operations in the stream up to and including that node. If any aspect of the stream definition has changed (for example, adding nodes, removing nodes, or changing operations on any of the nodes), the hash will be different. Note that the stream hash simply follows the stream definition and does not look at the underlying data.

[0321] A vector clock tracks the versioning of data used by a node. It is a vector because a given node can use data from multiple sources. Data sources include any data source accessed by any node up to and including the given node. The vector includes a monotonically increasing version value for each of the data sources. In some cases, the monotonically increasing value is a timestamp of the data source. Note that the value corresponds to the data source, not to when the data was processed by any nodes in the flow. In some cases, a data source may provide the monotonically increasing version value (for example, the data source has edit timestamps). If the data source cannot provide a version number like this, the Data Preparation Application 250 calculates a substitute value (for example, when the query was sent to or retrieved from the data source).In general, it is preferable to have a version value that indicates when the data was last modified rather than a value that indicates when the data preparation application last queried the data.

[0322] Using stream hash and vector clock, the 250 data preparation application limits the number of nodes that need to be recalculated.

[0323] The terminology used in the description of this invention serves only to Petition 870250077578, dated 01 / 09 / 2025, page 94 / 202 88 / 88 is intended to describe specific implementations and is not intended to limit the present invention. As used in the description of the invention and the appended claims, the singular forms "a," "an," and "or" are intended to include plural forms as well, unless otherwise indicated by the context. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should also be understood that the terms "comprises" and / or "comprising," when used in this descriptive report, specify the presence of mentioned aspects, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more different aspects, steps, operations, elements, components, and / or groups thereof.

[0324] The preceding description, for explanatory purposes, has been described with reference to specific implementations. However, the above illustrative discussions are not intended to be exhaustive or to limit the invention to the exact forms disclosed. Many modifications and variations are possible in light of the above teachings. The implementations have been chosen and described in order to best explain the principles of the invention and its practical applications, thereby enabling other individuals skilled in the art to make the best use of the invention and various implementations with diverse modifications as appropriate to the specific use contemplated. Petition 870250077578, dated 01 / 09 / 2025, page 95 / 202

Claims

1 / 8 CLAIMS 1. Computer system (200) for preparing data for subsequent analysis CHARACTERIZED in that it comprises: one or more processors (202); memory (214); and one or more instruction sets stored in memory (214) and configured for execution by the one or more processors (202), or one or more instruction sets comprising instructions for: displaying a user interface (100) that includes a simultaneous display of a data flow panel (303, 313), a tool panel (302, 312), a profile panel (304, 314) and a data panel (305, 315), wherein: the data flow panel (303, 313) displays a flowchart (323) including a plurality of linked nodes, each node specifying a respective operation and a respective intermediate data set generated by executing the respective operation;the tool panel (302, 312) includes a data source selector displaying possibilities for locating and connecting to data sources (240); the profile panel (304, 314) displays schemas of intermediate datasets corresponding to nodes selected in the flowchart (323), including data elements showing information about data fields of the intermediate datasets corresponding to the selected nodes and statistical information about data values ​​for the data fields, where user interaction with any of the data elements in the profile panel (304, 314) modifies the flowchart (323) displayed in the data flow panel (303, 313);and the data panel (305, 315) displays rows of data from the intermediate datasets corresponding to the nodes selected in the flowchart (323), where user interaction with individual data values ​​from the displayed data rows modifies the flowchart (323) displayed in the data flow panel (303, 313); detect a first user input directed to a data element in the profile panel (304, 314) or to a data value in the data panel (305, 315); In response to the detection of the first user input, modify the flowchart (323) to include a new node, in accordance with an operation corresponding to the first user input, and select the new node in the flowchart (323), where the operation is performed asynchronously to update the profile panel (304, 314) and the data panel (305, 315);wherein each node of the flowchart (323) has a corresponding flow hash in the form of a hash value that identifies all operations in the flow specification up to and including the respective node, wherein the hash value of the flow hash of a respective node of the nodes changes when the flow specification up to and including the said node changes, and wherein the operation corresponding to the respective node is recalculated only if the hash value of the flow hash of the respective node has been changed.; 2. Computer system (200), according to claim 1, CHARACTERIZED in that the data elements showing information about data fields displayed in the profile panel (304, 314) include data ranges for a first data field.

3. Computer system (200), according to claim 2, CHARACTERIZED by, in response to a first user action on a first data range for the first data field in the profile panel (304, 314): identifying the operation to be executed as a filter operation; and adding a new node to the flowchart (323) that filters data for the first data range. Petition 870250077578, dated 01 / 09 / 2025, p. 97 / 202 3 / 8 4. Computer system (200), according to claim 2, CHARACTERIZED in that user interaction in the profile panel (304, 314) mapping data ranges to the first data field for specified values ​​adds a new node to the flowchart (323) that performs the mapping.

5. Computer system (200), according to claim 1, CHARACTERIZED in that, in response to a first user interaction with a first data value from the data rows displayed in the data panel (305, 315), a new node is added to the flowchart (323) that filters data for the first data value.

6. Computer system (200), according to claim 1, CHARACTERIZED in that, in response to a user modification of a first data value of a first data field in the data panel (305, 315), a new node is added to the flowchart (323) that performs the modification in each data row whose data value for the first data field is equal to the first data value.

7. Computer system (200), according to claim 1, CHARACTERIZED in that, in response to a first user action on a first data field in the data panel (305, 315), a new node is added to the flowchart (323) that splits the first data field into two or more separate data fields.

8. Computer system (200), according to claim 1, CHARACTERIZED in that, in response to a first user action in the data flow panel (303, 313) to drag a first node to the tool panel (302, 312), a new operation is added to an operations palette, the new operation corresponding to the first node.

9. Computer system (200), according to claim 1, CHARACTERIZED in that the profile panel (304, 314) and the data panel Petition 870250077578, dated 09 / 01 / 2025, page 98 / 202 4 / 8 (305, 315) are configured to update asynchronously as selections are made in the data flow panel (303, 313).

10. Computer system (200), according to claim 1, CHARACTERIZED in that the profile panel (304, 314) displays one or more histograms that include distributions of data values ​​for data fields.

11. Computer-readable non-transient storage medium CHARACTERIZED in that it stores one or more sets of instructions configured for execution by a computer system (200) containing one or more processors (202), memory (214) and a display medium (208), or one or more sets of instructions comprising instructions for: displaying a user interface (100) that includes a simultaneous display of a data flow panel (303, 313), a tool panel (302, 312), a profile panel (304, 314) and a data panel (305, 315), wherein: the data flow panel (303, 313) displays a flowchart (323) including a plurality of linked nodes, each node specifying a respective operation and a respective set of intermediate data generated by executing the respective operation; the tool panel (302, 312) includes a data source selector displaying possibilities for locating and connecting to data sources (240);The profile panel (304, 314) displays schemas of intermediate datasets corresponding to nodes selected in the flowchart (323), including data elements showing information about data fields of the intermediate datasets corresponding to the selected nodes and statistical information about data values ​​for the data fields, wherein user interaction with any of the data elements in the profile panel (304, 314) modifies the flowchart (323) displayed in the data flow panel (303, 313); the data panel (305, 315) displays rows of data from the datasets. Petition 870250077578, dated 01 / 09 / 2025, page. 99 / 202 5 / 8 intermediates corresponding to the nodes selected in the flowchart (323), where user interaction with individual data values ​​from the displayed data rows modifies the flowchart (323) displayed in the data flow panel (303, 313);detect a first user input directed to a data element in the profile panel (304, 314) or to a data value in the data panel (305, 315); in response to the detection of the first user input, modify the flowchart (323) to include a new node, in accordance with an operation corresponding to the first user input, and select the new node in the flowchart (323), where the operation is executed asynchronously to update the profile panel (304, 314) and the data panel (305, 315);wherein each node of the flowchart (323) has a corresponding flow hash in the form of a hash value that identifies all operations in the flow specification up to and including the respective node, wherein the hash value of the flow hash of a respective node of the nodes changes when the flow specification up to and including the said node changes, and wherein the operation corresponding to the respective node is recalculated only if the hash value of the flow hash of the respective node has been changed.; 12. Computer-readable non-transient storage medium according to claim 11, CHARACTERIZED by, in response to a first user action on a first data range for a first data field in the profile panel (304, 314): identifying the operation to be executed as a filter operation; and adding a new node to the flowchart (323) that filters data for the first data range.

13. Computer-readable non-transient storage medium according to claim 11, CHARACTERIZED in that user interaction in the profile panel (304, 314) mapping data ranges to a first data field for specified values ​​adds a new node to the flowchart (323) that performs the mapping.

14. Computer-readable non-transient storage medium according to claim 11, CHARACTERIZED in that, in response to a first user interaction with a first data value from the data rows displayed in the data panel (305, 315), a new node is added to the flowchart (323) that filters data for the first data value.

15. Computer-readable non-transient storage medium according to claim 11, CHARACTERIZED in that, in response to a first user action on a first data field in the data panel (305, 315), a new node is added to the flowchart (323) that splits the first data field into two or more separate data fields.

16. Method for preparing data for subsequent analysis CHARACTERIZED in that it comprises: in a computer system (200) provided with a display medium (208), one or more processors (202), and memory (214) storing one or more sets of instructions configured for execution by the one or more processors (202): displaying a user interface (100) that includes a simultaneous display of a data flow panel (303, 313), a tool panel (302, 312), a profile panel (304, 314) and a data panel (305, 315), wherein: the data flow panel (303, 313) displays a flowchart (323) including a plurality of linked nodes, each node specifying a respective operation and a respective set of intermediate data generated by executing the respective operation;the tool panel (302, 312) includes a data source selector Petition 870250077578, dated 01 / 09 / 2025, page 101 / 202 7 / 8 displaying possibilities for locating and connecting to data sources (240); the profile panel (304, 314) displays schemas of intermediate datasets corresponding to nodes selected in the flowchart (323), including data elements showing information about data fields of the intermediate datasets corresponding to the selected nodes and statistical information about data values ​​for the data fields, where user interaction with any of the data elements in the profile panel (304, 314) modifies the flowchart (323) displayed in the data flow panel (303, 313);the data panel (305, 315) displays rows of data from intermediate datasets corresponding to the nodes selected in the flowchart (323), wherein user interaction with individual data values ​​from the displayed data rows modifies the flowchart (323) displayed in the data flow panel (303, 313); detect a first user input directed to a data element in the profile panel (304, 314) or to a data value in the data panel (305, 315); in response to the detection of the first user input, modify the flowchart (323) to include a new node, in accordance with an operation corresponding to the first user input, and select the new node in the flowchart (323), wherein the operation is performed asynchronously to update the profile panel (304, 314) and the data panel (305, 315);wherein each node of the flowchart (323) has a respective flow hash in the form of a hash value that identifies all operations in the flow specification up to and including the respective node, wherein the hash value of the flow hash of a respective node of the nodes changes when the flow specification up to and including the said node changes, and wherein the operation corresponding to the respective node is recalculated only if the hash value of the flow hash of the respective node has been changed. Petition 870250077578, dated 01 / 09 / 2025, page 102 / 202 8 / 8; 17. Method according to claim 16, CHARACTERIZED by, in response to a first user action on a first data range for a first data field in the profile panel (304, 314): identifying the operation to be performed as a filter operation; and adding a new node to the flowchart (323) that filters data for the first data range.

18. Method according to claim 16, CHARACTERIZED in that user interaction in the profile panel (304, 314) mapping data ranges to a first data field for specified values ​​adds a new node to the flowchart (323) that performs the mapping.

19. Method according to claim 16, CHARACTERIZED in that, in response to a first user interaction with a first data value from the data rows displayed in the data panel (305, 315), a new node is added to the flowchart (323) that filters data for the first data value.

20. Method according to claim 16, CHARACTERIZED in that, in response to a first user action on a first data field in the data panel (305, 315), a new node is added to the flowchart (323) that splits the first data field into two or more separate data fields. Petition 870250077578, dated 01 / 09 / 2025, pp. 103 / 202

Images