Data Preparation Using Semantic Roles

Abstract

A method for preparing data for subsequent analysis. The method obtains a data model of a tree that encodes a data source into logical tables. Each logical table has its own physical representation and includes logical fields. Each logical field corresponds to a data field or calculation across logical tables. The method associates each logical table in the data model with a corresponding concept in a concept graph. The concept graph embodies hierarchical inheritance of semantics. For each logical field, the method assigns a semantic role to the logical field based on the concept corresponding to the logical table. The method also validates the logical field based on the semantic role of the logical field. The method also shows transformations to clean the logical field based on the validation of the logical field. The method transforms the logical field according to user selection and updates the logical table.

Description

[0001] Related Applications

[0002] This application is related to U.S. Patent Application No. 16 / 234,470, filed on December 27, 2018, entitled "Analyzing Underspecified Natural Language Utterances in a Data Visualization User Interface", which is incorporated herein by reference in its entirety.

[0003] This application is also related to U.S. Patent Application No. 16 / 221,413, filed on December 14, 2018, entitled "Data Preparation User Interface with Coordinated Pivots", which is incorporated herein by reference in its entirety.

[0004] This application is also related to U.S. Patent Application No. 16 / 236,611, filed on December 30, 2018, entitled "Generating Data Visualizations According to an Object Model of Selected Data Sources", which is incorporated herein by reference in its entirety.

[0005] This application is also related to U.S. Patent Application No. 16 / 236,612, filed on December 30, 2018, entitled "Generating Data Visualizations According to an Object Model of Selected Data Sources", which is incorporated herein by reference in its entirety. Technical Field

[0006] The disclosed implementations generally relate to data visualization, and more particularly to systems, methods, and user interfaces for preparing and conditioning data for use in data visualization applications.

[0007] Background

[0008] Data visualization applications enable users to visually understand data sets, including distributions, trends, outliers, and other factors important for making business decisions. Some data sets are very large or complex and include many data fields. Various tools can be used to assist in understanding and analyzing the data, including control panels with multiple data visualizations. However, the data often needs to be manipulated or transformed to put it in a format that can be readily used by data visualization applications.

[0009] Overview

[0010] The disclosed implementations provide methods for cleaning and / or replacing data values in a dataset based on semantic roles of data fields, which can be used as part of a data preparation application.

[0011] According to some implementations, a method prepares data for subsequent analysis. The method is executed at a computer having a display, one or more processors, and a memory that stores one or more programs configured to be executed by the one or more processors. The method includes obtaining a data model of a tree that encodes a first data source as a logical table. Each logical table has its own physical representation and includes a corresponding one or more logical fields. Each logical field corresponds to a data field or a computation that spans one or more logical tables. Each edge of the tree connects two related logical tables. The method also includes associating each logical table in the data model with a corresponding concept in a concept graph. The concept graph (e.g., a directed acyclic graph) embodies the hierarchical inheritance of the semantics of the logical tables. The method also includes, for each logical field included in a logical table, assigning a semantic role to the logical field based on the concept corresponding to the logical table. The method also includes validating the logical field based on the assigned semantic role of the logical field. The method also includes displaying one or more transformations in a user interface on the display to clean (or filter) the logical field based on the validation of the logical field. In response to detecting a user input to select a transformation to transform the logical field, the method transforms the logical field according to the user input and updates the logical table based on the transformation of the logical field.

[0012] In some implementations, the method also includes, for each logical field, storing the assigned semantic role of the each logical field to the first data source (or an auxiliary data source).

[0013] In some implementations, the method also includes generating a second data source based on the first data source and, for each logical field, storing the assigned semantic role of the each logical field to the second data source.

[0014] In some implementations, the method also includes, for each logical field, retrieving a representative semantic role (e.g., a semantic role assigned to a similar logical field) from a second data source different from the first data source. Assigning the semantic role to the logical field is also based on the representative semantic role. In some implementations, a user input is detected from a first user, and the method also includes, before retrieving the representative semantic role from the second data source, determining whether the first user is authorized to access the second data source.

[0015] In some implementations, the semantic role includes the domain of the logical field, and validating the logical field includes determining whether the logical field matches one or more domain values of the domain. The method further includes determining the one or more transformations based on the one or more domain values before displaying the one or more transformations.

[0016] In some implementations, the semantic role is a validation rule (e.g., a regular expression) for validating a logical field.

[0017] In some implementations, the method further includes, in a user interface, displaying a first one or more semantic roles for the first logical field based on a concept corresponding to a first logical table including the first logical field. The method further includes, in response to detecting a user input selecting a preferred semantic role, assigning the preferred semantic role to the first logical field. In some implementations, the method further includes selecting a second one or more semantic roles for a second logical field based on the preferred semantic role. The method further includes displaying the second one or more semantic roles for the second logical field in the user interface. In response to detecting a second user input selecting a second semantic role from the second one or more semantic roles, the method includes assigning the second semantic role to the second logical field. In some implementations, the method further includes training one or more prediction models based on a data source of one or more semantic tags (e.g., a data source of data fields with assigned or tagged semantic roles). The method further includes determining the first one or more semantic roles by inputting a concept corresponding to the first logical table to the one or more prediction models.

[0018] In some implementations, the method further includes detecting a change to a first data source. In response to detecting the change to the first data source, the method includes updating a concept graph according to the change to the first data source, and repeating assignment, validation, display, transformation, and update for each logical field according to the updated concept graph. In some implementations, the detection of the change to the first data source is performed at a predetermined time interval.

[0019] In some implementations, the logical field is a calculation based on a first data field and a second data field. Assigning a semantic role to the logical field is further based on a first semantic role corresponding to the first data field and a second semantic role corresponding to the second data field.

[0020] In some implementations, the method includes determining a default format of a data field corresponding to the logical field. Assigning a semantic role to the logical field is further based on the default format of the data field.

[0021] In some implementations, the method further includes selecting default format options for displaying the logical fields based on the assigned semantic roles and storing the default format options in a first data source.

[0022] In some implementations, the method further includes, before assigning semantic roles to logical fields, displaying a concept map and one or more options for modifying the concept map in a user interface. In response to detecting user input for modifying the concept map, the method includes updating the concept map according to the user input.

[0023] In some implementations, the method further includes determining a first logical field to add to a first logical table based on concepts of the first logical table. The method further includes displaying a recommendation for adding the first logical field in a user interface. In response to detecting user input for adding the first logical field, the method includes updating the first logical table to include the first logical field.

[0024] In some implementations, the method further includes determining a second data set corresponding to a second data source to join with a first data set corresponding to the first data source based on the concept map. The method further includes displaying a recommendation for joining the second data set with the first data set of the first data source in a user interface. In response to detecting user input for joining the second data set, the method further includes creating a join between the first data set and the second data set and updating the tree of the logical table.

[0025] In some implementations, a computer system has one or more processors, a memory, and a display. One or more programs include instructions for performing any of the methods described herein.

[0026] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by a computer system having one or more processors, a memory, and a display. One or more programs include instructions for performing any of the methods described herein.

[0027] Accordingly, methods, systems, and graphical user interfaces are disclosed that enable a user to analyze, prepare, and organize data. Brief Description of the Drawings

[0029] For a better understanding of the aforementioned systems, methods, and graphical user interfaces, as well as additional systems, methods, and graphical user interfaces that provide data visualization analysis and data preparation, reference should be made to the following description of implementations in conjunction with the accompanying drawings, in which like reference numerals refer to corresponding parts throughout the drawings.

[0030] Figure 1 A graphical user interface used in some implementations is shown.

[0031] Figure 2 is a block diagram of a computing device according to some implementations.

[0032] Figure 3A and Figure 3B shows a user interface of a data preparation application according to some implementations.

[0033] Figure 4 shows an example conceptual diagram according to some implementations.

[0034] Figure 5A shows an example semantic service architecture according to some implementations.

[0035] Figure 5B is a schematic diagram showing synchronization between modules with read / write data roles according to some implementations.

[0036] Figure 6A is an example code snippet showing ranking heuristics based on usage statistics according to some implementations.

[0037] Figure 6B is an example data visualization 610 of a user query without using semantic information according to some implementations.

[0038] Figure 6C is for Figure 6B an example data visualization 630 of a user query using semantic information shown in

[0039] Figure 6D shows an example query according to some implementations.

[0040] Figure 6E provides an example of automatically generating suggestions according to some implementations.

[0041] Figure 6F shows an example usage data according to some implementations.

[0042] Figure 6G shows an example inference according to some implementations.

[0043] Figure 6H shows an example usage statistic according to some implementations.

[0044] Figure 6I shows an example suggestion regarding a natural language query according to some implementations.

[0045] Figure 6J shows an example of a smarter suggestion based on usage statistics according to some implementations.

[0046] Figure 6K A table of example implementations for obtaining an interface for usage statistics is shown according to some implementations.

[0047] Figure 7A A UML model of a data role that stores domain values using a data source is shown according to some implementations.

[0048] Figure 7B An example process for dispatching data roles is shown according to some implementations.

[0049] Figure 7C An example user interface for validating data is shown according to some implementations.

[0050] Figure 7D An example user interface for improved search using semantic information is shown according to some implementations.

[0051] Figure 7E An example user interface for controlling permissions to access data roles is shown according to some implementations.

[0052] Figure 8 An example user interface for previewing and / or editing cleansing recommendations is shown according to some implementations.

[0053] Figures 9A - 9D An example user interface for resource recommendations based on semantic information is shown according to some implementations.

[0054] Figures 10A - 10N A flowchart of method 1000 for preparing data for subsequent analysis is provided according to some implementations.

[0055] Implementations will now be described with reference to examples shown in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details.

[0056] Description of Implementations

[0057] Figure 1Shows a graphical user interface 100 for interactive data analysis. According to some implementations, the user interface 100 includes a data tab 114 and an analysis tab 116. When the data tab 114 is selected, the user interface 100 displays a scenario information area 110, also referred to as a data pane. The scenario information area 110 provides named data elements (e.g., field names) that can be selected and used to create data visualizations. In some implementations, the list of field names is divided into a group of dimensions (e.g., categorical data) and a group of measures (e.g., numerical quantities). Some implementations also include a list of parameters. When the analysis tab 116 is selected, the user interface displays analysis functions instead of a list of data elements (not shown).

[0058] The graphical user interface 100 also includes a data visualization area 112. The data visualization area 112 includes a plurality of shelf areas, such as a column shelf area 120 and a row shelf area 122. These are also referred to as a column toolbar 120 and a row toolbar 122. As shown here, the data visualization area 112 also has a large space for displaying visual graphics. Since no data elements have been selected, this space initially has no visual graphics. In some implementations, the data visualization area 112 has a plurality of layers referred to as tables.

[0059] Figure 2 Is a block diagram of a computing device 200 that can display the graphical user interface 100 according to some implementations. The computing device can also be used by a data preparation (“data prep”) application 230. Various examples of the computing device 200 include desktop computers, laptop computers, tablet computers, and other computing devices having a display and a processor capable of running a data visualization application 222 and / or a data preparation application 230. The computing device 200 generally includes one or more processing units / cores (CPUs) 202 for executing modules, programs, and / or instructions stored in the memory 214 and thereby performing processing operations; one or more network or other communication interfaces 204; a memory 214; and one or more communication buses 212 for interconnecting these components. The communication bus 212 can include circuitry for interconnecting system components and controlling communication between system components.

[0060] The computing device 200 includes a user interface 206, which includes a display device 208 and one or more input devices or mechanisms 210. In some implementations, the input device / mechanism includes a keyboard. In some implementations, the input device / mechanism includes a "soft" keyboard that is displayed on the display device 208 as needed, enabling the user to "press" the "keys" that appear on the display 208. In some implementations, the display 208 and the input device / mechanism 210 include a touchscreen display (also referred to as a touch-sensitive display).

[0061] In some implementations, the memory 214 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM, or other random access solid-state memory devices. In some implementations, the memory 214 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, the memory 214 includes one or more storage devices located remotely from the CPU 202. The memory 214 or optionally the non-volatile memory device within the memory 214 includes a non-transitory computer-readable storage medium. In some implementations, the memory 214 or the computer-readable storage medium of the memory 214 stores the following programs, modules, and data structures, or subsets thereof:

[0062] · An operating system 216, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0063] · A communication module 218, which is used to connect the computing device 200 to other computers and devices via one or more communication network interfaces 204 (wired or wireless) and one or more communication networks (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0064] ● A web browser 220 (or other application capable of displaying web pages), which enables the user to communicate with remote computers or devices via the network;

[0065] ● A data visualization application 222, which provides a graphical user interface 100 for the user to construct visual graphics. For example, the user selects one or more data sources 240 (which may be stored on the computing device 200 or remotely), selects data fields from the data sources, and uses the selected fields to define a visual graphic. In some implementations, the information provided by the user is stored as a visual specification 228. The data visualization application 222 includes a data visualization generation module 226, which takes user input (such as the visual specification 228) and generates a corresponding visual graphic (also referred to as a "data visualization")

[0066] or “data viz”). The data visualization application 222 then displays the generated visual graph in the user interface 100. In some implementations, the data visualization application 222 executes as a stand-alone application (e.g., a desktop application). In some implementations, the data visualization application 222 executes in a web browser 220 or another application using a web page provided by a web server;

[0067] ● Zero or more databases or data sources 240 (e.g., a first data source 240-1 and a second data source 240-2) used by the data visualization application 222. In some implementations, the data source is stored as a spreadsheet file, a CSV file, an XML file, or a flat file, or is stored in a relational database.

[0068] ● Zero or more semantic models 242 (e.g., a first semantic model 242-1 and a second semantic model 242-2), each semantic model being directly derived from a corresponding database or data source 240. The semantic model 242 represents the database schema and contains metadata about the attributes.

[0069] In some implementations, the semantic model 242 also includes metadata about alternative labels or synonyms of the attributes. The semantic model 242 includes data types (e.g., “text”, “date”, “geospatial”, “boolean”, and “numeric”), attributes (e.g., currency type such as US dollars), and semantic or data roles (e.g., “city” role for a geospatial attribute) of the data fields of the corresponding database or data source 240. In some implementations, the semantic model 242 also captures statistical values (e.g., data distribution, range limits, mean, and cardinality) for each attribute. In some implementations, the semantic model 242 is augmented with a syntactic vocabulary dictionary containing a set of analytical concepts found in many query languages (e.g., average, filter, and sort). In some implementations, the semantic model 242 also differentiates between attributes that are measures (e.g., attributes that can be measured, aggregated, or used in mathematical operations) and dimensions (e.g., fields that cannot be aggregated except by counting). In some implementations, the semantic model 242 includes one or more concept graphs encapsulating the semantic information of the data source 240. In some implementations, the one or more concept graphs are organized as directed acyclic graphs, and / or embody hierarchical inheritance of semantics between one or more entities (e.g., logical fields, logical tables, and data fields). Thus, the semantic model 242 helps to infer semantic roles and assign semantic roles to fields; and

[0070] ● One or more object models 108 that identify the structure of the data source 240. In an object model (or data model), data fields (attributes) are organized into classes, where the attributes within each class have a one-to-one correspondence with each other. The object model also includes many-to-one relationships between classes. In some instances, the object model maps each table in the database to a class, and the many-to-one relationship between classes corresponds to a foreign key relationship between tables. In some instances, the data model of the underlying source does not cleanly map to the object model in such a simple manner, and thus the object model includes information specifying how to transform the raw data into appropriate class objects. In some instances, the raw data source is a simple file (e.g., a spreadsheet) that is transformed into multiple classes.

[0071] In some instances, the computing device 200 stores a data preparation application 230 that can be used to analyze and process data for subsequent analysis (e.g., by the data visualization application 222). Figure 3B An example of a data preparation user interface 300 is shown. As described in more detail below, the data preparation application 230 enables a user to build a process 323.

[0072] Each of the executable modules, applications, or assemblies identified above can be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) identified above need not be implemented as separate software programs, processes, or modules, and thus various subsets of these modules can be combined or otherwise rearranged in various implementations. In some implementations, the memory 214 stores a subset of the modules and data structures identified above. Additionally, the memory 214 can store additional modules or data structures not described above.

[0073] Although Figure 2 the computing device 200 is shown, Figure 2 it is more intended as a functional description of the various features that may exist rather than a structural schematic of the implementations described herein. In practice and as recognized by those of ordinary skill in the art, the items shown separately can be combined and some items can be separated.

[0074] Figure 3AAn overview of the user interface 300 for data preparation is shown, displaying panes that group different functions together. In some implementations, the left pane 312 provides the user with options to locate and connect to data or perform operations on the already selected data. In some implementations, the flow area 313 shows one or more operations performed on the selected data at the nodes (e.g., data manipulation for preparing data for analysis). In some implementations, the profile area 314 provides information about the dataset at the currently selected node (e.g., a histogram of the data value distribution for some data fields in the dataset). In some implementations, the data grid 315 provides the raw data values in the rows and columns of the dataset at the currently selected node.

[0075] Figure 3B A specific example of the user interface 300 for data preparation is provided, showing the user interface elements in each pane. The menu bar 311 includes one or more menus, such as a file menu and an edit menu. Although the edit menu is available, more changes to the flow are performed by interacting with the flow pane 313, the profile pane 314, or the data pane 315.

[0076] In some implementations, the left pane 312 includes a data source palette / selector. The left pane 312 also includes an operation palette that shows operations that can be placed into the flow. In some implementations, the list of operations includes any join (joins of any type and with various predicates), union, convert rows to columns, rename and limit columns, projection of scalar calculations, filter, aggregation, data type conversion, data parsing, merge, fusion, split, aggregation, value replacement, and sampling. Some implementations also support operator creation of sets (e.g., dividing the data values of a data field into sets), binning (e.g., grouping the numeric data values of a data field into a range of groups), and table calculations (e.g., calculating data values for each row, such as percentages of a total, that depend not only on the data values in each row but also on other data values in the table).

[0077] The left pane 312 also includes a palette of other flows that can be incorporated, in whole or in part, into the current flow. This enables the user to reuse parts of a flow to create new flows. For example, if a part of a flow has been created that uses a combination of 10 steps to scrub a certain type of input, that 10-step flow part can be saved and reused in the same flow or in a completely separate flow.

[0078] The Process Pane 313 displays a visual representation (e.g., a node / link flowchart) 323 of the current process. The Process Pane 313 provides an overview of the process for documenting the process. When the number of nodes increases, implementations typically add a scroll box. The need for a scroll bar is reduced by combining multiple related nodes into a super node also known as a container node. This enables the user to more conceptually see the entire process and only allows the user to drill down into details when necessary. In some implementations, when the "super node" is expanded, the Process Pane 313 only shows the nodes within the super node, and the Process Pane 313 has a title identifying which part of the process is being shown. Implementations typically implement multiple hierarchical levels.

[0079] Complex processes may include several levels of node nesting. Different nodes in the flowchart 323 perform different tasks and thus the information inside the nodes is different. Additionally, some implementations display different information depending on whether a node is selected. The flowchart 323 provides an easy, visual way to understand how data is processed and keeps the process organized in a logical way for the user.

[0080] As described above, the Profile Pane 314 includes schema information about the data set at the currently selected node (or nodes) in the Process Pane 313. As shown here, the schema information provides statistical information about the data, such as a histogram 324 of the data distribution for each field. The user can directly interact with the Profile Pane to modify the process 323 (e.g., by selecting a data field for filtering data rows based on the value of a data field). The Profile Pane 314 also gives the user information about the relevant data for the currently selected node (or nodes) and visualizations that guide the user's work. For example, the histogram 324 shows the distribution of the domains for each column. Some implementations use brushing to show how these domains interact with each other.

[0081] The Data Pane 315 displays data rows 325 corresponding to the one or more nodes selected in the Process Pane 313. Each column 326 corresponds to one of the data fields. The user can directly interact with the data in the Data Pane to modify the process 323 in the Process Pane 313. The user can also directly interact with the Data Pane to modify individual field values. In some implementations, when the user makes a change to a field value, the user interface applies the same change to all other values in the same column whose value (or pattern) matches the value the user just changed.

[0082] Sampling of data in the data pane 315 is selected to provide valuable information to the user. For example, some implementations select rows that display the full range of values of a data field, including outliers. As another example, when the user selects a node with two or more data tables, some implementations select rows to assist in joining the two tables. Rows displayed in the data pane 315 are selected to show rows that match and do not match between the two tables. This can be useful in determining which fields are used for joining and / or determining what type of join to use (e.g., inner, left outer, right outer, or full outer).

[0083] Although the user can directly edit the flowchart 323 in the process pane 313, changes to operations are typically done in a more direct manner, directly operating on the data or schema in the profile pane 314 or data pane 315 (e.g., right - clicking on the statistics of a data field in the profile pane to add a column or remove a column from the process).

[0084] Traditional data visualization frameworks rely on the user to interpret the meaning of the data. Some systems understand low - level data constraints like data types, but lack an understanding of what the data represents in the real world. This limits the value such systems provide to the user in two key ways. First, the user needs expertise in each data table to understand its meaning and how best to produce useful visualizations (even curated data sources rarely provide context). Second, the user spends a significant amount of time manually manipulating the data and writing calculations to present the data in a meaningful form.

[0085] Some implementations overcome these limitations by enriching the data model with deeper semantics and providing intelligent automation using these semantics. Such implementations reduce the user's dependence on knowledge and expertise to access meaningful content. Semantics include metadata that helps computationally model what the data represents in the real world. Semantics come in various forms, ranging from exposing relationships between fields to enriching individual rows of data with additional information. In some implementations, row - level semantics include synonyms, geocoding, and / or entity enrichment. In some implementations, field - level semantics include data type, field role, data range type, bin type, default format, semantic role, unit conversion, validation rules, default behavior, and / or synonyms. In some implementations, object - level semantics include object relationships, field calculations, object validation, query optimization, and / or synonyms.

[0086] Field - level semantics

[0087] In some implementations, field-level semantics augment the existing metadata about a field by leveraging richer type information in the context of a single field. In some implementations, field-level semantics exclude knowledge about the relationships between fields or objects. In some implementations, field-level semantics are constructed from field type metadata. Some implementations use semantic role attributes (e.g., geographical role) for data source fields. Some implementations extend field-level semantics by adding support for additional field attributes.

[0088] Unit of measure

[0089] Some implementations add units as attributes of fields (especially measurements) to automate unit conversions, improve formatting, and improve default visualization behavior. Examples of unit scales include: currency ($), duration (hours), temperature (°F), length (km), volume (L), area (square feet), mass (kg), file size (GB), pressure (atm), percentage (%), and rate (km / hour).

[0090] Some implementations apply field-level semantics in different use cases and provide an improved user experience or results in various scenarios. Some implementations use field-level semantics to provide unit conversions in natural language queries. For example, assume a user queries "calls over 3.5 hours". Some implementations provide automatic unit conversions from hours to milliseconds (e.g., in a filter). Some implementations provide unit normalization in a dual-axis visualization. Assume a user compares a Fahrenheit field with a Celsius measurement. In this example, Fahrenheit is automatically converted to Celsius. Similarly, during data preparation, some implementations apply field-level semantics to format inference in calculations. Assume a user creates a calculated field by dividing "distance" (in miles) by "time" (in seconds). Some implementations infer the default format of "miles / second". Some implementations apply field-level semantics in visualizations. For example, assume a user creates a bar chart visualization with height. Some implementations format the measurement (e.g., the axis displays the unit, such as 156 cm). In some implementations, constant conversions (such as miles to kilometers) are encoded in the ontology, but variables like currency are derived from external sources (e.g., a computational knowledge engine).

[0091] Automatic data validation and cleaning

[0092] Some implementations add validation rules as attributes of fields to allow users to more easily identify and clean dirty data. For example, out-of-the-box validation rules include phone numbers, postal codes, addresses, and URLs. Some implementations use field-level semantics to clean dirty data. For example, assume that a user uploads a dataset with an incorrectly formatted address during data preparation. Some implementations automatically detect invalid data rows and suggest a cleaning process (e.g., in Tableau Prep). As another use case, some implementations use field-level semantics to perform field inference when processing natural language user queries. For example, assume a user query of "user sessions For name@company.com". Some implementations automatically detect that the value provided by the user is an email address and infer the "email" field for filtering.

[0093] Default behavior

[0094] Some implementations use other attributes of miscellaneous semantic concepts to automatically improve the default behavior across fields. Some implementations apply field-level semantics to determine the default sorting used to generate data visualizations. Assume a user creates a bar chart visualization using a "priority" field with data values of "high", "medium", and "low". Some implementations automatically sort the values in scalar order rather than alphabetical order. Some implementations apply field-level semantics to determine the default colors used to generate data visualizations. Assume a user creates a visualization of votes cast in an election. When the user visualizes party wins by county, some implementations automatically color the regions by their party colors. Some implementations apply field-level semantics during data preparation to determine the default role. Assume a user uploads a dataset with a primary key. Some implementations automatically set the role of the primary key field to dimension, even if it is a numeric data field.

[0095] Synonyms

[0096] Some implementations use knowledge about what fields and their domain values represent in the real world, and the different names people use for them, to improve the interpretation of natural language queries and improve data discovery through search.

[0097] During natural language processing, some implementations use field-level semantics to identify synonyms. For example, assume a user query of "average order size by category". Some implementations map "order size" to the "quantity" field and display a bar chart visualization showing the average quantity by category. Some implementations use field-level semantics to perform data source discovery. For example, assume a user searches for "customers" in a data visualization server (e.g., Tableau Server). Some implementations determine the data sources that contain data corresponding to "clients", "customers", and "subscribers".

[0098] Object - level semantics

[0099] Some implementations use object-level semantics to extend semantic roles, where new concepts have meaning in the context of a particular object. In this way, some implementations automatically associate natural language, related computations, analysis rules, and constraints with data elements.

[0100] Some implementations associate semantics with data attributes by assigning semantic roles to fields and associating them with concepts. In some implementations, directed acyclic concept graphs are used to represent concepts. In some implementations, concept chains form a hierarchy where each hierarchical level adds new real-world understanding to the data, inheriting the semantics of the previous level.

[0101] Figure 4 An example concept graph 400 according to some implementations is shown. In the example shown, the first node 402 corresponds to the concept "currency", the second node 406 corresponds to the concept "US dollar", the third node 406 corresponds to the concept "market value", and the fourth node 408 corresponds to the concept "stock". The edges connecting the nodes represent the relationships between the concepts. Data fields and / or tables are associated with one or more concepts. Assume that a field is associated with the concept currency. Some implementations infer that the concept currency is associated with the concept US dollar based on the concept graph. Based on this semantic relationship, some implementations indicate a possible unit for the field (in this example, US dollars). In some implementations, concepts are nested. In some implementations, the relationships are hierarchical, meaning that sub-concepts inherit the characteristics of the parent concept. In Figure 4 the example shown, the concept market value inherits the semantic role of the stock concept.

[0102] In some implementations, each semantic concept includes background information that defines the meaning of the concept, which natural language expressions users can use to refer to it, and / or what types of computations users should be able to perform (or prevent from performing). In some implementations, this background information is different in different object contexts - for example, the term "rate" is used differently in a tax or investment context than in a sound frequency or racing context.

[0103] Field calculation

[0104] Some implementations use the meaning of fields to automatically infer the computation of other relevant information that may be semantically meaningful. Some implementations of object-level semantics can infer computed fields to assist users during data preparation. For example, assume that a user publishes a data source with human objects that contain a birth date field. Some implementations automatically suggest adding a computed field named "age". Some implementations automatically interpret natural language queries that refer to age. Some implementations use object-level semantic information to interpret ambiguous natural language queries. For example, assume that a user queries "the largest country". Some implementations automatically filter the top-ranked countries in descending order of population. Some implementations use object-level semantics to interpret natural language queries that contain relationships between fields. For example, assume that a user queries "average event duration", and further assume that duration is not in any data source. Some implementations automatically compute the duration as a function of start and end dates (and / or times).

[0105] Object relationships

[0106] Some implementations use object-level semantics to reason about the relationships between an object and its fields. In some implementations, this reasoning is limited to relationships between pairs of objects. In some implementations, this reasoning is extended to a complete object network to form an entire known data set, such as "Salesforce" or "Stripe". In some implementations, this reasoning is used to make content recommendations, correlate similar data sets, or understand natural language queries.

[0107] Some implementations use object-level semantics to interpret natural language queries to determine relationships between objects. For example, assume that a user queries "messages sent by John". Some implementations determine one or more tables to join (e.g., users and messages). Some implementations determine that a filtering operation should be performed on the relationship joined by a foreign key (e.g., the sender_id foreign key).

[0108] Some implementations use object-level semantics to perform query evaluation optimization. For example, assume that a user evaluates a query on "user count", and further assume that the user has many messages. Some implementations perform an efficient query on the count of different normalized messages.

[0109] Object validation

[0110] Some implementations perform data and / or object validation based on object-level semantics. Some implementations use the context of an object to understand which validations can be applied to a field. Some implementations constrain the analysis to determining the validations to apply to a field based on the context. Some implementations use object-level semantics during data preparation to assist the user. For example, assume the user publishes a data source with earthquake magnitude data. Some implementations detect dirty data (e.g., magnitude < 1). In response, some implementations provide the user with options to clean or filter the data, or automatically clean and / or filter out the bad data.

[0111] Row - level semantics

[0112] Some implementations identify entities at the row level of the data and enrich the identified entities with additional information exported from other data sources by intelligently joining the identified entities together. In some implementations, the enriched data is sourced from existing data sources provided by the customer, or data provided by a data visualization platform (e.g., geocoding data from Tableau) or even data provided by a third party (e.g., public or government data sets).

[0113] Some implementations assist the user during data preparation. For example, assume the user publishes a data source with stock ticker symbols. Some implementations perform entity enrichment using external data. For example, some implementations recommend joining with another data set (provided by the user or derived from an external source) to obtain data about each publicly traded company (e.g., headquarters location). For this example, some implementations then explain questions regarding investments in companies headquartered in Canada.

[0114] Derivation of semantic information

[0115] Some implementations enrich the data model with semantics even when it is not certain how a piece of data should be classified. Some implementations use inferential semantics by defining deterministic rules that are used to infer semantic classifications from existing metadata stored in the data source, such as inferring whether a metric is a percentage by examining the default format of the metric. Some implementations use manual classification and allow the user to manually tag fields by selecting one or more semantic roles from a list of options. Some implementations perform automatic semantic classification by studying patterns in how the user tags their data sources to make recommendations. In some implementations, these recommendations are explicit semantic classifications that the user can override. Some implementations use these patterns for fingerprint fields that are used in similarity-based recommendation algorithms (e.g., "similar fields are typically used this way").

[0116] Global semantic concepts

[0117] Some implementations provide users with an ontology of global semantic concepts for selection when they label their fields. These concepts have universal semantic meanings. For example, concepts such as "currency" or "length" are independent of context, and some implementations make reasonable assumptions about the expected behavior of fields of these types.

[0118] User - defined semantic concepts

[0119] Some implementations start from a stable model that describes semantics and enable users to extend their ontology with custom semantic concepts. Preferably, the valuable concepts are unique to the customer dataset or their business, and / or are reconfigurable.

[0120] For example, a customer chooses to build their own semantic concept package related to "retail". If the user subsequently uploads a dataset and selects to apply the "retail" package, some implementations will automatically suggest which semantic tags can be applied to which fields.

[0121] Through semantic governance, some implementations allow an organization to organize the ontology of semantic packages shared across teams. Some implementations have developed large repositories of domain-specific semantic concepts and created a marketplace where these semantic concepts can be shared among customers.

[0122] Some implementations include modules that provide semantic information. In some implementations, such modules and / or semantic information are configurable. Some implementations automatically detect data roles. Some implementations use a framework that describes the structure and representation of semantic concepts and the architecture and interfaces of semantic services responsible for persisting, accessing, and / or managing semantic concepts. Some implementations use this framework to generate a global semantic concept library (e.g., "default ontology"). Some implementations make the library available for users to manage or edit. Some implementations use examples of semantically tagged data sources to train a prediction model to make recommendations for semantic tagging, thereby reducing the effort required to semantically prepare data for analysis.

[0123] Semantic service architecture

[0124] Figure 5AAn example semantic service architecture 500 according to some implementations is provided. In some implementations, the semantic service 524 runs on a data visualization server 522 (e.g., a Tableau server either on-premises, online, or in the cloud) and / or on a data preparation server (e.g., a Tableau Prep server). In some implementations, the semantic service 524 is responsible for persisting and / or managing semantic concepts used by the data role service 512 and related features. In some implementations, the semantic service 524 is written in Go or a similar programming language (e.g., a language that provides memory safety, typing, and concurrency). In some implementations, the semantic service uses a gRPC interface or a similar high-performance remote procedure call (RPC) framework that runs in various environments to connect services within and across data centers. The data role service 512 captures the semantic properties of data that are easy to reuse, share, and govern. In some implementations, the data role represents a content type. In some implementations, the data regarding the data role has two components: (i) content metadata, which is typically stored on a monolith 502 in a data_roles table, and (ii) semantic concept data, which is typically stored at the semantic service 524 (e.g., in a Postgres database 518, an Elasticsearch database 526, and / or a similar analytics engine).

[0125] In some implementations, the data role service 512 (e.g., a Java module) runs in the monolith 502 and is responsible for managing data role content metadata. In some implementations, the data role service 512 is an in-process service and does not listen on any ports. In some implementations, the data role service 512 receives requests from external APIs such as a REST API 504 (used by data preparation), a Web client API 506 (used by the server front end), and a client XML service 508.

[0126] In some implementations, the semantic service 524 is a Go module that runs in an NLP service 522 that provides services such as natural language query processing. In some implementations, the semantic service 524 has an internally exposed gRPC interface used by the data role service 512.

[0127] Authoring Data Roles

[0128] In some implementations, there are two types of data roles - built-in data roles (e.g., country or URL) and custom data roles defined by customers / users.

[0129] In some implementations, the custom data role is only orchestrated in data preparation. In some implementations, the custom data role is also orchestrated in the desktop version of the data visualization software, the server, and / or any environment where the data source is orchestrated or manipulated (e.g., catalog, web orchestration, or querying data).

[0130] According to some implementations, Figure 5A the arrows in [figure] show an example process flow. In some implementations, data preparation sends a request to the REST API service 504. In some implementations, the data role service 512 uses the authorization service 510 to verify the permissions 516 of the request. In some implementations, the data role service 512 saves the content metadata of the data role in the data_roles table in a database (e.g., the Postgres database 518). In some implementations, the data role service 512 sends a request to the semantic service 524 to save the field concept data of the data role. According to some implementations, the field concept data contains the semantic content of the data role. In some implementations, the data role service 512 notifies the search service 514 that the content segment has been updated and needs to be indexed in Solr 520 (or a similar enterprise search platform).

[0131] Match data roles with data fields

[0132] In some implementations, the semantic service 524 provides a gRPC (or similar) interface to expose the function of using the field concept data to detect the data role of a field and semantically enrich / validate it or its value. Some implementations use the field concept data to provide value pattern matching. In this case, the field concept data encodes a regular expression for validating whether a value is valid in the context of the data role. Some implementations use the field concept data to provide name pattern matching. In this case, the field concept data encodes a regular expression for validating whether the name of a data field is valid in the context of the data role. Some implementations use the field concept data to provide value range matching. In this case, the field concept data references the identifier and field name of the published data source, which defines the domain of valid member values for the data role.

[0133] Figure 5BFIG. 530 is a schematic diagram showing synchronization between modules of read and write data roles 532 according to some implementations. In some implementations, if the data role uses value range matching, the semantic service 524 retrieves values from the published data source 536 and indexes them in Elasticsearch 526 to perform the matching. In some instances, the underlying data source may be slow, and a service like ElasticSearch is used to help determine if a value matches any value of any (accessible) data role within a short duration (e.g., less than a millisecond). In some implementations, each data role references the published data source 536 as the ground truth source of the values for that data role. When creating a data role or updating the data source, in some implementations, the semantic service 524 queries the data server to extract the values and indexes them in Elasticsearch 526.

[0134] In some implementations, the data for the data role 532 comes from a data preparation stream 534 and / or workbook 538 with an embedded data source. In some implementations, the published data 540 for the data role 532 is stored in a database 240 (e.g., as part of a semantic model 242).

[0135] Data - driven natural language query processing

[0136] In some implementations, natural language commands and questions provided by the user (e.g., questions asked by the user about information included in a data visualization or published workbook) can be utilized to improve recommendations, the quality and relevance of inferences, and for ambiguity resolution.

[0137] Recommendations

[0138] In some implementations, interpreting user input such as a question or natural language command can include inferring an expression or part of an expression included in the user input. In such cases, one or more recommendations or suggestions can be provided to the user. For example, when the user inputs to select a data source of interest, the automatically generated list of suggestions can include any of the following suggestions: "by neighborhood", "sort neighborhoods alphabetically", "top neighborhoods by sum of record counts", "sum of square feet", "sum of square feet and sum of total host list count as a scatter plot", or "square feet is at least 0". In some cases, the automatically generated suggestions can include suggestions that are unlikely to be relevant to the user.

[0139] In some implementations, one or more models are used to provide recommendations related to a user. For example, when a user selects or identifies a data source of interest, the recommendations may include, for example: one or more top fields in the selected data source, one or more top concepts in the selected data source (e.g., "average" filter), or one or more fully specified sub-expressions (e.g., filtering, sorting, limiting, aggregating). In a second example, when a user selects a data source and a data field in the selected data source, the recommendations may include one or more top sub-expressions (e.g., filtering, sorting, limiting, aggregating) or one or more top values. In a third example, when a user selects a data source and one or more expressions, the recommendations may include one or more top data visualization types. In another example, when a user selects a data source, a data field, and a filter, the recommendations may include one or more top values in the data field that meet the filter criteria. In yet another example, when a user selects a data source and a sub-expression type, the recommendations may include one or more top related sub-expression types.

[0140] Existing visualization context

[0141] In some implementations, one or more models for providing recommendations may consider the data visualization type of the currently displayed data visualization (e.g., bar chart, line chart, scatter plot, heat map, geographic map, or pie chart) and historical usage behavior with similar data visualization types. For example, when a user specifies a data source and a data visualization type, the recommendations may include one or more top fields to add to the existing content. Optionally, the recommendations may include one or more top expressions (e.g., popular filters) to add to a given existing content.

[0142] User context

[0143] In some implementations, one or more models may include one or more users (e.g., user accounts and / or user profiles) to provide recommendations customized for each user's individual behavior. For example, a user in the business team may prioritize the "order date" field, while a member of the shipping and logistics team may prioritize the "ship date" field. When a user in the business team selects a data source, the model may recommend the "order date" field, while when a user in the shipping and logistics team selects the same data source, the model may recommend the "ship date" field instead of the "order date" field, or may recommend the "ship date" field in addition to the "order date" field. Thus, the model can provide personalized recommendations that are most relevant and appropriate for each user.

[0144] Ambiguity resolution

[0145] In some implementations, the natural language input can include conflicting expressions. While a default expression can be selected using heuristics, the default selection may not always be the best choice given the selected data source, the existing visualization context, or the user.

[0146] Some examples of conflict types include:

[0147] · Conflicts between multiple fields

[0148] · Conflicts between multiple values across fields

[0149] · Conflicts between multiple values within a field

[0150] · Conflicts between analytical concepts or expressions

[0151] · Conflicts between a value and a field

[0152] · Conflicts between a value / field and an analytical concept or expression.

[0153] To resolve such conflicts in natural language input, some implementations use various types of weights to select the most appropriate or relevant expression. Some examples of weights include: hard-coded weights for specific expression types, popularity scores for fields, and frequencies of occurrence of values and / or key phrases.

[0154] In some implementations, the weights can be updated based on the frequency of occurrence of an expression in the natural language input and / or in the visualization of a published data visualization workbook.

[0155] For example, when a user provides the natural language input "Avg price seventh ward" while accessing a data source that includes holiday rental information, the recommendations can include any of the following options:

[0156] · Average daily price, filtering the neighborhood to the seventh ward;

[0157] · Average weekly price, filtering the neighborhood to the seventh ward;

[0158] · Average monthly price, filtering the neighborhood to the seventh ward;

[0159] · Average daily price, filtering the host neighborhood to the seventh ward;

[0160] · Average weekly price, filtering the host neighborhood to the seventh ward.

[0161] In a general example, when a user selects a data source and provides a string (such as "Seventh District"), the recommendation can include one or more text fields that include the string as a data value. In another example, when a user selects a data source and a visualization and provides a string, the recommendation can include one or more expressions (e.g., value expressions or field expressions). Similarly, when a user selects a data source and provides a string when logging into an account or profile (so that one or more models can consider personal preferences), the recommendation can include one or more expressions (e.g., value expressions or field expressions). In some implementations, one or more expressions include regular expressions, such as patterns to assist the user in making selections.

[0162] There are instances where heuristics do not adequately resolve conflicting expressions in natural language input. Figure 6A Is an example code snippet 600 showing a sorting heuristic based on usage statistics according to some implementations. For this example, assume that filter FilterTo 604 has a higher usage count than filter AtLeast 602. This could be because there is only one numeric field (salary) in the corresponding data source, while there are many text or geographic fields such as city, state, country, league, player, team. In such cases, some implementations sort the filters accordingly. In this example, filter FilterTo 604 is ranked at least above filter AtLeast 602.

[0163] Figure 6B Is an example data visualization 610 of a user query that does not use semantic information according to some implementations. Assume that the user queries for the maximum salary 612 by league 614 and by country 616. Further assume that the user has not selected a visualization type. In the absence of semantic information, some implementations show maps 622 and 624 corresponding to leagues 618 and 620 respectively.

[0164] Figure 6C Is for according to some implementations Figure 6B Is an example data visualization 630 of a user query that uses semantic information in. Some implementations automatically derive the visualization type (a bar chart 632 in this example) based on the semantic information of the underlying data fields. In this case, data visualization 630 includes bar charts 634 and 636 for leagues 618 and 620 respectively.

[0165] Figure 6DShows an example query according to some implementations. Some implementations track repeated expressions in past queries, and / or track usage counts of expressions in natural language queries, and associate such statistics with data fields (e.g., compensation or alliance) to automatically derive visualization types. In Figure 6D In the example shown, the expression "as a bar chart" 640 appears explicitly multiple times. For this example, when the system (e.g., the parser module) determines that the natural language expression involves compensation or alliance, the system automatically displays a bar chart.

[0166] Figure 6E Shows an example of automatically generating suggestions according to some implementations. Assume that the user queries the transaction amount by merchant type description 642-2 and specifies a filter 642-4 (to filter the merchant type description into a home supply warehouse). Further assume that the user makes a typing error 642-6 when refining the query, or assume that natural language query processing fails to understand the request. Some implementations provide the user with some suggestions for refining the query. In this example, the suggestion 642-8 asks the user if they wish to add "transaction amount at least -20.870". In the absence of semantic information about the transaction amount, natural language processing information allows the filter to use negative amounts.

[0167] Figure 6F Shows example usage data according to some implementations. For the running example, the usage statistics include data about the usage of "transaction amount". From this usage data, some implementations determine that the value "185.05" is frequently used 644 in the user's queries, and thus infer that this value is a more reasonable value for the transaction amount. Some implementations Figure 6E mark the exported "transaction amount" value suggested in

[0168] Figure 6G as a suspicious value instead of inferring a reasonable value. Figure 6H Shows an example of usage statistics according to some implementations. For the current example, based on the usage data 648 of "audit type" (corresponding to "transaction amount"), some implementations determine that "alcohol" is more popular than all other audit type values. Based on this usage data, some implementations determine the inferred value in this instance to be "alcohol".

[0169] Figure 6IShows example suggestions for natural language queries according to some implementations. In the example shown, after aggregating the merchant names 650, various suggestions 652, 654, and 656 fail to provide useful hints to the user. In particular, in this instance, it doesn't make sense to suggest adding a filter to the count of merchant names. On the other hand, suggestions 660 (add fields / filters such as transaction amount of at least 100) or 662 (add salary of at least 0) make more sense. In this way, some implementations apply usage data associated with data fields (e.g., usage statistics stored in semantic roles or semantic information) to improve suggestions.

[0170] Figure 6J Shows an example of smarter suggestions based on usage statistics according to some implementations. Some implementations provide smarter recommendations for complete expressions or partial expressions (as part of a natural language query) provided by the user. In Figure 6J the example shown, assume that the value "Country A" 664 exists in both countries and states, and assume that states are more popular than countries. The first suggestion 666 incorrectly treats Country A as a state. Based on usage, since few people would choose Country A as a state, in some implementations, the natural language parser is able to automatically correct the data source and correctly interpret Country A as a country (as shown in suggestions 668 and 670).

[0171] In some implementations, usage statistics are represented using a data structure that associates lookup maps in order to obtain for each value each data source, and an example of it is shown below:

[0172] UsageStats struct{

[0173] datasourceURI string

[0174] Lookup map[StatKey][]ValueCount

[0175] }.

[0176] Some implementations use one or more interfaces that represent keys for obtaining top values and counts. Figure 6K Table 672 shows an example implementation of an interface for obtaining usage statistics according to some implementations. In the table, the first column 674 corresponds to the respective interfaces, the second column 676 corresponds to the types of statistics supported by the interfaces in the first column, the third column 678 corresponds to the keys passed to the interfaces in the first column, and the last column 680 corresponds to the values returned by the respective interfaces in the first column.

[0177] Some implementations use data structures to represent the values returned by the interfaces explained above with reference to Figure 6K and an example of this is shown below:

[0178] ValueCount of type {

[0179] value interface{}

[0180] count int32

[0181] }}.

[0182] Some implementations convert values to a specific type based on the StatKey type attached to the value.

[0183] Some implementations use interfaces (such as StatKey) in a parser (e.g., a natural language query parser) to obtain values and counts. For example, to obtain the most popular values for the field sales and the filter atLeast, some implementations perform the following:

[0184] values := usageStats.Loopup[NewFieldFilterToValueKey(“sales”, “atLeast”)]

[0185] mostPopularValue := values[0].value.(complexValue).

[0186] Some implementations ensure that the above conversions will succeed and that all values are validated before being added to UsageStats.

[0187] Some implementations use one or more semantic model interfaces. Examples of semantic model interfaces are provided below:

[0188] func(s *UsageStats) GetTopVizTypes(interpretation_nlg string) vizTypes

[0189] []string

[0190] func(s *UsageStats) GetRecommendedExpsForToken(token string, datasource *Datasource) []ExpCount

[0191] type ExpCount struct {

[0192] Type string(Field,,AnalyticalConcept,TextValue)

[0193] Value string(fieldGraphID,ConceptID or textValue)

[0194] Count int32

[0195] }.

[0196] Some implementations interface with the ArkLangData module to retrieve usage statistics (e.g., func(parser)GetStats(exp ArkLangExp)[]stat). Some implementations store natural language processing recommendation records for tracking usage statistics. In some implementations, each row in the recommendation record represents a daily count for an analysis concept and includes time information (e.g., the month the record was created), a data source URI string, a statistic type string, a string representing a key, a data string, and / or a count string. Some implementations also include a data visualization type column for natural language processing-based visualizations.

[0197] Some implementations store natural language processing usage statistics. Some implementations include the data source URI string and usage statistics (e.g., in JSON format) in the statistics.

[0198] Some implementations store performance estimates, such as the number of visualizations over a period of time (e.g., for 90 days). Some implementations store the number of statistics (e.g., 20 statistics) or the range of statistics for each visualization. Some implementations store monthly aggregate counts for natural language processing statistics (e.g., 5 per month means 60,000 records / 5=12,000 records). Some implementations store the number of active data sources (e.g., 200 active data sources) and / or the number of records per data source.

[0199] Various applications of data roles

[0200] In some implementations, knowledge about the real world, when associated with data elements (such as objects, fields, or values), is used to automate or enhance the analysis experience. This knowledge provides an understanding of the semantics of the data fields and is then used to help users clean their data, analyze it, present it effectively, and / or associate it with other data to create a rich data model.

[0201] Some implementations standardize the same data values from different sources or different representations of manually entered data values. In some implementations, the semantics of a field help describe the expected domain values for standardization.

[0202] In some implementations, data knowledge includes concepts that are common across many different contexts, such as geocoding, email, and URLs. Sometimes, these concepts are referred to as global data roles.

[0203] In addition to global data roles, in some implementations, data knowledge also includes concepts related to domain-specific contexts. These roles are referred to as user-defined data roles. In many cases, customer use cases involve non-standard domains, such as product names or health codes. For example, a user can set a custom data role (e.g., a user-defined data role) to help standardize domain values by automatically identifying invalid values and helping the user fix them (e.g., applying fuzzy matching to known values).

[0204] In some implementations, user-defined data roles are only available to a user when they are connected to (e.g., logged in to) a server. In some implementations, the semantics and standardization rules included in a user-defined data role of a first user can be shared via the server with other users for data preparation and analysis. Thus, users connected to the server can share, find, and discover content in the organization, such as user-defined data roles created by other users in the same team, group, or company.

[0205] In some implementations, a user is able to access and reuse a user-defined data role previously defined in another application different from the current application within the current application.

[0206] In some implementations, multiple applications share and utilize a pool of data roles in order to obtain added value unique to the context of each application. Examples of application-specific semantic functions include:

[0207] · Applying a data role to a data field to identify domain values that do not match the role, so that the user can clean the data field;

[0208] · Analyzing a user's data and suggesting matching data roles to apply to the data fields;

[0209] · For a data field with a data role, analyzing the user's data and recommending cleaning transformations to apply to the data field;

[0210] · Once a data field has a data role, selecting good default format options to display the value in the data field;

[0211] ·Clean the user's data fields and save the cleaned data fields as data roles to the server, thereby providing access to the data roles for other users in the user's organization connected to the server for additional data preparation and analysis;

[0212] ·Identify data fields by synonyms entered in the user's query (e.g., natural language commands, questions, or search inputs);

[0213] ·Automatically convert the unit of a data field from its canonical unit to the unit entered in the user's query;

[0214] ·Infer fields from the data role of specific data values entered in the user's query (e.g., user query such as "user sessions for name@company.com");

[0215] ·Infer calculated data fields (e.g., calculate duration from start date and end date in a data source) by understanding the relationship between the data field and the user query goal;

[0216] ·Infer joins between tables by understanding the relationship between the data field and the user query goal. (e.g., natural language input "messages sent by John" results in joining users and messages by sender_id filtered by "John");

[0217] ·The search field can match the field names in the table with the synonyms used in the query.

[0218] ·Cross-flow search can generally handle more expressive queries (e.g., "all input steps connected to a customer");

[0219] ·Create calculated data fields (e.g., calculate duration from start date and end date in a table) by understanding the relationship between the data field and the user query goal;

[0220] ·Create joins between tables from the relationship between the data field and the user query goal (e.g., natural language input "messages sent by users" joins users and messages by sender_id),;

[0221] ·The search data source can match the data field names in the data source with the synonyms used in the query;

[0222] ·Search can generally handle more expressive queries (e.g., "all bug data sources used by at least 5 workbooks");

[0223] ·View and edit a directory of user-defined data roles shared across organizations or companies;

[0224] ·Automatically label axes or legends with units;

[0225] ·Automatically normalize units on dual-axis charts (e.g., compare measurements in Celsius to those in Fahrenheit);

[0226] ·Automatically convert units when performing calculations with values in different units (e.g., add a Fahrenheit temperature to a Celsius temperature);

[0227] ·For bar charts sorted by priority (e.g., "high", "medium", "low"), the default sorting is by the associated scalar values (e.g., 1, 2, 3);

[0228] ·The color coding of regions identified by political party affiliation on a map defaults to the party colors;

[0229] ·Assign data roles to data fields, export the data source to be cleaned to a data preparation application to clean (e.g., remove) invalid values, and import the cleaned data source back into the initial data application or desktop to continue analysis;

[0230] ·Search data sources can match the data field names in the data source with synonyms used in queries;

[0231] ·Searches can generally handle more expressive queries (e.g., "all error data sources used by at least 5 workbooks");

[0232] ·Impact analysis: Find all flows and data sources that contain data fields using a specific data role;

[0233] ·Identify semantically related data fields via an established object model (e.g., when name, address, and ID fields are in a customer object, they are all associated with the customer);

[0234] ·Automatically join data fields and / or data tables using object model relationships;

[0235] ·Suggest a group of semantically related data fields (e.g., it can be suggested to group Product_Name, Product_Code, and Product_Details into a product object); and

[0236] ·During the object model construction phase, suggest relationships based on joins made with other tables.

[0237] In some implementations, data roles have a short - term impact and effect on the user's workflow. For example, data roles are used to automatically detect dirty data in a data preparation application (e.g., flag invalid phone numbers so that the user knows they need to be cleaned). In another example, data roles are used to automatically interpret synonyms in natural language input on a data source in a server (e.g., map "Great Britain" to "United Kingdom"). In yet another example, user - defined data roles that are created are published to the server for shared use.

[0238] In some implementations, data roles have a long - term impact and effect on the user's workflow. For example, data roles are used to recommend or infer calculated data fields on a published data source (e.g., infer "age" when "date of birth" is known). In another example, data roles are used to add support for units of measure (e.g., perform a unit conversion from kilometers to miles in response to receiving a natural language input such as "distance is at least 4,000 km").

[0239] By adopting user - defined data roles, new experiences in authoring, associating, and governing workflows are introduced to the user.

[0240] In some implementations, when a user adds a data source to the user's desktop, data preparation application, or connected server, the relevant data fields in the data source are automatically associated with known (e.g., predefined or previously used) field - level data roles. This association is visible to the user, and the user can choose to override the inferred data role by selecting one from a set of existing data roles. In cases where there are many data roles, the user can search and / or navigate a catalog of options to more easily select concepts relevant to the current data source context and / or the user's own preferences.

[0241] In some implementations, there may be no existing data roles that meet the user's needs. In such cases, the user can author (e.g., create, generate, or customize) new field - level data roles. For example, the user can publish metadata in an existing data field as a data role to a connected server. In some implementations, the metadata includes the name, synonyms, definition, validation rules (e.g., regular expressions), or known domain values of the data field. In some implementations, the user can edit these properties before publishing the data role to the server. In some implementations, the user can also author new field - level data roles from scratch without inheriting properties from existing data fields. In some implementations, the newly authored data roles are saved to a storage device (e.g., a storage device managed by a semantic service), and / or are automatically detected with other data sources. Additionally, the user can choose whether to share their data roles with other users who use applications provided by the same server.

[0242] In some implementations, a user can browse a catalog of curated data roles to view their metadata and track the lineage of the data roles to understand which data sources have elements associated with them. In some implementations, the user can also modify data roles within a conceptual catalog on a connected server. For example, the user can modify existing concepts in the metadata (e.g., add synonyms, change validation rules, change known domain values, etc.), create new concepts (e.g., copy an existing concept with modifications, author a new concept from scratch), eliminate duplicate concepts and update data sources to point to the same concept, delete concepts, and control the permissions for other users on the server to modify the user's data roles.

[0243] Example data analysis use cases

[0244] Some implementations provide data analysis capabilities, examples of which are shown below.

[0245] · The user cleans a data field that includes a list of product names (e.g., using regular expressions (“regex”), using group and replace to map value synonyms to canonical values).

[0246] · The user saves the domain of a data field as a data role (including synonym mapping) so that the user can assign the data role to data fields from other data sources for validating and cleaning other data fields and data sources.

[0247] · The user saves the domain of a cleaned data field as a data role so that other users in his or her organization can use the data role for validation and cleaning.

[0248] · The user uses a data role previously defined in one or more applications to validate and clean other data fields (e.g., automatically or by using natural language input).

[0249] · The user edits a previously defined data role (e.g., to correct an error or update the data role).

[0250] For example, a user can connect to a data source that includes information about product inventory. The user creates a data cleansing step, and the application can suggest that the user apply the data role "product name" to the "prod_name" data field in the data source. For example, although the user has used this data source before, this might be the first time the application has made this suggestion. After accepting the recommendation and applying the suggested data role to the suggested data field, the user sees that some of the product names are not valid names. The user then receives another recommendation to automatically cleanse the data by mapping the invalid names to the corresponding valid names. The user accepts the recommendation, and the invalid values in the data field "prod_name" are replaced with the valid names.

[0251] In another example, the user publishes an existing data source and promotes a data role from one of the data fields in the published data source so that the values in the data field stay in sync (so that the data role is automatically updated when the data source is republished). In some cases, one or more of the data fields in the data source need to be cleansed in some way, and the user creates a data preparation flow to cleanse the data fields and update the published data source. The user publishes the data preparation flow to the server and specifies the data fields for the data role. The user then places the data preparation flow on a weekly refresh schedule so that the data role is updated weekly. In some instances, the user (or a user different from the user who published the data role) retrieves the data role from the server and / or applies the data role to fields in other data preparation flows.

[0252] Example data catalog use cases

[0253] Some implementations provide data cataloging capabilities, examples of which are shown below.

[0254] · The user promotes a data field in a published data source to a data role so that the user can reuse it in another application or with other data sources.

[0255] · The user applies an existing data role to a data field in a published data source so that an application that includes a natural language input interface can associate synonyms and language patterns.

[0256] For example, a user may be using two different data visualizations that display the number of alerts by priority. The user suspects that the two data visualizations use the same data source, but one data visualization has several priority values that are different from the other. The user can use the data catalog to check the lineage of each data visualization and determine, for example, that one data visualization is directly connected to a database, while the other data visualization uses a published data source that is connected to a data preparation flow, which in turn is connected to the same database. The priority field in the published data source has a data role and a set of valid values associated with it, and the data preparation includes a cleaning step that filters out rows with priority values that do not match the data role. The user notifies the author of the first data visualization to consider using the published data source.

[0257] In another example, the user updates the "product name" data role by removing some obsolete product names and adding some new product names. In another example, the user promotes a data role from a data field.

[0258] In some implementations, the data role values are kept in sync with the data fields such that if the user republishes the data source, the data role will be automatically updated. In this way, other analysts can start using it in their data preparation flows. In some implementations, the natural language query processing system creates better insights.

[0259] In another example, the user promotes a data field in a published data source to a data role such that the user can reuse the data role with other data sources or in other applications.

[0260] In another example, the user consults the list of data roles saved on the server to ensure that they are valid. The user can delete any data roles that may be inappropriate (e.g., obsolete or data roles that include incorrect information).

[0261] In another example, the user confirms that a data role containing sensitive data has the correct permissions to make it available only to the intended people. The user can edit the permissions to ensure that the list of people who have access to the data role is up to date.

[0262] In another example, the user edits the synonyms associated with a data role on the server in order to improve the effectiveness of an application that has a natural language input interface with the data source.

[0263] Use cases including natural language input interfaces in applications

[0264] In one example, the user applies an existing data role to a data field in a published data source so that the application can associate synonyms and language patterns (e.g., "like geography").

[0265] In another example, a user provides a natural language command or query that includes a unit different from the units stored in the selected data source. The application uses data roles to automatically convert the data in the data source to the unit specified in the user's natural language input.

[0266] For example, the user can provide the natural language input "Average order size by category". The application maps the phrase "order size" to the "quantity" data field and displays a bar chart data visualization showing the average quantity by category.

[0267] For example, the user can provide a natural language query for "Largest countries". The application creates a data visualization showing the top countries ranked in descending order of population (from most populous to least populous).

[0268] For example, the user can provide a natural language query for "Average event duration", and the term "duration" is not included in the data source. The application calculates the duration based on the start date and end date included in the data source and creates a bar chart data visualization. The duration can also be calculated using the start time and end time.

[0269] Example data roles

[0270] In some implementations, the data roles include: the name of the data role, the description of the data role, synonyms of the data role name, a data role identification string, a data role version number, a data type (e.g., string, integer, date, boolean, or 64-bit floating point), and / or a data role type. Some examples of data role types include: (i) a dictionary data role type, which is a discrete list of valid domain values, (ii) a range of values for the data role type, which is a defined range within which values are valid (e.g., a numerical range), and (iii) a regular expression data role type, which includes one or more values that match one or more regular expressions considered valid. Each domain value in the dictionary can have an associated list of synonym values. For example, a "month" data role (e.g., a data role named "month") can have an integer type with domain values: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12) and can have a synonym domain that matches string values: ("January", "February",..., "December").

[0271] In some implementations, the data role type makes more sense when understood in combination with other data fields. For example, when the city and state data fields are provided, the postal code data field may be invalid. In some implementations, the importance or priority of a given data field can be determined based on the hierarchy among related data fields. For example, in a geographical hierarchy, the city data field can take precedence over the state and country data fields because, compared with the state and country data fields, the city data field provides information corresponding to a more precise location.

[0272] Some implementations represent semantic types as objects with a set of associated defined attributes. For example, the city type is represented using the city name, state name, and country name attributes. Some implementations expose only one of these attributes in the field itself.

[0273] In some implementations, the data role includes optional attributes such as aggregation, dimension / measure, continuous / discrete, default view type (data visualization type), default format, unit, and visual encoding information. The visual encoding information can include related fields and derived attributes. For example, for the attribute profit, the derived attribute can be "profitable", which has the calculation profitable:=(Profit>0).

[0274] Associated data

[0275] In some implementations, the data role can be associated with field domain data. For example, the dictionary data role is defined by a list of all valid domain values. As another example, the regular expression data role references a list of valid values that is used as sample data to inform users of valid data. In some implementations, these data roles are stored in the data source, allowing the user to: (i) use the data source embedded in the data role to keep the data of the data role private, (ii) create a data role from a previously published data source, (iii) use the published data source, output from the data preparation stream on the server that is refreshed according to a schedule, as the data of the dictionary data role, and / or (iv) manage the connection to the data source used by the data role in bulk, as well as connections from other data sources.

[0276] In some implementations, the user can publish a workbook with a data source that is private to the workbook. The workbook can also reference published data sources that are available to others to which it is connected. Some implementations allow the user to make changes to the connections. Some implementations allow the user to make bulk changes (e.g., across many data sources). In some implementations, the connections used by the published data source and the embedded data source can be edited together.

[0277] Example use cases for the associated data in the data role include:

[0278] · The user publishes a user - defined dictionary data role from the data preparation stream of the source data from which the data fields have been cleaned. If the user does not want the data to be visible on the server outside the data role, the data source is marked accordingly (e.g., using tags such as "embedded extraction only").

[0279] · The user publishes a user - defined dictionary data role from the data preparation stream of the source data from which the data fields have been cleaned. If the user wants the data to be visible on the server outside the data role so that the data can be connected separately from the data role for analysis, the data source is marked accordingly (e.g., using tags such as "published extraction only").

[0280] · The user creates a user - defined dictionary data role from a data source that has been published on the server and has a real - time connection to a database (e.g., a Hadoop database). In this case, the data source is published as a real - time connection to the database.

[0281] · The user uses a published data source that is a CSV as an extraction, as an example data source for a regular expression data role. In this case, the data source is a published extraction connected to a file.

[0282] Modify data

[0283] In some implementations, the application includes a user interface that allows the user to edit and modify the data source values. For example, when the data role uses an embedded data source with a file connection, the user can modify the data source values on the server via the user interface.

[0284] In some implementations, such as when the data role uses a published data source, the user can use any application or tool used to create the data role and its associations to modify the data role.

[0285] Figure 7A A UML model 700 of a data role using a data source to store domain values is shown according to some implementations. Figure 7AThe illustrated example data role 702 stores semantic information, which includes a name, name synonyms, a description, a role type (regular expression or dictionary), and a data type. In some implementations, data fields associated with the data role 702 have one or more display formats 704 for displaying values. According to some implementations, the data role 702 can have a dictionary role 706 and / or a regular expression role 708. According to some implementations, if the data role 702 performs the role of a dictionary, the domain values of the dictionary role are stored in the data source 710. According to some implementations, if the data role performs the role of a regular expression (sometimes referred to as "regexp"), the data source 710 stores example domain values. In some implementations, the data source 710 is an embedded data source 714. In some implementations, the data source 710 is a published data source 712.

[0286] Figure 7B An example process for dispatching data roles is shown according to some implementations. Assume a user wishes to change (720) a data field identified as a string to a latitude (i.e., change the semantic role or information). In response to the user selecting to open the type menu option 722, some implementations provide a menu with different options to choose from. Further assume the user selects (724) the latitude option from the geographic category 726. Some implementations display various subcategories of geographic type data, including latitude 728. Assume the user selects the latitude option 728. Some implementations allow the user to reopen (730) the type menu to view the current selection. For example, refresh the type menu to display the subcategory latitude 732 in the geographic type field. In this way, some implementations allow the user to change the semantic role of a data field.

[0287] Figure 7CAn example user interface 734 for validating data is shown according to some implementations. Some implementations allow a user to set data validation rules 736. Some implementations allow a user to set triggers or notifications 738 (e.g., actions that occur when validation fails). Some implementations allow a user to input validation rules and convert them into data roles. In the example shown, the user selects to use a regular expression 740 to validate data. Some implementations provide various options for validation using a drop-down menu. The example also shows the regular expression 742 that the user inputs for validating data. After the user selects a validation rule, some implementations provide an option 744 to save the validation rule as a data role. In some implementations, each customer (identified by customer ID 746) is shown specific options based on permissions. (In some implementations, the options shown also depend on past usage, data fields, data roles of similar fields, and / or object-level information.) Some implementations also provide an option to switch between data sources or databases 748 when setting validation rules and / or data roles.

[0288] Figure 7DAn example user interface window for improved search using semantic information is shown according to some implementations. Some implementations provide an interface (e.g., a first interface 750) that provides an overview of columns. Some implementations provide a summary bar for an entire profile card so that a user can evaluate the quality of filters, data roles, or suggestions. Some implementations provide an option 752 for selecting search criteria. Some implementations provide a consistency indicator 754 that indicates the extent to which underlying data conforms to search criteria 13. Some implementations also provide an indication 756 of a confidence level for suggestions (e.g., low to very high). Some implementations provide a second interface (e.g., interface 758) that allows a user to set default search behavior. According to some implementations, records that match search results are displayed by default. In the example shown, the user is viewing options related to an email 760. Some implementations select an appropriate email domain address (e.g.,.com 762) and / or display various options 764 for an email address. Some implementations provide a third interface (e.g., interface 766) for in&out searching. In some implementations, a user can select an "Output" section of a chart to view input and output records. In the example shown, the user selects an email 768, and the system responds by selecting a domain 770 and / or multiple email options 772. Some implementations provide a fourth interface (e.g., interface 774) for regex filtering. In some implementations, a data preparation application identifies a regex search pattern and uses that pattern for filtering. In some implementations, a user can switch to directly view the input and output portions of a chart. In the example shown, the user selects to filter emails 776. Some implementations display a regex 778 and a sampling 780 of email addresses that match the regex.

[0289] Some implementations provide user interfaces and / or options to orchestrate and / or edit data roles. Some implementations provide user options for editing domain values. Some implementations allow a user to import or export CSV or Excel files to change embedded data roles. Some implementations allow a user to edit regex (e.g., validation rules for data roles). In some implementations, an embedded data source is an extraction file without an associated source data document. In some implementations, data roles have a specific data format that enhances machine readability and / or is easier to manipulate. In some implementations, the specific file format enables the export of data roles without a server or sharing of data roles with other preparation (or data preparation) users. In some implementations, an embedded data source is embedded in the same document that includes data fields. Some implementations allow a user to place a file in a preparation or prep repository folder to add a data role.

[0290] In some implementations, a data role has its own format (e.g., a JSON file format stored in a semantic service) that includes information about the data role, including a name, validation criteria, and sort order. In some implementations, the data role is associated with a published or embedded data source (e.g., a specific data format that a data visualization platform or data preparation flow knows how to consume, update, edit, and / or dispatch permissions for).

[0291] Some implementations expose the data role format to users, while other implementations prohibit or hide such information from users. Some implementations allow users (e.g., via data preparation or a preparation flow) to publish a data role from a shared data role file. Some implementations allow users to create a data role using a command line and / or using bulk addition or batch processing.

[0292] Some implementations allow users to view data values and test applying a regex on the data values. Some implementations allow users to import or connect to a database table or system to set a data role. In some implementations, embedded data roles are excluded from searches.

[0293] Some implementations allow users to set or change permissions to access a data role. For example, a data preparation user may want to save a data role for an individual user without publishing the data role to share with others. Figure 7E An example user interface 782 for controlling permissions to access a data role is shown according to some implementations. Some implementations provide a search box 784 for searching for users, and options to grant permissions 788 to view 790, interact / edit 792, and / or edit 794 the data role (and / or underlying data source or view 798). Some implementations allow users to select a user or user group 786. Some implementations allow users to add (796) another user or a set of rules 796. In this way, various implementations provide access control for data roles.

[0294] Some implementations provide differential access (and / or control) to global data roles versus local data roles (local to data objects for a group), and / or differential access (and / or control) to custom data roles versus built-in data roles. In some implementations, built-in data roles are not editable. In some implementations, users are allowed to view data sources that reference values corresponding to built-in roles (e.g., geographic roles).

[0295] Some implementations allow users to catalog data roles and / or search for the lineage included in data roles in a data catalog. Some implementations allow users to search for data roles and / or search for objects (e.g., data sources) that use data roles. Some implementations provide browsable and / or searchable content for semantic types (e.g., content hosted by a data visualization server). Some implementations exclude lineage information for custom data roles. Some implementations treat data roles similar to other content on a database server, allowing users to search for data roles by name rather than value without the need for a catalog. Some implementations allow data roles to be packaged as a product that can be sold like other database products. Some implementations allow users to specify data roles when orchestrating desktop or web documents. Some implementations allow users to associate data roles with fields in a desktop application such that when information is exported to a data source and / or brought into a data preparation data stream, the data is automatically validated.

[0296] Data roles and cleaning

[0297] Some implementations automatically update data roles to reflect changes in the user workflow. Some implementations automatically maintain data roles by cleaning up data roles (e.g., updating and / or removing old, obsolete, or irrelevant data roles).

[0298] Some implementations output data roles to databases that do not support semantic types. Some implementations discover semantic types when outputting data roles to a database and write the discovered semantic types. Some implementations use semantic types as an aid in cleaning up data. In some implementations, there are multiple output steps, some of which write back to the database (which does not support semantic types). Some implementations do not output semantic types from the workflow, even if semantic types are used to facilitate data role cleaning and / or shaping.

[0299] Some implementations allow users to connect to a data source and use the data provided by the data source without type changes or data cleaning. This step allows users to view the data before making any changes to it. In some implementations, for databases with strictly typed fields, the data types will be displayed at the input step. In the case of text files (e.g., CSV), some implementations use a string data type to identify all data and perform additional data type identification in subsequent transformation steps.

[0300] For further illustration, assume that the user wishes to filter data so that it only contains data from the last 3 months. Some implementations provide the user with at least two options at the input step: a moderate option that includes the primitive type identifier and a flexible option that includes the semantic type identifier. Further assume that the user selects the moderate option. Some implementations respond by identifying the primitive data type (e.g., number, string, date, or boolean) without cleaning the data. Some implementations perform an initial data type inference on text files. Some implementations support filtering. For fields where the data type identifier results in dropping data or values, some implementations notify the user and allow the user to, for example, change the data type to a string and perform cleaning in subsequent transformation steps. More advanced semantic type identification is only done in subsequent transformation steps. On the other hand, assume that the user selects the flexible option. Some implementations allow the user to discover and / or assign data types. Some implementations allow the user to discover semantic types during the input step. In some implementations, the user can initiate the discovery of semantic types to ensure that the initial data type identification is fast and efficient. For example, the initial step can include an option to initiate semantic type analysis. In some implementations, the user can selectively clean data during the input step. Some implementations allow for full type discovery and / or do not allow cleaning during the input step.

[0301] Some implementations perform filtering during the input step to remove unwanted fields, thus excluding the unwanted fields from the workflow. For example, rows can be filtered out to reduce the data running in the flow. In some cases, such as when applying sampling limits, some implementations perform filtering during the input step. Some implementations identify semantic types so that the user can more easily understand which data fields should be excluded or filtered during the input step. Some implementations provide data cleaning or semantic type suggestions regardless of whether the full domain of the data fields is provided or known.

[0302] In some implementations, data cleaning is an iterative process that balances the interaction performance of the tool with respect to the stability of running the flow over all data. In some cases, for many reasons, such as when operating on sampled data and transitioning to data that includes the full domain, it may be necessary to update or clean the data. In other words, the cleaning can remain accurate and sufficient for a limited period of time, but the transition to data that includes the full domain causes data changes and introduces new domain values, invalidating assumptions about the data. For example, an iterative data cleaning process includes providing suggestions to the user such that the user can clean the data based on the sampled data. Subsequently, the user runs the flow, and various assertions result in notifications informing the user where the data being processed diverges from the assumptions interactively made during the first step. In a specific example, the user sets a mapping of data field groups and replacement data fields. When the user runs the full flow, new values outside of the original sample data are found. Optionally, the user can use the groups and replacements to change the data type of the data fields such that the data values in the data fields map to values in the specification. In this case, new values in the data fields are found that are invalid for the previously defined data type (when operating on the sampled data). As a result of either of these two cases, in some implementations, when the user re-opens the flow, the user receives a series of result notifications, and the user can edit the flow to account for this new information. After any edits are made, the user can run the flow again.

[0303] Interact with semantic roles of data types

[0304] Some implementations treat semantic types as an extension of the existing type system. For example, the user selects a single data type name (such as "email address"), and that single data type name identifies a primitive data type (such as string) and any associated semantics. When the user selects the "email address" type, the data type of the data field is changed to "string" (if it is not already), and invalid values within the data field are identified to the user. In such implementations, the handling of semantic types allows for a single underlying data type for the semantic role. The selected data type is the data type that best reflects the semantics of the role and allows the user to perform the intended manipulations and / or clean the values of the role. In some cases, not forcing the values to the ideal underlying data type can cause problems, such as preventing the user from normalizing the values to a single representation or performing meaningful calculations or cleaning operations.

[0305] Some implementations treat semantic roles as roles independent of the underlying primitive data types, and these two attributes can be changed independently of each other. For example, a user can set the data type to "integer" and then apply the semantic role "email address" to a data field to see which values are invalid (e.g., the values are all numbers and the data type remains integer). In another example, a postal code can be stored with an integer data type and the semantic role set to "postal code / zip code". In this way, the semantic role can be applied without changing the data type of the data field, even though the most general data type required to allow all valid values is actually "string" (to handle alphanumeric zip codes and possibly hyphens). In some cases, this is very useful if the user ultimately wishes to write the data field back to the database so that the data field remains an integer data type.

[0306] In some implementations, a user can access all semantic roles available for any data via a user interface. Additionally, the user interface can also include any of the following: a list of available data types that should be suitable for any data field, a list of available semantic roles that should be suitable for any data field, a list of available data types and / or data roles, a list of semantic roles from which the user can choose, where each semantic role is independent of the current field data type (e.g., the user can change from any permutation of data types or data roles to any other permutation). In some implementations, the user interface (e.g., via a representative icon summarizing the data field on the field header) displays both the semantic role and the primitive data type. In some implementations, changing the format of a data field does not change the data type for the purposes of maintaining calculations. In some implementations, the user is able to incorporate the format into the data field during an output step (e.g., export, write, or save).

[0307] Some implementations preserve the data type independent of semantic roles. This is helpful when the output target does not preserve semantic roles (e.g., modeling attributes). In such cases, it is useful to maintain the user's perception of stable data type elements throughout the workflow. Some implementations maintain the underlying primitive data type without changing the data type and store semantic roles independently. For example, when the system detects an input (e.g., the user moves the cursor or hovers it over a data type icon in a profile), some implementations display this information in a tooltip. The user's perception of the data type is also important in computations (computations only apply to primitive data types). Some implementations maintain the user's perception of the data type without changing the data type and semantic roles independently of each other. In some implementations, the representation of the data is maintained throughout the workflow so that the information displayed in the user interface remains in the context of the user's entire workflow. For example, the data representation is maintained from the input step (e.g., data cleaning step) to the output step (e.g., save, publish, export, or write step).

[0308] In some implementations, semantic roles are applied to more than one data type. In some implementations, when a semantic role is selected, the data type is not automatically changed in order to maintain data type consistency for output purposes.

[0309] In some implementations, when there are different data field types, the data field can be automatically changed to a more general type (without semantics) that can represent the values in both data fields. Optionally, when dealing with different data field types, the data field can be automatically changed to a more general type that can represent the values in both fields. On the other hand, if the semantic roles are different, the semantic roles are automatically cleared.

[0310] Some implementations identify invalid joins based on the semantic role of the join clause field. For example, assume the join clause field has different data types or different semantic types. Some implementations identify invalid joins when the join clauses have similar semantic roles.

[0311] In some implementations, regardless of the format, the data type of a data field retains its initial data type (e.g., even when the user changes the display format of the data field) so that the data field can be manipulated as expected. For example, a date data field can have a pure numeric format or a string format, but regardless of how it is displayed, it will retain the same canonical date data type. In another example, a "day of the week" data field retains its underlying numeric type even if the data field value is displayed as text, such as "Monday", so that calculations can still be performed using the data value in the "day of the week". For example, if the value is "Monday", then calculating "day + 1" will give the result "Tuesday". In this example, the accepted data values are strings (e.g., "Sunday", "Monday",... "Friday", "Saturday"). At the output node, it may be necessary to change the data type of the "day of the week" data field according to the user's goals and the output target. For example, the data can be output to a strongly typed database, such that the data output defaults to the base data type and the date data field needs to be switched to the "string" data type to maintain the format. Optionally, in some implementations, the data output defaults to the "string" data type and thus there is no need to change the data type to maintain the format. In some implementations, the user can change the data type.

[0312] Some implementations support various semantic type manipulations even while maintaining the underlying primitive data type. For example, assume the user manipulates a date field using the date type to change the format of the date. Some implementations keep the primitive data type of the date as an integer even if the format has changed. Some implementations only identify the primitive data type and indicate the semantic type when the user requests the semantic type.

[0313] In some implementations, the data is not cleaned at the input node (described above). Some implementations notify the user of any data discarded through type dispatch, and the user will be provided with the opportunity to edit the data in subsequent transformation nodes. In some implementations, the transformation node can be initiated by the user from the input node. Additionally, data quality and cardinality may not be displayed during the input step but only during the transformation step.

[0314] In some implementations, semantic types have corresponding schemas. Some implementations include a strict mode and a non-strict mode. If a semantic type is in strict mode, values outside of that type definition are not retained in the domain. If a semantic type is set to non-strict mode, values outside of the type definition are retained in the domain and passed through the flow. In some implementations, primitive types are always strict and do not support values that fall outside of the type definition throughout the flow.

[0315] Some implementations perform text scans and notify the user whether to discard specific types of data. Some implementations provide the user with the values of the data to be discarded. In some implementations, for type change operations, when a type change processing method (recipe) is selected, values that fall outside the type will be marked in the configuration file view (these values will be discarded). In some implementations, once the user moves to a new (e.g., next) processing method, these discarded values will no longer be displayed. Thus, immediate visibility of the discarded (or to-be-discarded) values is provided to the user so that the user can select them and perform remapping if needed. Additionally, some implementations allow the user to select an action from a list of actions, which will create a remapping to a null value for all values outside the type definition. In some implementations, the user can also edit or refine the remapping or select a remapping action from one or more provided suggestions or recommendations. In some implementations, remapping is performed immediately before the type change operation so that the remapped values will flow into the type change (since discarded values do not remain in the type change operation and thus cannot be remapped after the type change). In some implementations, semantic types are used as data assertions instead of types, or in addition to being considered types, semantic types are also used as data assertions. Some implementations allow the user to indicate to the system "this is the data type I expect here; if not, please notify me", and the system automatically notifies the user accordingly.

[0316] In some implementations, additional automatic cleaning steps are included to provide type identification and suggestions. In some implementations, the additional automatic cleaning steps can run in the background (e.g., by a daemon), or can be cancelled by the user.

[0317] For example, when copying an existing column, if any data of a specific type is to be discarded, the values to be discarded in that column will be remapped to null values. Some implementations provide a side-by-side comparison of the original column and the copied column.

[0318] Some implementations allow the user to add an automatic cleaning step, which will add a step and start type identification and suggestions. In some implementations, this step is performed as a background job. Some implementations provide the user option to cancel the job while it is running.

[0319] Some implementations copy columns to clean data. Some implementations copy existing columns, map type specification values in the columns to null values, perform or display a side-by-side comparison of the columns (e.g., select null values in the mapped column to brush the mapped values in the copied column), allow the user to select data field values from the type values in the copied column, and filter to obtain only those rows that contain those values so that other fields in the rows are displayed as background for correction. Some implementations allow the user to correct the values that are mapped to null values in the copied column and then remove the initial column. Some implementations use an interface similar to the join clause tuple remapping user interface to allow the user to perform the operations described herein.

[0320] Some implementations obtain global values for fields for robust correction. Some implementations indicate domain values.

[0321] In response to a type change action, some implementations incorporate remapping to automatically map values that are out of specification to null values. Some implementations allow the user to edit the remapping from the type change handling method.

[0322] Some implementations display values that are out of type specification when the user selects a type change handling method. In some implementations, if another handling method is added, the values that are out of specification disappear (so they do not move in the stream). Some implementations perform an inline or edit remapping on these values, which creates a remapping handling method before the type change handling method. Some implementations display grouping icons for the values that are grouped when the user selects a handling method.

[0323] In some implementations, the type change action creates a remapping handling method to map values that are out of specification to null values and then creates the type change handling method such that the remapped values are before hitting the type change. When the user selects a type change handling method, some implementations use this as an alternative if it is not possible to display the values that are out of specification marked in the domain.

[0324] In some implementations, the type change action implicitly excludes any values that are out of type specification. In some implementations, the handling method corresponds to many of the excluded values. Some implementations allow the user to create an upstream remapping for the type change mapping (e.g., map the excluded values to null values). In some implementations, when the clean stream is running, a list of the excluded values is provided to the user such that the user can add these values to the upstream remapping to clean the data.

[0325] In some implementations, the user interface allows the user to delete a processing method by dragging a selected annotation (corresponding to the processing method) out of the configuration file. In some implementations, type dispatch is strict in the input and output steps and is not strict (e.g., flexible) during intermediate steps (e.g., steps between the input and output steps) in order to preserve values.

[0326] In some implementations, the data type and / or semantic role of a field indicate the user's expectation of what the data in the data field is. Used in combination with the data name, the data type helps convey the meaning of the data field and provides an understanding of the data field. For example, if the value is a decimal number called "Profit", the value may be in dollars or as a percentage. Thus, the data type can inform or determine what operations are allowed on the data field, thereby providing the user and the application with a higher confidence in the background (including inferred background) of the data field (and sometimes the data source) and the results of operations such as calculations, remapping, or applying filters. For example, if a data field has a numeric type, the user can be confident that the user can perform mathematical calculations.

[0327] Some implementations use semantic roles for data fields to help the user clean data by highlighting values that do not match the data type and need to be cleaned and / or filtered. In some implementations, the data type and / or semantic role of a data field provide the context that allows a data preparation application to perform automatic cleaning operations on the data field.

[0328] In some implementations, when the user selects a type change for a field, values outside the type specification are grouped and mapped to null values. Some implementations exclude such values and / or allow the user to inspect and / or edit the values in the group. Some implementations group values that are outside the type specification and values that are outside the semantic role separately. Some implementations allow the user to select a group and merge / apply the combination to a new field.

[0329] Some implementations display a visualization and indicate a profile summary view that selects outliers (e.g., null values). Some implementations display each value in a histogram. Some implementations display values that do not match the data type (e.g., values marked as "discarded"), values that do not match the semantic role (e.g., values marked as "invalid"), and / or selected outliers. Some implementations use strikethrough text to indicate values that do not match the data type. Some implementations indicate values that do not match the semantic role in red text or via a special icon. Some implementations filter the values to show only outliers, values that do not match the data type, and / or values that do not match the semantic role. Some implementations provide a search box for filtering that allows the user to activate the search using options in a drop-down menu to "search within invalid values". Some implementations provide an option to filter the list. Some implementations provide an option to select individual outliers or entire summary histogram bars, and to move the selection to another field (e.g., right-click option or drag-and-drop functionality), or to move values between fields to create a new field. Some implementations display the data type of the new field and / or indicate that all values match the data type. Some implementations provide the user with an option to clean the values in the new field and / or subsequently drag the discarded values or the entire field back to the initial field to merge the changes back.

[0330] Recommendations in data preparation

[0331] In some implementations, the auto-suggestions include converting the values of individual fields. In some cases, this can be facilitated by assigning a data role to the data field. In other cases, it is not necessary to assign a data role to the data field. When a data role is applied to a data field, validation rules are applied to the data values in the data field, and outliers are displayed with different visual treatments in the profile and the data grid (e.g., outliers can be emphasized, such as highlighted or displayed in a different-colored font, or de-emphasized, such as displayed in gray or a light-colored font). This visual treatment of outliers is maintained across the various steps of the entire workflow.

[0332] In some implementations, the re-mapping suggestions include: (i) manual re-mapping of outliers, (ii) re-mapping outliers by leveraging fuzzy matching of the domain associated with the assigned data role, and / or (iii) automatically re-mapping outliers to null values.

[0333] Some implementations provide a one - click option to filter outliers. Some implementations provide a one - click option to extract outliers into a new data field so that the user can edit the extracted values (manually, through remapping operations, or by applying data roles), and then merge the edited values back into the data field or store the edited values separately. Some implementations provide an option to discard a data field if most of its values are null values.

[0334] In some implementations, when the user selects a data role for a specific data field, the data preparation application provides transformation suggestions. For example, for the domain of a URL or for the area code of a phone number, the transformation suggestions can include extraction / splitting transformations (e.g., from 1112223333 to "area code" = "111" and "phone number" = "2223333"). In another example, the transformation suggestions can include reformatting suggestions for the data field, such as: (i) changing the state name to the state abbreviation (e.g., California to CA), (ii) changing the phone number to a specific format (e.g., from 1112223333 to (111)222 - 3333), (iii) converting the date to a different representation format or represented by different extracted parts (e.g., extracting December 1, 1990 as only the year, or changing the format to 12 / 01 / 1990).

[0335] Some implementations automatically parse, validate, and / or correct / clean date fields in various formats. Some implementations standardize country values. Some implementations display values based on a canonical list of names. Some implementations identify ages as positive numbers and / or provide options to search for and / or standardize age values. Some implementations allow the user to standardize a list of database names based on connector names. Some implementations allow the user to further edit options and / or provide manual overrides. Some implementations standardize cities, states, or similar values based on semantic roles.

[0336] In some implementations, remapping recommendations are used to notify the user that the semantics of a data field have been recognized and to provide the user with suggestions for cleaning the data. In some implementations, the cleaning suggestions include showing the user at least a portion of the metadata so that the user can better understand what each suggestion is and why the suggestion is relevant to the user's needs. In some implementations, the user's selections drive the analysis and / or further suggestions. In some implementations, the user interface includes options for the user to view overviews and / or details, zoom in and / or filter data fields and / or data sources.

[0337] In some implementations, the user interface includes a result preview (e.g., a preview of the results of operations such as filtering, zooming, or applying data roles) so that the user can proceed with confidence.

[0338] Figure 8 According to some implementations, an example user interface 800 for previewing and / or editing sanitization recommendations is shown. Figure 8 The example shown displays recommendations 802 for a city 804. Some implementations display a list of initially recommended options, followed by further recommendations 806. Some implementations show a confidence level 808 for additional recommendations 806. Some implementations display a specific recommendation (or a single value) that is particularly useful for long values or names. When the confidence is below a predetermined threshold (e.g., an 80% confidence level), some implementations display multiple possible recommendations. Some implementations display more recommendations in response to a user click or selection. Some implementations allow a user to select from a list 810 rather than having to type in additional recommendations, especially for long-winded recommendations. Some implementations provide such an option as part of a viewing mode (e.g., a mode in a group and replace editor). Some implementations provide user options to switch between viewing modes. For example, one viewing mode may be more suitable for a set of data roles, while other viewing modes may be more suitable for other sets of data roles.

[0339] Figures 9A - 9D According to some implementations, an example user interface for resource recommendations based on semantic information is shown. Figure 9A According to some implementations, various interfaces for manipulating partial dates are shown. Example interface 900 is a pill context menu that displays various options for a date. Example interface 902 shows a right drag-and-drop option that allows a user to drag and drop a field (in this example, a date type). Example interface 904 shows an option for a user to enter text to search for an action. Example interface 906 shows an option for creating a custom date. Example interface 908 shows an option for adding and / or editing field mappings to allow a user to edit relationships (e.g., relationships between fields). Some implementations provide DATEPART and / or DATETRUNC functions to manipulate date fields. Figure 9B Various interfaces for manipulating parts of a URL are shown. Example interface 910 is a pill context menu that displays various options for a URL. Example interface 912 shows a right drag-and-drop option that allows a user to drag and drop a field (in this example, a URL type). Example interface 914 shows an option for a user to enter text to search for an action corresponding to the URL. Example interface 916 shows an option for creating a custom URL. Example interface 918 shows an option for adding and / or editing field mappings to allow a user to edit relationships between URL fields. Some implementations provide functions for manipulating data fields and / or data roles, similar to the HOST(), DOMAIN(), and TLD() functions in BigQuery. Figure 9CShows various interfaces for manipulating parts of an email address. Example interface 920 is a capsule background menu showing various options for an email. Example interface 922 shows a right drag-and-drop option that allows a user to drag and drop a field (in this example, a part of the email). Example interface 924 shows an option for a user to enter text to search for actions corresponding to the email. Example interface 926 shows an option to create a custom email or filter emails (e.g., using a date). Example interface 928 shows an option for adding and / or editing field mappings to allow a user to edit the relationships between email fields. Figure 9D Shows various interfaces for manipulating parts of a name. Example interface 930 is a capsule background menu showing various options for a name. Example interface 932 shows a right drag-and-drop option that allows a user to drag and drop a field (in this example, a part of the name). Example interface 934 shows an option for a user to enter text to search for actions corresponding to the name. Example interface 936 shows an option to create a custom name or filter names (e.g., using a date). Example interface 938 shows an option for adding and / or editing field mappings to allow a user to edit the relationships between email fields. Some implementations include examples of field names, group parts in one or more user interfaces, including truncation and / or matching parts (for editing field mappings).

[0340] Figures 10A - 10N According to some implementations, a flowchart 1000 of a method for preparing data for subsequent analysis is provided. The method is generally executed at a computer 200 having a display 2080, one or more processors 202, and a memory 214 that stores one or more programs configured to be executed by the one or more processors.

[0341] The method includes obtaining (1002) a data model (e.g., object model 108) of a tree that encodes a first data source as logical tables. Each logical table has its own physical representation and includes a corresponding one or more logical fields. Each logical field corresponds to a data field or calculation spanning one or more logical tables. Each edge of the tree connects two related logical tables. The method also includes associating (1004) each logical table in the data model with a corresponding concept in a concept graph. The concept graph (e.g., a directed acyclic graph) embodies a hierarchical inheritance of the semantics of the logical tables. According to some implementations, an example concept graph is described above with reference to Figure 4 The method also includes, for each logical field included in a logical table (1006), dispatching (1008) a semantic role (sometimes called a data role) to the logical field based on the concept corresponding to the logical table.

[0342] Next, with reference to Figure 10B, in some implementations, the method further includes, for each logical field, storing (1016) its assigned semantic role in a first data source (or auxiliary data source).

[0343] Next, refer to Figure 10C , in some implementations, the method further includes generating (1018) a second data source based on the first data source, and for each logical field, storing its assigned semantic role in the second data source.

[0344] Next, refer to Figure 10D , in some implementations, the method further includes, for each logical field, retrieving (1020) a representative semantic role (e.g., the semantic role assigned to a similar logical field) from a second data source different from the first data source. Assigning the semantic role to the logical field is also based on the representative semantic role. In some implementations, the user input is detected as coming from a first user, and the method further includes determining (1022) whether the first user is authorized to access the second data source before retrieving the representative semantic role from the second data source.

[0345] Next, refer to Figure 10E , in some implementations, the method further includes, in a user interface, displaying (1024) one or more first semantic roles of a first logical field based on a concept corresponding to a first logical table including the first logical field. The method further includes, in response to detecting a user input selecting a preferred semantic role, assigning (1026) the preferred semantic role to the first logical field. In some implementations, the method further includes: selecting (1028) one or more second semantic roles regarding a second logical field based on the preferred semantic role. The method further includes displaying (1030) the one or more second semantic roles regarding the second logical field in the user interface. In response to detecting a second user input selecting a second semantic role from the one or more second semantic roles, the method includes assigning (1032) the second semantic role to the second logical field. In some implementations, the method further includes training (1034) one or more prediction models based on a data source of one or more semantic tags (e.g., a data source of data fields with assigned or tagged semantic roles). The method further includes determining (1036) the one or more first semantic roles by inputting a concept corresponding to the first logical table into the one or more prediction models.

[0346] Next, refer to Figure 10F , in some implementations, the logical field is (1038) a calculation based on a first data field and a second data field, and assigning the semantic role to the logical field is also based on a first semantic role corresponding to the first data field and a second semantic role corresponding to the second data field.

[0347] Next, refer to Figure 10G , in some implementations, the method includes determining (1040) a default format of a data field corresponding to a logical field, and dispatching a semantic role to the logical field is also based on the default format of the data field.

[0348] Next, refer to Figure 10H , in some implementations, the method further includes selecting and storing (1042) a default format option for displaying the logical field based on the dispatched semantic role to a first data source.

[0349] Next, refer to Figure 10I , in some implementations, the method further includes displaying (1046) a concept map and one or more options for modifying the concept map in a user interface before dispatching (1044) a semantic role to the logical field. In response to detecting user input for modifying the concept map, the method includes updating (1048) the concept map according to the user input.

[0350] Return to reference Figure 10A , the method further includes validating (1010) the logical field based on the semantic role dispatched to the logical field. Next, refer to Figure 10J , in some implementations, the semantic role includes (1050) a domain of the logical field, and validating the logical field includes determining whether the logical field matches one or more domain values of the domain. The method further includes determining one or more conversions based on one or more domain values before displaying one or more conversions. Next, refer to Figure 10K , in some implementations, the semantic role is (1052) a validation rule (e.g., a regular expression) for validating the logical field.

[0351] Return to reference Figure 10A , the method further includes displaying (1012) one or more conversions in a user interface on a display to clean (or filter) the logical field based on the validation of the logical field. In response to detecting user input for selecting a conversion to convert the logical field, the method converts (1014) the logical field according to the user input and updates the logical table based on the conversion of the logical field.

[0352] Next, refer to Figure 10L , in some implementations, the method further includes determining (1054) a first logical field to be added to a first logical table based on a concept of the first logical table. The method further includes displaying (1056) a recommendation for adding the first logical field in the user interface. In response to detecting user input for adding the first logical field, the method includes updating (1058) the first logical table to include the first logical field.

[0353] Next, refer toFigure 10M , in some implementations, the method further includes determining (1060) a second data set corresponding to a second data source based on a concept graph for joining with a first data set corresponding to a first data source. The method further includes displaying (1062) in a user interface a recommendation to join the second data set with the first data set of the first data source. In response to detecting a user input to join the second data set, the method further includes creating (1064) a join between the first data set and the second data set and updating the tree of the logical table.

[0354] Next, refer to Figure 10N , in some implementations, the method further includes detecting (1066) a change to the first data source. In some implementations, detecting a change to the first data source is performed (1068) at a predetermined time interval. In response to (1070) detecting a change to the first data source, the method includes updating (1072) the concept graph based on the change to the first data source and repeating (1074) dispatching, validating, displaying, transforming, and updating for each logical field based on the updated concept graph.

[0355] The terms used in the description of the present invention are for the purpose of describing particular implementations only and are not intended to limit the present invention. As used in the description of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also to be understood that the term "and / or" as used herein refers to any and all possible combinations of one or more of the related listed items and includes these combinations. It is also to be understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof.

[0356] For purposes of explanation, the foregoing description has been presented with reference to specific implementations. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The implementations were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling those skilled in the art to best utilize the invention and various implementations with various modifications as are suited to the particular use contemplated.

Claims

1. A method for preparing data for subsequent analysis, comprising: at a computer having a display, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors: obtaining a data model of a tree that encodes a first data source into logical tables, each logical table having its own physical representation and including a corresponding one or more logical fields, each logical field corresponding to a computational or data field spanning one or more logical tables, wherein each edge of the tree connects two related logical tables; associating each logical table in the data model with a corresponding concept in a concept graph; and for each logical field included in the logical tables: assigning a semantic role to the logical field based on the concept corresponding to the logical table, wherein the concept graph embodies semantic hierarchical inheritance and sub - concepts in the concept graph inherit characteristics of the parent concept in the concept graph; validating the logical field based on the assigned semantic role of the logical field; displaying, in a user interface on the display, one or more transformations to clean the logical field based on the validation of the logical field; and in response to detecting a first user input to select a transformation to transform the logical field, transforming the logical field according to the first user input and updating the logical table based on the transformation of the logical field; wherein a first logical table in the logical tables includes a first logical field that is a computation based on a first data field and a second data field; and wherein assigning the semantic role to the first logical field is further based on a first semantic role corresponding to the first data field and a second semantic role corresponding to the second data field.

2. The method according to claim 1, further comprising: for each logical field included in a logical table, storing the assigned semantic role of each logical field in the first data source.

3. The method according to claim 1, further comprising: generating a second data source based on the first data source; and for each logical field included in a logical table, storing the assigned semantic role of each logical field in the second data source.

4. The method according to claim 1, further comprising: for each logical field in a logical table, retrieving a representative semantic role from a second data source different from the first data source, wherein assigning the semantic role to the logical field is further based on the representative semantic role.

5. The method according to claim 4, wherein the first user input is detected from a first user, and the method further comprises: determining whether the first user is authorized to access the second data source before retrieving the representative semantic role from the second data source.

6. The method according to claim 1, further comprising: in the user interface, displaying one or more first semantic roles of the first logical field based on the concept corresponding to the first logical table including the first logical field; and In response to detecting a second user input that selects a preferred semantic role from the one or more first semantic roles, assign the preferred semantic role to the first logical field.

7. The method according to claim 6, further comprising: Select one or more second semantic roles for a second logical field based on the preferred semantic role; Display the one or more second semantic roles for the second logical field in the user interface; and In response to detecting a third user input that selects a third semantic role from the one or more second semantic roles, assign the third semantic role to the second logical field.

8. The method according to claim 6, further comprising: Train one or more prediction models based on a data source of one or more semantic tags; and Determine the one or more first semantic roles by inputting the concepts corresponding to the first logical table into the one or more prediction models.

9. The method according to claim 1, further comprising determining a default format of a data field corresponding to the logical field, wherein assigning the semantic role to the logical field is further based on the default format of the data field.

10. The method according to claim 1, further comprising: Select a default format option for displaying the logical field based on the assigned semantic role and store the default format option in the first data source.

11. The method according to claim 1, further comprising: Before assigning the semantic role to the logical field: Display the concept map and one or more options for modifying the concept map in the user interface; and In response to detecting a fourth user input for modifying the concept map, update the concept map according to the fourth user input.

12. The method according to claim 1, wherein, the semantic role includes a domain of the logical field, and verifying the logical field includes determining whether the logical field matches a domain value of the domain, and the method further comprises: Determine the one or more conversions based on the domain value of the domain before displaying the one or more conversions.

13. The method according to claim 1, wherein, the semantic role is a verification rule for verifying the logical field.

14. The method according to claim 1, further comprising: Determine the first logical field to be added to the first logical table based on the concept of the first logical field; Display a recommendation for adding the first logical field in the user interface; and In response to detecting a user input for adding the first logical field, update the first logical table to include the first logical field.

15. The method according to claim 1, further comprising: Determine a second data set corresponding to a second data source based on the concept map for joining with a first data set corresponding to the first data source; Display a recommendation for joining the second data set with the first data set in the user interface; and In response to detecting a fifth user input for joining the second data set: Create a join between the first data set and the second data set; and Update the tree of the logical table.

16. The method according to claim 1, further comprising: detecting a change to the first data source; and in response to detecting the change: updating the concept map according to the change; and for each of the plurality of data fields included in the logical table: repeating the dispatching, the verification, the display, the conversion, and the update for each logical field according to the updated concept map.

17. The method according to claim 16, wherein the detection of the change to the first data source is performed at a predetermined time interval.

18. A computer system for preparing data for subsequent analysis, comprising: a display; one or more processors; and a memory; wherein the memory stores one or more programs configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 1 - 17.

19. A non - transitory computer - readable storage medium storing one or more programs configured to be executed by a computer system having a display, one or more processors, and a memory, the one or more programs including instructions for performing the method according to any one of claims 1 - 17.

Images