System and method for automatically enriching data sets using system knowledge data

Abstract

In accordance with an embodiment, described herein is a system and method for automatically enriching data sets in a data analysis environment using system knowledge data. The system may operate after analyzing the dataset to automatically enrich the dataset. A user of a data analysis environment, such as a commercial user preparing a data visualization, may not know additional data and system knowledge data that may be used to improve the data visualization. Systems and methods described herein may provide for automatic enrichment of data from, for example, a knowledge repository that may be delivered to a data analysis customer using various delivery means.

Description

[0001] Copyright Notice

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0003] Priority Claim:

[0004] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 416,379, filed on October 14, 2022, entitled “SYSTEM AND METHOD FORAUTOMATICALLY ENRICHING DATASETS WITH SYSTEM KNOWLEDGE DATA”; and U.S. Patent Application No. 18 / 137,963, filed on April 21, 2023, entitled “SYSTEM AND METHOD FORAUTOMATICALLY ENRICHING DATASETS WITH SYSTEM KNOWLEDGE DATA”; the above applications and their contents are incorporated herein by reference. Technical Field

[0005] Embodiments described herein relate generally to computer data analysis and computer-based methods for providing business intelligence or other data, and particularly to systems and methods for automatically enriching data sets in a data analysis environment with system knowledge data. Background Art

[0006] Data analytics enables computer-based examination of large amounts of data, such as to draw conclusions or other information from the data. For example, business intelligence tools can be used to provide users with business intelligence describing their enterprise data in a format that enables them to make strategic business decisions. Summary of the invention

[0007] According to embodiments, systems and methods are described herein that automatically enrich a data set in a data analysis environment using system knowledge data. The system may operate after analyzing a data set to automatically enrich the data set. Users of the data analysis environment, such as business users preparing data visualizations, may not be aware of additional data and system knowledge data that may be used to improve data visualizations. The systems and methods described herein may provide automatic enrichment of data from, for example, a knowledge repository that may be delivered to a data analysis client using a variety of delivery means. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example data analysis environment is illustrated according to an embodiment.

[0009] Figure 2 Further illustrated is an example data analysis environment according to an embodiment.

[0010] Figure 3 Further illustrated is an example data analysis environment according to an embodiment.

[0011] Figure 4 Further illustrated is an example data analysis environment according to an embodiment.

[0012] Figure 5 Further illustrated is an example data analysis environment according to an embodiment.

[0013] Figure 6 The use of a system to transform, analyze, or visualize data according to an embodiment is illustrated.

[0014] Figure 7 Various examples of user interfaces for use with a data analysis environment are illustrated according to an embodiment.

[0015] Figure 8 Various examples of user interfaces for use with a data analysis environment are illustrated according to an embodiment.

[0016] Fig. 9 Various examples of user interfaces for use with a data analysis environment are illustrated according to an embodiment.

[0017] Fig.10 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0018] Fig.11 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0019] Fig.12 Various examples of user interfaces for use with a data analysis environment are illustrated according to an embodiment.

[0020] Fig.13 Various examples of user interfaces for use with a data analysis environment are illustrated according to an embodiment.

[0021] Fig.14 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0022] Fig.15 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0023] Fig.16 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0024] Fig.17 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0025] Fig.18 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0026] Fig.19 A system for automatically enriching a data set in a data analysis environment using system knowledge data according to an embodiment is illustrated.

[0027] Fig. 20 is a flowchart of a method for automatically enriching a data set in a data analysis environment using system knowledge data according to an embodiment. DETAILED DESCRIPTION

[0028] Generally speaking, within an organization, data analytics enables computer-based examination of large amounts of data, such as to draw conclusions or other information from the data. For example, business intelligence (BI) tools can be used to provide users with business intelligence describing their enterprise data in a format that enables them to make strategic business decisions.

[0029] Examples of such business intelligence tools / servers include Oracle Business Intelligence Applications (OBIA), Oracle Business Intelligence Enterprise Edition (OBIEE), or Oracle Business Intelligence Server (OBIS), which provide query, reporting, and analytical servers that operate with a database to support features such as data mining or analysis and analytical applications.

[0030] Increasingly, data analytics may be provided within the context of an enterprise software application environment (such as, for example, an Oracle Fusion Application environment); or within a Software as a Service (SaaS) or cloud environment (such as, for example, an Oracle Analytics Cloud or Oracle Cloud Infrastructure environment); or in other types of analytical applications or cloud environments.

[0031] introduce

[0032] According to an embodiment, a data warehouse environment or component (such as, for example, Oracle Autonomous Data Warehouse (ADW), Oracle Autonomous Data Warehouse Cloud (ADWC), or other types of data warehouse environments or components suitable for storing large amounts of data) may provide a central repository for storing data collected by one or more business applications.

[0033] For example, according to an embodiment, a data warehouse environment or component may be provided as a multidimensional database that generates business-related data from multiple different data sources using online analytical processing (OLAP) or other techniques. An organization may extract such business-related data from one or more vertical and / or horizontal business applications and inject the extracted data into a data warehouse instance associated with the organization.

[0034] Examples of horizontal business applications may include ERP, HCM, CX, SCM, and EPM as described above, and provide a wide range of functionality across various enterprise organizations.

[0035] Vertical business applications are typically narrower in scope than horizontal business applications, but provide access to data further up and downstream in the data chain within a defined scope or industry. Examples of vertical business applications might include medical software or banking software for use within a specific organization.

[0036] While software vendors increasingly offer enterprise software products or components as SaaS or cloud-oriented offerings, such as, for example, Oracle Fusion Applications; while other enterprise software products or components (such as, for example, Oracle ADWC) may be offered as one or more of SaaS, Platform as a Service (PaaS), or hybrid subscriptions; enterprise users of conventional business intelligence applications and processes are typically faced with the task of extracting data from their horizontal and vertical business applications and bringing the extracted data into a data warehouse - a process that can be both time-consuming and resource-intensive.

[0037] According to an embodiment, the analytical application environment allows a customer (tenant) to develop computer-executable software analytical applications for use with a BI component (such as, for example, an OBIS environment or other type of BI component suitable for examining large amounts of data originating from the customer (tenant) itself or from multiple third-party entities).

[0038] As another example, according to an embodiment, the analytical application environment can be used to pre-populate a reporting interface of a data warehouse instance with relevant metadata describing business-related data objects in the context of various business productivity software applications, for example, to include predefined dashboards, key performance indicators (KPIs), or other types of reports.

[0039] Data analysis

[0040] Generally speaking, data analytics enables computer-based examination or analysis of large amounts of data to draw conclusions or other information from the data; while business intelligence tools (BI) provide an organization's business users with information describing their enterprise data in a format that enables them to make strategic business decisions.

[0041] Examples of data analytics environments and business intelligence tools / servers include Oracle Business Intelligence Server (OBIS), Oracle Analytics Cloud (OAC), and Fusion Analytics Warehouse (FAW), which support features such as data mining or analysis and analytical applications.

[0042] Figure 1 An example data analysis environment is illustrated according to an embodiment.

[0043] Figure 1 The example embodiments shown in are used to illustrate examples of data analysis environments in which various embodiments described herein may be used in association. According to other embodiments and examples, the methods described herein may be used with other types of data analysis, database or data warehouse environments. Figure 1 The components and processes shown in and further described herein with respect to various other embodiments may be provided as software or program code executable by, for example, a cloud computing system or other appropriately programmed computer system.

[0044] like Figure 1 As shown in , according to an embodiment, a data analysis environment 100 may be provided by or otherwise operated at a computer system having computer hardware (e.g., processor, memory) 101 and including one or more software components operating as a control plane 102 and a data plane 104, and providing access to a data warehouse, a data warehouse instance 160 (a database 161 or other type of data source).

[0045] According to an embodiment, the control plane operates to provide control of cloud or other software products provided in the context of a SaaS or cloud environment (such as, for example, an Oracle Analytics Cloud environment or other type of cloud environment). For example, according to an embodiment, the control plane may include a console interface 110 that enables access by customers (tenants) and / or cloud environments having a provisioning component 111.

[0046] According to embodiments, the console interface may enable customers (tenants) to access by operating a graphical user interface (GUI) and / or a command line interface (CLI) or other interface; and / or may include an interface for use by a provider of a SaaS or cloud environment and its customers (tenants). For example, according to embodiments, the console interface may provide an interface that allows customers to provision services for use within their SaaS environment and configure those provisioned services.

[0047] According to an embodiment, a customer (tenant) may request to provision a customer schema within a data warehouse. The customer may also provision a number of properties associated with a data warehouse instance via a console interface, including required properties (e.g., login credentials) and optional properties (e.g., size or speed). The provisioning component may then provision the requested data warehouse instance, including the customer schema for the data warehouse; and populate the data warehouse instance with the appropriate information provided by the customer.

[0048] According to an embodiment, the provisioning component may also be used to update or edit the data warehouse instance and / or the ETL process operating at the data plane, for example, by changing or updating the request frequency at which the ETL process runs for a particular customer (tenant).

[0049] According to an embodiment, the data plane may include a data pipeline or processing layer 120 and a data transformation layer 134, which together process operational or transactional data from an organization's enterprise software applications or data environment (such as, for example, business productivity software applications delivered in a customer's (tenant's) SaaS environment). The data pipeline or processing may include various functions that extract transactional data from business applications and databases delivered in the SaaS environment and then load the transformed data into a data warehouse.

[0050] According to an embodiment, the data transformation layer may include data models used by the system to transform transactional data received from business applications and corresponding transactional databases supplied in a SaaS environment into a model format understood by a data analysis environment, such as, for example, a knowledge model (KM) or other type of data model. The model format may be provided in any data format suitable for storage in a data warehouse. According to an embodiment, the data plane may also include a data and configuration user interface and a mapping and configuration database.

[0051] According to an embodiment, the data plane is responsible for performing extract, transform, and load (ETL) operations, including extracting transactional data from an organization's enterprise software applications or data environment (such as, for example, business productivity software applications and corresponding transactional databases provided in a SaaS environment), transforming the extracted data into a model format, and loading the transformed data into a customer model of a data warehouse.

[0052] For example, according to an embodiment, each customer (tenant) of the environment may be associated with its own customer tenancy within the data warehouse associated with its own customer schema; and may additionally be provided read-only access to the data analysis schema, which may be updated periodically or otherwise via a data pipeline or process (e.g., ETL processing).

[0053] According to an embodiment, a data pipeline or process can be scheduled to execute at certain intervals (e.g., hourly / daily / weekly) to extract transactional data from an enterprise software application or data environment (such as, for example, a business productivity software application and corresponding transactional database 106 provided in a SaaS environment).

[0054] According to an embodiment, the extraction process 108 can extract transaction data, so that the extraction data pipeline or process can insert the extracted data into a data staging area, which can serve as a temporary staging area for the extracted data. Data quality components and data protection components can be used to ensure the integrity of the extracted data. For example, according to an embodiment, the data quality component can perform validation on the extracted data while the data is temporarily held in the data staging area.

[0055] According to an embodiment, when the extraction process has completed its extraction, a transformation process may begin using a data transformation layer to transform the extracted data into a model format to be loaded into a customer schema of the data warehouse.

[0056] According to an embodiment, the data pipeline or process can operate in conjunction with the data transformation layer to transform the data into a model format. A mapping and configuration database can store data mappings and metadata that define the data model used by the data transformation. A data and configuration user interface (UI) can facilitate access and modification to the mapping and configuration database.

[0057] According to an embodiment, the data transformation layer can transform the extracted data into a format suitable for loading into the customer schema of the data warehouse, for example, according to the data model. During the transformation, the data transformation can appropriately perform dimension generation, fact generation, and aggregation generation. Dimension generation can include generating dimensions or fields for loading into the data warehouse instance.

[0058] According to an embodiment, after transforming the extracted data, the data pipeline or process can perform a warehouse loading process 150 to load the transformed data into the customer schema of the data warehouse instance. After the transformed data is loaded into the customer schema, the transformed data can be analyzed and used in various additional business intelligence processes.

[0059] In order to provide data analysis or business intelligence data or develop software analysis applications, different customers of the data analysis environment may have different requirements on how their data is classified, aggregated or transformed. According to an embodiment, in order to support these different requirements, the semantic layer 180 may include data defining the semantic model of customer data; this is used to help users understand and access the data using commonly understood business terms; and provide customized content to the presentation layer 190.

[0060] According to an embodiment, the semantic model can be defined as a BI repository (RPD) file in, for example, an Oracle environment, having metadata defining the logical schema, physical schema, physical to logical mapping, aggregate table navigation, and / or other constructs that implement the various physical layers, business model and mapping layers, and presentation layer aspects of the semantic model.

[0061] According to an embodiment, customers may perform modifications to their data source models to support their specific requirements, for example by adding custom facts or dimensions associated with the data stored in their data warehouse instance; and the system may extend the semantic model accordingly.

[0062] Depending on the embodiment, the presentation layer may enable access to data content using, for example, a software analytics application, a user interface, a dashboard, key performance indicators (KPIs); or other types of reports or interfaces as may be provided by a product (such as, for example, Oracle Analytics Cloud or Oracle Application Analytics).

[0063] Business Intelligence Server

[0064] According to an embodiment, query engine 18 (eg, an OBIS instance) operates in a federated query engine manner to service analytical queries or requests from clients within, for example, an Oracle Analytics Cloud environment for data stored in a database.

[0065] According to an embodiment, the OBIS instance can push down operations to supported databases based on the query execution plan 56, where the logical query may include a structured query language (SQL) statement received from the client; and the physical query includes a database-specific statement that the query engine sends to the database to retrieve data when processing the logical query. In this way, the OBIS instance translates the business user query into an appropriate database-specific query language (e.g., Oracle SQL, SQLServer SQL, DB2 SQL, or Essbase MDX). The query engine (e.g., OBIS) can also support internal execution of SQL operators that cannot be pushed down to the database.

[0066] According to an embodiment, a user / developer may interact with a client computer device 10 that includes computer hardware 11 (e.g., processor, storage, memory), a user interface 12, and an application 14. A query engine or business intelligence server (such as an OBIS) typically operates to process inbound (e.g., SQL) requests against a database model, construct and execute one or more physical database queries, process the data appropriately, and then return the data in response to the request.

[0067] To accomplish this, according to an embodiment, a query engine or business intelligence server may include various components or features, such as metadata or a logical or business model that describes data that can be used as a query subject area; a request generator that receives incoming queries and converts them into physical queries for use with connected data sources; and a navigator that receives incoming queries, navigates the logical model, and generates those physical queries that best return the data required for a particular query.

[0068] For example, according to an embodiment, by creating a simplified star schema business model on various data sources so that users can query data as if it originated from a single source, a query engine or business intelligence server can adopt a logical model mapped to the data in the data warehouse. The information can then be returned to the presentation layer as a subject area according to the business model layer mapping rules.

[0069] According to an embodiment, a query engine (e.g., OBIS) may process a query against a database according to a query execution plan, which may include various child (leaf) nodes, generally referred to as RqLists in various embodiments, such as:

[0070] Execution plan:

[0071]

[0072]

[0073] Within a query execution plan, each execution plan component (RqList) represents a query block in the query execution plan and is typically translated into a SELECT statement. An RqList may have nested sub-RqLists, similar to how a SELECT statement can select from a nested SELECT statement.

[0074] According to an embodiment, the query engine can talk to different databases and use a data source specific code generator for each of these databases. A typical strategy is to ship as much SQL execution as possible to the database by sending it as part of the physical query - this reduces the amount of information returned to the OBIS server.

[0075] According to an embodiment, during operation, the query engine or business intelligence server can create a query execution plan, which can then be further optimized, such as to perform data aggregation required to respond to the request. Before returning the results to the calling application, for example via an ODBC interface, the data can be combined and further calculations applied.

[0076] Depending on the embodiment, complex multi-pass requests requiring multiple data sources may require the query engine or business intelligence server to decompose the query, determine which sources, multi-pass calculations and aggregations can be used, and generate a logical query execution plan across multiple databases and physical SQL statements, where the results can then be passed back and further joined or aggregated by the query engine or business intelligence server.

[0077] Figure 2 Further illustrated is an example data analysis environment according to an embodiment.

[0078] like Figure 2 As shown in , according to an embodiment, the provisioning component may also include a provisioning application programming interface (API) 112, a plurality of workers 115, a metering manager 116, and a data plane API 118, as further described below. When a command, instruction, or other input is received at the console interface, the console interface may communicate with the provisioning API, for example, by making an API call, to provision services in the SaaS environment or make configuration changes to the provisioned services.

[0079] According to an embodiment, the data plane API may communicate with the data plane. For example, according to an embodiment, provisioning and configuration changes for services provided by the data plane may be communicated to the data plane via the data plane API.

[0080] According to embodiments, the metering manager may include various functions for metering services provisioned via the control plane and usage of the services. For example, according to embodiments, the metering manager may record usage of processors provisioned via the control plane for a particular customer (tenant) over time for billing purposes. Similarly, the metering manager may record the amount of storage space of a data warehouse partitioned for customer use of a SaaS environment for billing purposes.

[0081] According to an embodiment, the data pipeline or processing provided by the data plane may include a monitoring component 122, a data staging component 124, a data quality component 126, and a data projection component 128, as further described below.

[0082] According to an embodiment, the data transformation layer may include a dimension generation component 136 , a fact generation component 138 , and an aggregation generation component 140 , as described further below. The data plane may also include a data and configuration user interface 130 and a mapping and configuration database 132 .

[0083] According to an embodiment, the data warehouse may include a default data analysis schema (referred to herein as an analysis warehouse schema according to some embodiments) 162 and, for each customer (tenant) of the system, a customer schema 164 .

[0084] According to an embodiment, in order to support multiple tenants, the system may enable the use of multiple data warehouses or data warehouse instances. For example, according to an embodiment, a first warehouse customer lease for a first tenant may include a first database instance, a first staging area, and a first data warehouse instance among multiple data warehouses or data warehouse instances; and a second customer lease for a second tenant may include a second database instance, a second staging area, and a second data warehouse instance among multiple data warehouses or data warehouse instances.

[0085] According to an embodiment, based on the data model defined in the mapping and configuration database, the monitoring component can determine the dependencies of several different data sets (data sets) to be transformed. Based on the determined dependencies, the monitoring component can determine which of the several different data sets should be transformed into the model format first.

[0086] For example, according to an embodiment, if a first model dataset does not include a dependency on any other model dataset; and a second model dataset includes a dependency on the first model dataset; then the monitoring component may determine to transform the first dataset before transforming the second dataset to accommodate the dependency of the second dataset on the first dataset.

[0087] For example, according to an embodiment, a dimension may include data categories such as, for example, "name," "address," or "age." Fact generation includes the generation of values ​​or "measures" that the data can take. Facts can be associated with appropriate dimensions in a data warehouse instance. Aggregation generation includes the creation of a data map that computes aggregations of the transformed data with existing data in the customer schema of the data warehouse instance.

[0088] According to an embodiment, once any transformations are in place (as defined by the data model), a data pipeline or process may read the source data, apply the transformations, and then push the data to the data warehouse instance.

[0089] According to an embodiment, data transformations can be expressed as rules, and once transformed, values ​​can be intermediately maintained at a staging area where a data quality component and a data projection component can validate and check the integrity of the transformed data before it is uploaded to a customer schema at a data warehouse instance. Monitoring can be provided while the extract, transform, load process is running, for example, at many compute instances or virtual machines. Dependencies can also be maintained during the extract, transform, load process, and data pipelines or processes can participate in such ordering decisions.

[0090] According to an embodiment, after transforming the extracted data, the data pipeline or process can perform a warehouse loading process to load the transformed data into the customer schema of the data warehouse instance. After the transformed data is loaded into the customer schema, the transformed data can be analyzed and used in various additional business intelligence processes.

[0091] Figure 3 Further illustrated is an example data analysis environment according to an embodiment.

[0092] like Figure 3 As shown in , according to an embodiment, using data pipeline processing, data can originate from, for example, a customer's (tenant's) enterprise software application or data environment (106); or as custom data 109 originating from one or more customer-specific applications 107; and is loaded into a data warehouse instance, which in some examples includes using an object storage device 105 to store the data.

[0093] According to an embodiment of an analytical environment such as, for example, Oracle Analytics Cloud (OAC), a user can create a data set that uses tables from different connections and schemas. The system uses the relationships defined between these tables to create relationships or joins in the data set.

[0094] According to an embodiment, for each customer (tenant), the system uses a data analysis schema maintained and updated by the system within the system / cloud tenancy 114 to pre-populate a data warehouse instance for the customer based on an analysis of data within the customer's enterprise application environment and within the customer tenancy 117. Thus, the data analysis schema maintained by the system enables data to be retrieved from the customer's environment through a data pipeline or process and loaded into the customer's data warehouse instance.

[0095] According to an embodiment, the system also provides each customer of the environment with a customer model that is easily modified by the customer and allows the customer to supplement and utilize data within its own data warehouse instance. For each customer, its resulting data warehouse instance operates as a database whose content is partially controlled by the customer; and partially controlled by the environment (system).

[0096] For example, according to an embodiment, a data warehouse (e.g., ADW) may include data analysis models and, for each customer / tenant, customer models derived from their enterprise software applications or data environments. Data provisioned in a data warehouse lease (e.g., ADW cloud lease) may only be accessed by that tenant; while allowing access to various features of the shared environment, such as ETL-related features or other features.

[0097] According to an embodiment, to support multiple customers / tenants, the system enables the use of multiple data warehouse instances; where, for example, a first customer lease may include a first database instance, a first staging area, and a first data warehouse instance; and a second customer lease may include a second database instance, a second staging area, and a second data warehouse instance.

[0098] According to an embodiment, for a particular customer / tenant, after extracting its data, the data pipeline or process can insert the extracted data into the tenant's data staging area, which can act as a temporary staging area for the extracted data. Data quality components and data protection components can be used to ensure the integrity of the extracted data; for example, by performing validation on the extracted data while the data is temporarily held in the data staging area. When the extraction process completes its extraction, a transformation process can be started using the data transformation layer to transform the extracted data into a model format in the customer schema to be loaded into the data warehouse.

[0099] Figure 4 Further illustrated is an example data analysis environment according to an embodiment.

[0100] like Figure 4 As shown in , according to an embodiment, using the data pipeline processing as described above, for example, extracting data from a customer's (tenant's) enterprise software application or data environment; or as custom data originating from one or more customer-specific applications; and loading data into a data warehouse instance or refreshing data in a data warehouse, typically involves three main stages performed by an ETP service 160 or processing (including one or more extraction services 163 executed by one or more computing instances 170; transformation services 165; and load / publish services 167).

[0101] For example, according to an embodiment, a list of view objects for extraction can be submitted to, for example, an Oracle BI Cloud Connector (BICC) component via a REST call. The extracted files can be uploaded to an object storage component, such as, for example, an Oracle Storage Service (OSS) component, for storing the data. The transformation process obtains the data file from the object storage component (e.g., OSS) and applies business logic while loading it into a target data warehouse (e.g., an ADW database) that is internal to the data pipeline or process and not exposed to the customer (tenant). The load / publish service or process obtains data from, for example, an ADW database or warehouse and publishes it to a data warehouse instance accessible to the customer (tenant).

[0102] Figure 5 Further illustrated is an example data analysis environment according to an embodiment.

[0103] like Figure 5As shown in , which illustrates the operation of a system with multiple tenants (customers) according to an embodiment, using the data pipeline processing as described above, data can originate from, for example, each of the enterprise software applications or data environments of multiple customers (tenants); and be loaded into a data warehouse instance.

[0104] According to an embodiment, for each of a plurality of customers (tenants), e.g., Customer A 180, Customer B 182, a data pipeline or process maintains a data analysis schema that is regularly updated by the system according to best practices for a particular analysis use case.

[0105] According to an embodiment, for each of a plurality of customers (e.g., customer A, B), the system uses data analysis models 162A, 162B maintained and updated by the system to pre-populate a data warehouse instance for the customer based on analysis of data within the customer's enterprise application environment 106A, 106B and within each customer's lease (e.g., customer A lease 181, customer B lease 183); thereby, a data pipeline or process retrieves data from the customer's environment and loads it into the customer's data warehouse instance 160A, 160B.

[0106] According to an embodiment, the data analysis environment also provides each of the multiple customers of the environment with a customer schema (e.g., Customer A schema 164A, Customer B schema 164B) that is easily modified by the customer and allows the customer to supplement and utilize data within its own data warehouse instance.

[0107] As described above, according to an embodiment, for each of the multiple customers of the data analysis environment, its resulting data warehouse instance operates as a database whose contents are partially controlled by the customer; and partially controlled by the data analysis environment (system); including that its database is presented as pre-filled with appropriate data that has been retrieved from its enterprise application environment to solve various analysis use cases. When the extraction process 108A, 108B for a particular customer completes its extraction, a transformation process can be started using a data transformation layer to transform the extracted data into a model format in the customer schema to be loaded into the data warehouse.

[0108] According to an embodiment, activation plan 186 may be used to control the operation of a data pipeline or processing service for a customer of a specific functional area to address the specific needs of that customer (tenant).

[0109] For example, according to an embodiment, an activation plan may define multiple extract, transform, and load (publish) services or steps that are run in a specific order, at a specific time of day, and within a specific time window.

[0110] According to an embodiment, each customer may be associated with its own activation plan(s). For example, the activation plan of the first customer A may determine the tables to be retrieved from the customer's enterprise software application environment (e.g., its converged application environment), or determine how to run services and their processes in sequence; while the activation plan of the second customer B may also determine the tables to be retrieved from the customer's enterprise software application environment, or determine how to run services and their processes in sequence.

[0111] Figure 6 The use of a system to transform, analyze, or visualize data according to an embodiment is illustrated.

[0112] like Figure 6 As shown in , according to an embodiment, the systems and methods disclosed herein can be used to provide a data visualization environment 192 that enables users of the analysis environment to gain insight into analysis artifacts and the relationships between them. The model can then be used to visualize the relationships between such analysis artifacts via, for example, a user interface as a visualization or network diagram of the relationships and connections between artifacts (e.g., users, roles, DV projects, data sets, connections, data flows, sequences, ML models, ML scripts).

[0113] According to an embodiment, the client application can be implemented as software or computer readable program code executable by a computer system or processing device and has a user interface, such as, for example, a software application user interface or a web browser interface. The client application can retrieve or access data via the Internet / HTTP or other types of network connections with the analysis system, or in the example of a cloud environment, retrieve or access data via a cloud service provided by the environment.

[0114] In accordance with an embodiment, the user interface may include or provide access to various data flow action types, as described in further detail below, which enable self-service text analysis, including allowing a user to display a data set, or interact with the user interface to transform, analyze, or visualize data, such as to generate a graph, chart, or other type of data analysis or data flow visualization.

[0115] According to an embodiment, the analysis system enables retrieval, reception, or preparation of data sets from one or more data sources, for example, via one or more data source connections. Examples of data types that can be transformed, analyzed, or visualized using the systems and methods described herein include HCM, HR, or ERP data, email or text messages, or other free-format or unstructured text data provided at one or more databases, data storage services, or other types of data repositories or data sources.

[0116] For example, according to an embodiment, a request for data analysis or visualization information may be received via a client application and user interface as described above and communicated to an analysis system (in the example of a cloud environment, via a cloud service). The system may retrieve an appropriate data set to address the user / business context for generating and returning the requested data analysis or visualization information to the client. For example, the data analysis system may use, for example, a SELECT statement or a logical SQL instruction to retrieve a data set.

[0117] According to an embodiment, the system can create a model or data flow that reflects an understanding of the data flow or input data set by applying various algorithmic processing to generate a visualization or other type of useful information associated with the data. The model or data flow can be further modified within the data set editor 193 by applying various processing or techniques to the data flow or input data set, including, for example, one or more data flow actions 194, 195 or steps that operate on the data flow or input data set. The user can interact with the system via a user interface to control the use of data flow actions to generate data analysis, data visualization 196, or other types of useful information associated with the data.

[0118] According to an embodiment, a dataset is a self-service data model that a user can build for data visualization and analysis needs. The dataset contains data source connection information, tables and columns, data enrichment and transformation. Users can use the dataset in multiple workbooks and data flows.

[0119] According to an embodiment, when users create and build datasets, they can, for example: choose between multiple types of connections or spreadsheets; create a dataset based on data from multiple tables in a database connection, Oracle data source, or local subject area; or create a dataset based on data from tables in different connections and subject areas.

[0120] For example, according to an embodiment, a user may build a dataset that includes tables from an autonomous data warehouse connection, tables from a Spark connection, and tables from a local subject area; specify joins between tables; and transform and enrich columns in the dataset.

[0121] Depending on the embodiment, additional artifacts, features, and operations associated with a data set may include, for example:

[0122] View available connections: Datasets use one or more connections to data sources to access and serve data for analysis and visualization. The user list of connections includes the connections they have built and the connections they have permission to access and use.

[0123] Create a dataset from a connection: When users create a dataset, they can add tables, add joins, and enrich data from one or more data source connections.

[0124] Add multiple connections to a dataset: A dataset can include more than one connection. Adding more connections allows users to access and join all the tables and data they need to build a dataset. Users can add more connections to a dataset that supports multiple tables.

[0125] Create dataset table joins: Joins indicate the relationship between the tables of a dataset. If the user is creating a dataset based on facts and dimensions, and if joins already exist in the source tables, then joins are automatically created in the dataset. If the user is creating a dataset from multiple connections and schemas, then they can manually define joins between tables.

[0126] According to an embodiment, a user can use data flows to create a data set by combining, organizing, and integrating data. Data flows enable users to organize and integrate data to produce a tidy data set that they or other users can visualize.

[0127] For example, depending on an embodiment, a user might use data flows to: create data sets; combine data from different sources; aggregate data; and train machine learning models or apply predictive machine learning models to their data.

[0128] According to an embodiment, the dataset editor as described above allows a user to add actions or steps, where each step performs a specific function, such as adding data, joining tables, merging columns, transforming data, or saving data. As the user adds or changes each step, each step is validated. When they have configured the data flow, they can execute the data flow to generate or update the dataset.

[0129] According to an embodiment, a user can collate data from a dataset, subject area, or database connection. A user can execute a data flow individually or in sequence. A user can include multiple data sources in a data flow and specify how to join them. A user can save the output data of a data flow in a dataset or supported database type.

[0130] Depending on the embodiment, additional artifacts, features, and operations associated with the data flow may include, for example:

[0131] Add Column: Add a custom column to the target dataset.

[0132] Add Data: Adds a data source to the dataflow. For example, if the user is merging two data sets, they add both data sets to the dataflow.

[0133] Aggregate: Create group totals by applying an aggregate function; for example, count, sum, or average.

[0134] Branching: Creating multiple outputs from a data flow.

[0135] Filtering: Select only the data that interests you.

[0136] Joins: Combine data from multiple data sources using database joins based on common columns.

[0137] Graph Analytics: Perform geospatial analysis, such as calculating the distance or number of hops between two vertices.

[0138] The above are provided only as examples; other types of steps may be added to the data flow to transform the data set or provide data analysis or visualization, depending on the embodiment.

[0139] Dataset analysis and visualization

[0140] According to an embodiment, the system provides functionality that allows a user to generate datasets, analyses, or visualizations for display within a user interface, such as browsing datasets or data originating from multiple data sources.

[0141] Figure 7-Figure 18 Various examples of user interfaces for use with a data analysis environment are illustrated according to embodiments.

[0142] Figure 7-Figure 18 The user interfaces and features shown in are provided by way of example to illustrate the various features described herein; alternative examples of user interfaces and features may be provided according to various embodiments.

[0143] like Figure 7-Figure 8 As shown in , according to an embodiment, a user may access a data analysis environment, for example, to submit an analysis or query against organizational data.

[0144] For example, according to an embodiment, a user can select between various types of connections to create a dataset based on data from, for example, a database connection, an Oracle subject area, an Oracle ADW connection, or a table in a spreadsheet, file, or other type of data source. In this way, the dataset operates as a self-service data model from which the user can build data analysis or visualization.

[0145] like Figure 9-10 As shown in , according to an embodiment, a dataset editor can display a list of connections that a user is permitted to access, and allow the user to create or edit a dataset containing tables, joins, and / or enriched data. The editor can display the schema and tables of a data source connection, from which the user can drag and drop them into the dataset diagram. If a particular connection does not itself provide a list of schemas and tables, the user can use manual queries to find the appropriate tables. Adding a connection provides the ability to access and join its associated tables and data to build a dataset.

[0146] like Figure 11-12As shown in, according to an embodiment, in a data set editor, a join diagram displays the tables and joins in a data set. For example, by creating joins based on column name matches found between tables, joins defined in a data source can be automatically created between tables in a data set.

[0147] According to an embodiment, when a user selects a table, the preview data area displays a sample of the data for that table. The displayed join links and icons indicate which tables are joined and the type of join used. The user can create a join by dragging and dropping one table onto another; click a join to view or update its configuration; or click the Type property of a column to change its type, for example, from a measure to an attribute.

[0148] According to an embodiment, the system can generate source-specific optimized queries for visualizations, where the dataset is treated as a data model and only those tables required to satisfy the visualization are used in the query.

[0149] By default, the granularity of a dataset is determined by the table with the lowest granularity. Users can create measures in any table in a dataset; however, this may result in duplication of measures on one side of a one-to-many or many-to-many relationship. Fig.13 In the embodiment shown in , to solve this problem, the user can set the table on the cardinality side to preserve granularity to maintain its level of detail.

[0150] like Fig.14 As shown in , according to an embodiment, a data set table can be associated with a data access setting that determines whether the system will load the table into a cache, or alternatively, whether the table will receive its data directly from a data source.

[0151] According to an embodiment, when automatic cache mode is selected for a table, the system loads or reloads the table data into the cache, which provides faster performance when refreshing the table's data (e.g., from a workbook) and causes the reload menu option to be displayed at the table and dataset level.

[0152] According to an embodiment, when real-time mode is selected for a table, the system retrieves table data directly from the data source; and the source system manages the data source query for the table. This option is useful when the data is stored in a high-performance data warehouse (such as, for example, Oracle ADW); and it also ensures that the latest data is used.

[0153] According to an embodiment, when a data set uses multiple tables, some tables may use automatic cache while other tables may contain live data. During a reload of multiple tables using the same connection, if the data reload fails on one table, any tables that are currently set to use automatic cache will switch to using live mode to retrieve their data.

[0154] According to an embodiment, the system allows users to enrich and transform their data before it is available for analysis. When a workbook is created and a dataset is added to it, the system performs column-level profiling on a representative sample of the data. After profiling the data, the user can implement transformation and enrichment recommendations provided for identifiable columns in the dataset; such as, for example, GPS enrichments such as latitude and longitude of a city or zip code.

[0155] According to an embodiment, data transformation and enrichment changes applied to a dataset can affect workbooks and data flows that use the dataset. For example, when a user opens a workbook that shares a dataset, they receive a message indicating that the workbook uses updated or refreshed data.

[0156] According to an embodiment, dataflows provide a means to organize and integrate data to produce a collated dataset that your users can visualize. For example, a user might use a dataflow to create a dataset, combine data from different sources, aggregate data, or train a machine learning model or apply a predictive machine learning model to their data.

[0157] like Fig.15 As shown in, according to an embodiment, within a data flow, each step performs a specific function, such as to add data, join tables, merge columns, transform data, or save data. Once configured, the data flow can be executed to perform operations to generate or update a data set, including, for example, using SQL operators (such as BETWEEN, LIKE, IN), conditional expressions, or functions.

[0158] According to an embodiment, data flows can be used to merge data sets, clean data, and output the results to a new data set. Data flows can be executed individually or in a sequence. If any data flow within a sequence fails, all changes made in the sequence will be rolled back.

[0159] like Figure 16-Figure 18 As shown in , according to an embodiment, visualizations can be displayed within a user interface, for example to browse a data set or data originating from multiple data sources, and to add insights.

[0160] For example, according to an embodiment, a user may create a workbook, add a data set, and then drag and drop its columns onto a canvas to create a visualization. The system may automatically generate a visualization based on the contents of the canvas, with one or more visualization types automatically selected for the user to select. For example, if a user adds a revenue measure to the canvas, the data element may be placed in the value area of ​​the grammar panel and the Tile visualization type may be selected. The user may continue to add data elements directly to the canvas to build a visualization.

[0161] According to an embodiment, the system can provide automatically generated data visualizations (auto-generated insights, auto-insights) by suggesting visualizations that are expected to provide the best insights for a particular data set. A user can view an automatically generated summary of insights, for example, by hovering a mouse over an associated visualization in a workbook canvas.

[0162] Automatic enrichment of datasets

[0163] According to embodiments, systems and methods are described herein for automatically enriching data sets in a data analysis environment using system knowledge data.

[0164] According to an embodiment, users of a data analysis environment (particularly business users and users of, for example, data visualization) often lack or are unaware of additional knowledge that may improve data visualization.

[0165] According to an embodiment, the systems and methods described herein can provide a "no-click" automated way to enrich customer data with a knowledge repository. Such a knowledge repository can be delivered to customers using, for example, a delivery system (such as the three delivery mechanisms described below).

[0166] First, according to embodiments, the system and method can deliver a static knowledge data set. Such a static knowledge data set can be automatically and dynamically joined with customer data to enrich the customer data. Such a static knowledge data set can include, for example, basic calculations on the customer data set, such as, for example, automatically calculating and providing a suggested minimum, maximum, average, or standard deviation.

[0167] Second, according to embodiments, the systems and methods can deliver out-of-the-box third-party integrations, for example, making runtime REST calls to external services and dynamically pulling rich knowledge columns based on auto-join keys detected in customer data. The auto-join key detection algorithm is not limited to matching column names, but is more advanced by detecting the semantic type of the column based on sampled data profiling. Currency conversion can be one of the target use cases for this type of integration. In some embodiments, this auto-join is achieved by leveraging a service (such as Oracle BI Server) that has the ability to combine heterogeneous sources and present them as a whole to the end user.

[0168] Third, in accordance with an embodiment, the systems and methods may provide clients with a well-defined integration handshake to enable customized integration with external, third-party datasets and services or the client's internal dataset hosting services.

[0169] According to embodiments, systems and methods may provide multiple system key-value system data sets, where the key is a common attribute that may be used elsewhere. For example, a useful key may be related to geography or geography plus time elements. A key consisting of "zip code" or "country / state / city / year" is another good example. The value may reflect some domain knowledge.

[0170] According to an embodiment, when a user is using his or her own dataset to construct a visualization, the system and method can dynamically analyze it, find columns containing similar keys that can be used for joins, and in addition to customer columns, the system and method can also provide columns from a knowledge dataset (e.g., external to the user dataset) for constructing the visualization.

[0171] Fig.19 A system for automatically enriching a data set in a data analysis environment using system knowledge data according to an embodiment is illustrated.

[0172] like Fig.19 As shown in , according to an embodiment, architecture 1900 can be utilized in which multiple enhancements can be made or suggested to a user's data set to generate one or more data visualization (DV) workbooks.

[0173] According to an embodiment, when a user accesses or uploads a data set, the system and method may perform one or more of data preparation enhancement, OBIS function calculation, data set insight statistics, and external enrichment. It should be noted that each of these functions can be selectively presented to the user as an optional enhancement of their data set for data visualization (e.g., providing suggested enhancements as selectable options), or the enhancements can be automatically applied to the user's data visualization.

[0174] According to an embodiment, data preparation enrichment, OBIS function calculations, and data set insight statistics may include multiple functions and enhancements 1910 that are automatically applied to existing or newly uploaded client data sets. For example, Fig.19 As shown in , the dataset "AccidentsDemoFinal" includes a knowledge base that contains city and state zip codes, as well as other data such as, for example, vehicle speed limits or vehicle occupants. Data preparation enrichment, OBIS function calculations, and dataset insight statistics can automatically calculate multiple statistics such as minimum, maximum, mean, standard deviation (e.g. Fig.19 , for the sum, average, minimum, maximum, and various percentiles of the number of occupants). By automatically providing this functionality, it enables users who would not otherwise know or be able to see or use the join functionality (due to lack of product knowledge) to generate better data visualizations more quickly and easily.

[0175] According to an embodiment, in addition to the functions and calculations automatically applied to the user data set as described above, the systems and methods described herein may also extract data from sources external to the user data set. Fig.19 As shown in , upon detecting a zip code column in a user dataset, the system and method can automatically pull census data and automatically add or suggest a join between the census data and the user dataset.

[0176] According to an embodiment, although census data is used in the above examples, a person of ordinary skill in the art will readily appreciate that additional data sources may be used in the described embodiments, such as, for example, demographic data, public health information, publicly available statistics.

[0177] According to an embodiment, for example, AccidentsDemoFinal is an uploaded dataset. The dataset contains a data column named "zip". From this "zip" data column within the customer dataset, the system can automatically detect the data column and extract external data, such as from census data, and make this newly accessed data accessible. This is an example of dynamic, non-persistent data. Upon selection, the system creates a join between the customer's dataset and the external data.

[0178] Depending on the embodiment, data external to the user dataset may be extracted directly from a source (eg, the U.S. Census website), or such data may be extracted from another local source (such as, for example, a database containing previously extracted census data).

[0179] Fig. 20 is a flowchart of a method for automatically enriching a dataset using system knowledge data according to an embodiment.

[0180] like Fig. 20 As shown in , according to an embodiment, at step 2010, the method may provide, at a computer including a microprocessor, a data analysis environment operable to display a data visualization associated with a data set.

[0181] According to an embodiment, at step 2020, the method may access the user data set through the data analysis environment.

[0182] According to an embodiment, at step 2030, the method may analyze at least a portion of the user data set through a data analysis environment.

[0183] According to an embodiment, at step 2040, the method may determine, through the data analysis environment, one or more data enrichment actions to be performed, wherein the determination of the one or more data enrichment actions is based on the analysis of the portion of the user data set.

[0184] According to an embodiment, at step 2050, the method may automatically perform the determined one or more data enrichment actions by the analysis environment; and

[0185] According to an embodiment, at step 2060, the method may generate one or more data visualizations of the user dataset, wherein one or more data enrichment actions determined and performed on the user dataset are reflected within the generated one or more data visualizations.

[0186] According to various embodiments, the teachings herein can be implemented using one or more computers, computing devices, machines or microprocessors, including one or more processors, memories and / or computer-readable storage media programmed according to the teachings herein. Skilled programmers can readily prepare appropriate software coding based on the teachings of the present disclosure, as will be apparent to those skilled in the art of software.

[0187] In some embodiments, the teachings herein may include a computer program product, which is (one or more) non-transitory computer-readable storage medium having instructions stored thereon / in which instructions may be used to program a computer to perform any of the processes of the present teachings. Examples of such storage media may include, but are not limited to, a hard disk drive, a hard drive, a fixed disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, or other types of storage media or devices suitable for non-transitory storage of instructions and / or data.

[0188] The above description is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the scope of protection to the precise form disclosed. Further modifications and variations will be apparent to those skilled in the art.

[0189] The embodiments were chosen and described in order to best explain the principles taught herein and their practical applications, thereby enabling others skilled in the art to understand the various embodiments and with various modifications as are suited to the particular use contemplated. The scope is intended to be defined by the following claims and their equivalents.

Claims

1. A system for data analysis including automatic enrichment of a data set, comprising: a computer including a microprocessor and a data analysis environment disposed on the microprocessor and operable to display a data visualization associated with the data set; wherein a user data set specified by a user is accessed through a data analysis environment; The data analysis environment includes: performing analysis on at least a portion of the user data set, and determining one or more data enrichment actions to be performed on the user data set based on the analysis of the user data set; automatically performing at least one of the determined enrichment actions; as well as generating one or more data visualizations of the user data set; The one or more data enrichment actions determined and performed on the user data set are reflected within the generated one or more data visualizations. 2 . The system of claim 1 , wherein the user data set comprises one of a data set that is triggered to be uploaded to the data analysis environment and a data set that pre-exists at the data analysis environment. 3 . The system of claim 2 , wherein at least one of the determined one or more data enrichment actions that is performed comprises performing automatic calculations on at least one data column within the data set. 4 . The system of claim 2 , wherein at least one of the determined one or more data enrichment actions that is performed comprises automatically performing a join between the user dataset and a data column external to the user dataset.

5. The system of claim 4, wherein the data columns external to the data set are identified based on at least the data columns of the user data set.

6. The system of claim 5, wherein the data columns external to the data set are retrieved from a source external to the data analysis environment.

7. The system of claim 5, wherein the data columns external to the data set are retrieved from a source internal to the data analysis environment.

8. A method for automatically enriching a data set for use in data analysis, comprising: providing, at a computer including a microprocessor, a data analysis environment operable to display a data visualization associated with the data set; Access user data sets through the data analysis environment; analyzing at least a portion of the user data set via the data analysis environment; determining, by the data analysis environment, one or more data enrichment actions to be performed, the determination of the one or more data enrichment actions being based on an analysis of the portion of the user data set; automatically performing the determined one or more data enrichment actions by the analytical environment; as well as generating one or more data visualizations of the user data set; The one or more data enrichment actions determined and performed on the user data set are reflected within the generated one or more data visualizations. 9 . The method of claim 8 , wherein the user data set comprises one of a data set that is triggered to be uploaded to the data analysis environment and a data set that pre-exists at the data analysis environment.

10. The method of claim 9, wherein at least one of the determined one or more data enrichment actions that is performed comprises performing automatic calculations on at least one data column within the data set.

11. The method of claim 9, wherein at least one of the determined one or more data enrichment actions that is performed includes automatically performing a join between the user data set and a data column external to the user data set.

12. The method of claim 11, wherein the data columns external to the data set are identified based on at least the data columns of the user data set.

13. The method of claim 12, wherein the data columns external to the data set are retrieved from a source external to the data analysis environment.

14. The method of claim 12, wherein the data columns external to the data set are retrieved from a source internal to the data analysis environment.

15. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when read and executed by one or more computers cause the one or more computers to perform a method, the method comprising: providing, at a computer including a microprocessor, a data analysis environment operable to display a data visualization associated with the data set; Access user data sets through the data analysis environment; analyzing at least a portion of the user data set via the data analysis environment; determining, by the data analysis environment, one or more data enrichment actions to be performed, the determination of the one or more data enrichment actions being based on an analysis of the portion of the user data set; automatically performing the determined one or more data enrichment actions by the analytical environment; as well as generating one or more data visualizations of the user data set; The one or more data enrichment actions determined and performed on the user data set are reflected within the generated one or more data visualizations. 16 . The non-transitory computer-readable storage medium of claim 15 , wherein the user data set comprises one of a data set that is triggered to be uploaded to the data analysis environment and a data set that pre-exists at the data analysis environment. 17 . The non-transitory computer-readable storage medium of claim 16 , wherein at least one of the determined one or more data enrichment actions that is performed comprises performing an automatic calculation on at least one data column within the data set.

18. The non-transitory computer-readable storage medium of claim 16, wherein at least one of the determined one or more data enrichment actions that is performed comprises automatically performing a join between a user data set and a data column external to the user data set.

19. The non-transitory computer-readable storage medium of claim 18, wherein the data columns external to the data set are identified based on at least the data columns of the user data set.

20. The non-transitory computer-readable storage medium of claim 19, wherein the data columns external to the data set are retrieved from a source external to the data analysis environment.

Images