Machine learning model generation for time-dependent data

Abstract

The embodiment generates a machine learning ("ML") model. The embodiment receives training data, which includes time-dependent data and multiple dates corresponding to the time-dependent data. The embodiment divides the training data by two or more of the multiple dates to generate multiple date-divided training data. For each of the multiple date-divided training data, the embodiment divides the date-divided training data into a training dataset and a corresponding test dataset using one or more different ratios to generate multiple training / test splits. For each training / test split, the embodiment determines the difference in distribution between the training dataset and the corresponding test dataset. The embodiment then selects the training / test split with the smallest difference in distribution, and uses the selected training / test split to train and test the ML model.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 524,949, filed on July 5, 2023, the disclosure of which is incorporated herein by reference.

[0002] Field One embodiment generally relates to machine learning models, and more particularly to the generation of machine learning models.

Background Art

[0003] Background Information The process of generating or building a machine learning (「ML」) model involves multiple steps. Those steps include collecting an appropriate dataset for training the model, and pre - processing the data by performing tasks such as cleaning the data, normalizing it, and converting it into a suitable format for training. Thereafter, the dataset is divided or split into two or three parts: a training set, a validation set, and a test set. The training set is used to train the model, the validation set helps in adjusting hyperparameters and evaluating the performance of the model, and the test set is used for the final evaluation.

[0004] Next, an ML model architecture / algorithm suitable for the problem to be solved using machine learning is selected. The problem can be classification, regression, clustering, or any other type of problem. The model selected can be a decision tree, random forest, support vector machine, neural network, or any other model depending on the nature of the data and the problem.

[0005] Subsequently, the training set is used to train the selected model, and the validation set is used to evaluate the model's performance. After the model's performance is sufficient, it is evaluated using the test set. This provides an unbiased estimate of the model's performance and its ability to generalize to new data. Finally, the model can be deployed, its performance can be monitored over time, and adjustments or retraining can be performed as needed. [Overview of the Initiative] [Means for solving the problem]

[0006] overview The embodiment generates a machine learning ("ML") model. The embodiment receives training data, which includes time-dependent data and multiple dates corresponding to the time-dependent data. The embodiment splits the training data by two or more of the multiple dates to generate multiple date-split training data. For each of the multiple date-split training data, the embodiment splits the date-split training data into a training dataset and a corresponding test dataset using one or more different ratios to generate multiple train / test splits. For each of the train / test splits, the embodiment determines the difference in distribution between the training dataset and the corresponding test dataset. The embodiment then selects the train / test split with the smallest difference in distribution and uses the selected train / test split to train and test the ML model.

[0007] The accompanying drawings incorporated herein and constituting part thereof illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the boundaries of elements shown in the drawings (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundary. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Furthermore, elements may not be drawn to a constant scale. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of a system including a machine learning ("ML") model generator system according to an embodiment. [Figure 2] Figure 1 is a block diagram of the ML model generator system in the form of a computer server / system according to an embodiment of the present invention. [Figure 3] This is a block diagram of a prediction system according to one embodiment. [Figure 4] Figure 2 is a flowchart of the ML model generator module when determining the time partitioning of training data according to the embodiment. [Figure 5] This figure shows a simple example of the function according to the embodiment. [Figure 6] This figure shows an exemplary data analysis environment according to an embodiment. [Figure 7] This figure shows an exemplary data analysis environment according to an embodiment. [Figure 8] This figure shows an exemplary data analysis environment according to an embodiment. [Figure 9] This figure shows an exemplary data analysis environment according to an embodiment. [Figure 10] This figure shows an exemplary data analysis environment according to an embodiment. [Modes for carrying out the invention]

[0009] Detailed explanation The embodiment generates a machine learning ("ML") model using time-dependent datasets that are split by time for use as training, test, and validation data. The embodiment creates vector markers and determines vector distances to automatically determine how the distribution of the datasets differs between training and test / validation, and which particular date variables should undergo time splitting to optimize the performance of the generated model. The embodiment addresses the problem of unintended distribution shifts between the training, test, and validation datasets due simply to the suboptimal selection of splitting date variables for time splitting. The embodiment determines the optimal dates (e.g., purchased order approved date, promised delivery date, transaction date, payment due date, payment receipt date, etc.) for splitting the training / test datasets by determining the greatest distribution similarity between the training / test datasets.

[0010] Herein, detailed references to embodiments of the present disclosure are made, examples of which are shown in the accompanying drawings. The following detailed description provides many specific details to enable a full understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be carried out without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure the aspects of the embodiments. Wherever possible, similar reference numerals are used for similar elements.

[0011] Figure 1 shows an example of a system 100 including an ML model generator system 10 according to an embodiment. The ML model generator system 10 may be implemented within a computing environment including a communication network / cloud 154. The network 154 may be a private network that can communicate with a public network (e.g., the Internet) to access additional services 152 provided by a cloud service provider. Examples of communication networks include mobile networks, wireless networks, cellular networks, local area networks ("LAN"), wide area networks ("WAN"), other wireless communication networks, or combinations of these and other networks. The ML model generator system 10 may be managed by a service provider, such as Oracle Cloud Infrastructure ("OCI") from Oracle Corporation.

[0012] A cloud service provider tenant may be an organization or group whose members include users of services provided by the service provider. Services may include, but are not limited to, applications, resources, files, documents, data, media, or combinations thereof, or access to such services. Users may have individual accounts with the service provider, and organizations may have an enterprise account with the service provider, which may encompass or aggregate multiple individual user accounts.

[0013] System 100 further includes client devices 158, which may be any type of device capable of accessing the network 154 and taking advantage of the functionality of the ML model generator system 10 that generates ML models. As disclosed herein, a “client” (also disclosed as a “client system” or “client device”) may be a device or an application running on a device. System 100 includes a number of different types of client devices 158, each capable of communicating with the network 154.

[0014] One or more ML models 125 are running on the cloud 154, each of which is generated by the ML model generator 10. Each ML model 125 is run by a customer / client / organization on the cloud 154 and may be used to generate predictions for a corresponding customer, such as whether a particular customer's invoices will be paid on time or late. In this embodiment, the ML models 125 are accessible from a client 158 ​​via a REST API (representational state transfer application programming interface) and can function as an endpoint to the API. The ML models 125 can be any type of machine learning model and are generally trained on some training and test / validation data, and then can process additional incoming "live" data to make predictions. Examples of ML models 125 include, but are not limited to, artificial neural networks ("ANNs"), decision trees (including, but not limited to, a set of random forests and gradient boosted trees), support vector machines ("SVMs"), Bayesian networks, and others. The training data can be any set of data on which the ML model 125 can be trained (for example, a set of features with corresponding labels, such as labeled data for supervised learning). In an embodiment, the training data may be used to train the ML model 125 and generate a trained ML model 125. In an embodiment, each tenant or client may have exclusive access to its corresponding ML model 125, and the model 125 is trained using only the data provided by its corresponding client (i.e., data from other clients is not used to train the client's model).

[0015] Figure 2 is a block diagram of the ML model generator system 10 of Figure 1 in the form of a computer server / system 10 according to an embodiment of the present invention. Although shown as a single system, the functions of system 10 may be implemented as a distributed system. Furthermore, the functions disclosed herein may be implemented on separate servers or devices that can be coupled together over a network. Furthermore, one or more components of system 10 may not be included. One or more components of Figure 2 may also be used to implement any of the elements of Figure 1.

[0016] System 10 includes a bus 12 or other communication mechanism for transmitting information, and a processor 22 coupled to the bus 12 for processing information. The processor 22 may be any type of general-purpose processor or a processor for a specific purpose. System 10 further includes memory 14 for storing information and instructions executed by the processor 22. Memory 14 may consist of any combination of random access memory ("RAM"), read-only memory ("ROM"), static storage such as magnetic disks or optical disks, or any other type of computer-readable media. System 10 further includes a communication interface 20, such as a network interface card, for providing access to a network. Thus, a user may interface with System 10 directly, remotely via a network, or in any other way.

[0017] The computer-readable medium may be any available medium accessible by the processor 22, and includes both volatile and non-volatile media, removable and non-removable media, and communication media. The communication medium may include any information distribution medium, which may contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other carrier mechanism.

[0018] Processor 22 is further coupled to a display 24, such as a liquid crystal display (“LCD (Liquid Crystal Display)”), via bus 12. A keyboard 26 and a cursor control device 28, such as a computer mouse, are further coupled to bus 12 to enable a user to interface with system 10.

[0019] In one embodiment, memory 14 stores software modules that provide functionality when executed by processor 22. These modules include an operating system 15 that provides operating system functionality to system 10. These modules further include an ML model generator module 16 that generates one or more ML models, and all other functions disclosed herein. System 10 may be part of a larger system. Thus, system 10 may include one or more additional functional modules 18, such as the generated ML models, or a business intelligence application or a data warehouse application (e.g., Oracle's “Fusion Analytics Warehouse”) that utilizes the generated ML models. A file storage device or database 17 is coupled to bus 12 and provides centralized storage for modules 16 and 18, including the training data used to generate the ML models. In one embodiment, database 17 is a relational database management system (“RDBMS (relational database management system)”) and can manage the stored data using Structured Query Language (“SQL (Structured Query Language)”).

[0020] In an embodiment, the communication interface 20 provides a bi-directional data communication link to a network link 35 connected to a local network 34. For example, the communication interface 20 may be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line or Ethernet (registered trademark). As another example, the communication interface 20 may be a local area network (LAN) card for providing a data communication connection with a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 20 transmits and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0021] The network link 35 typically provides data communication to one or more other data devices via one or more networks. For example, the network link 35 may provide a connection to a host computer 32 via the local network 34 or to a data device operated by an Internet service provider (ISP) 38. The ISP 38 then provides a data communication service via the Internet 36. Both the local network 34 and the Internet 36 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals via various networks that carry digital data between the computer system 10 and the signals on the network link 35 via the communication interface 20 are exemplary forms of transmission media.

[0022] System 10 can send messages and receive data, including program code, via the network, network link 35, and communication interface 20. In the internet example, server 40 may send the requested code for an application program via the internet 36, ISP 38, local network 34, and communication interface 20. The received code may be executed by processor 22 when received and / or stored in database 17 or other non-volatile storage for later execution.

[0023] In one embodiment, system 10 is a computing / data processing system that includes applications for an enterprise organization or a collection of distributed applications, and may also implement logistics, manufacturing, and inventory management functions. The applications and computing system 10 may be configured to operate locally or implemented as a cloud-based network system in, for example, an "IAAS (infrastructure-as-a-service)", "PAAS (platform-as-a-service)", "SAAS (software-as-a-service)" architecture, or other types of computing solutions.

[0024] Figure 3 is a block diagram of a prediction system according to one embodiment. The system 300 includes a machine learning model 302, training data 304, input data 306, predictions 308, and observation data 310. In some embodiments, the machine learning model 302 may be a designed model that includes one or more machine learning elements (e.g., a neural network, support vector machine, Bayesian network, random forest classifier, gradient boosting classifier, etc.) or a single ML model. The training data 304 may be any set of data on which the machine learning model 302 can be trained (e.g., a set of features with corresponding labels, such as labeled data for supervised learning). In embodiments, the training data 304 is time-dependent data. The training data 304 is divided into a test / validation dataset 305 and a training dataset 307 according to the functionality disclosed below. The training dataset 307 is used to train the machine learning model 302, and the test / validation dataset 305 is used to test and / or validate the trained ML model 302 and / or adjust or retrain it as necessary. In this embodiment, the splitting of the training data into a test / validation dataset 305 and a training dataset 307 is performed by the ML model generator system 10.

[0025] In some embodiments, a prediction 308 is observed (310), triggering an update of the training data 304. The updated training data 304 can then be used to retrain the ML model 302.

[0026] In some embodiments, the design of the machine learning model 302 may be adjusted during training, retraining, and / or updated training. For example, adjustments may include adjusting the number of hidden layers in the neural network, or adjusting the kernel computation used in the implementation of the support vector machine. This adjustment may also include adjusting / selecting the features used by the machine learning model. Embodiments include implementing various adjustment configurations (e.g., different versions of the machine learning model and features) during training to reach a configuration of the machine learning model 302 that achieves desired performance when trained (e.g., making predictions with desired accuracy, performing according to desired resource utilization / time metrics, etc.).

[0027] In some embodiments, retraining the machine learning model 302 and updating the training of the machine learning model 302 may include training the model with updated training data. For example, the training data may be updated to incorporate observed data or otherwise labeled data (e.g., for use in supervised learning). In some embodiments, the machine learning model 302 may include unsupervised learning components. For example, one or more clustering algorithms such as hierarchical clustering, K-means clustering, or an unsupervised neural network such as an unsupervised autoencoder may be implemented.

[0028] In one embodiment, the training data 304 consists of multiple data points and is time-dependent data. For example, in one embodiment, the system 300 is adapted to predict whether a customer will pay accounts receivable on schedule in accordance with one or more past orders or transactions of that customer. In this embodiment, the training data 304, which is historical data from past purchases / transactions, includes time-dependent data for those transactions, formed by multiple dates such as order approval date, transaction date, shipping date, promised receipt date, shipping receipt date, and invoice payment date. Some of these dates are fixed dates (e.g., order approval date) and some are variable dates (e.g., shipping date).

[0029] One known method of splitting data is to randomly select a portion of the training data (e.g., 80%) as the training dataset 307 and the remaining portion (e.g., 20%) as the test / validation dataset. However, for optimized model training, with such time-dependent data, if the training data 304 is time-dependent, then the training data 304 should be split by time (i.e., according to one of several dates) rather than randomly. For example, for a customer who has made 5000 purchases / transactions in the past five years, one time split might be to use the oldest 4000 transactions as training data and the most recent 1000 transactions as test data. However, to determine the most recent 1000 transactions, one of several dates that make up a transaction (e.g., transaction date, shipping date, etc.) must be used as the criterion for determining the transaction date or for splitting the data.

[0030] One problem with time-dependent data containing multiple dates is deciding which of these dates should be used to split the data. Choosing from different time splitting dates, such as order approval date, transaction date, shipping date, promised receipt date, shipping receipt date, and invoice payment date, will result in very different outcomes for model metrics and cause very different distributions of delays in payment or shipping. These distributions of delays will differ by customer and region, leading to a decrease in model performance. Therefore, the embodiment automatically determines the optimal splitting date to use to create a model that performs well over time for all customers across all regions, addressing the problem of unintended distribution shifts between training, test, and validation datasets that simply result from a suboptimal selection of splitting dates for time splitting. The embodiment determines the optimal splitting date that leads to the smallest distribution difference between training dataset 307 and test / validation dataset 305.

[0031] Figure 4 is a flowchart of the ML model generator module 16 of Figure 2 when determining the time partitioning of training data, according to an embodiment. In one embodiment, the function of the flowchart in Figure 4 is implemented by software stored in memory and other computer-readable or tangible media and executed by a processor. In other embodiments, the function may be performed by hardware (e.g., by using application-specific integrated circuits ("ASICs"), programmable gate arrays ("PGAs"), field-programmable gate arrays ("FPGAs")), or by any combination of hardware and software. The function of Figure 4 may be performed to initially train an ML model, or to retrain an ML model that has insufficient metrics / performance or otherwise requires improvement.

[0032] At 401, historical time-dependent training data 304 for a specific customer is received. In one embodiment, the time-dependent training data corresponds to each transaction in the form of a database table where each column corresponds to one of the dates in the transaction (e.g., order approval date, transaction date, shipping date, etc.). Time-dependent data is generally data that corresponds to a specific date. While the disclosed example relates to purchase transactions, other types of transactions may be used, including but not limited to procurement and sales data, supply chain data, production and manufacturing data, etc., along with embodiments of the present invention.

[0033] In 402, the training data is split by each date column (i.e., each date). In embodiments, it is generally observed that date columns called "fixed" dates, which are predetermined and do not change during the course of a transaction (e.g., order confirmation date, promised delivery date), are more suitable for splitting, while dates called "variable" dates, which are subject to change due to internal and external factors as a transaction progresses (e.g., goods receipt date, invoice end date), are less suitable for splitting.

[0034] Figure 5 shows a simplified example of the functionality of Figure 4 according to an embodiment. With respect to 402, Figure 5 shows data divided by three of the dates (receipt date 501, promised delivery date 502, and order approval date 503). In the embodiment, the data is divided by all fixed dates of time-dependent data, but those additional divisions are not shown in Figure 5.

[0035] In 404, for each date split, one or more training / test splits are created using one or more different ratios. For example, a "90 / 10" ratio split means that the first 90% of the data points (corresponding to the selected time split) of the training data 304 are in the training dataset 307, and the most recent 10% of the data points are in the test / validation dataset 305. In one embodiment, multiple splits to find the optimal split include 90 / 10, 75 / 25, and 50 / 50, but the splits are not limited to these split ratios. If computational power and temporal performance are critical, the embodiment may explore optimality for only one split, such as 90 / 10 or 75 / 25 as shown in Figure 5. If computational power allows, the embodiment may extend the search to 95 / 5, 90 / 10, 85 / 15, 80 / 20, 75 / 25, 70 / 30, etc. Figure 5 shows only the 75 / 25 split.

[0036] In 406, for each of the splits from 404, a delayed percentile vector is determined to determine the distribution shift / distribution difference between the training dataset and the test dataset. In one embodiment, the percentiles used are the 1st, 5th, 10th, 20th, 30th, 40th, 50th, 60th, 70th, 80th, 90th, 95th, and 99th percentiles. The delay is the percentile of the delay for the corresponding time split, such as delivery delay, payment delay, or other dependent variable of interest. The amount of delay in an embodiment is determined and stored in a column, including, but not limited to, items such as delay in payment, calculated as the number of days elapsed from the due date to the date the payment was actually made, or delay in shipping, calculated as the number of days from the expected shipping date to the actual shipping date.

[0037] In relation to column 560, when delays in receiving goods after they have been shipped (i.e., the dependent variable) are predicted, the total time it takes from when the goods are ordered until they are received is called "Receipt Time" ("RT"), where "MIN-RT" refers to the minimum number of days it takes for a shipment to be received, and RT_100_PCTILE in 562 refers to the highest percentile (100th percentile, or maximum value) of RT.

[0038] While the embodiment typically has one target variable predicted by the model at a time (e.g., the prediction of payment delays), multiple target variables may be predicted when using a multi-objective, multi-target model that includes multiple models within a larger model structure. The prediction of the target variable is based on the relative influence of the independent variable on the change in the target variable.

[0039] Figure 5 shows the delayed percentile vector for the training dataset 307 in row 510, the first percentile in 513, the fifth percentile in 514, the tenth percentile in 515, and so on. Similarly, the delayed percentile vector for the test dataset 305 is shown in row 511. Each trade that is part of the test dataset or training dataset has an outcome for the dependent variable (i.e., the number of days of delay for the dependent variable (e.g., delay in payment)), and the outcome for the dependent variable is grouped with the outcome for the dependent variable of all trades in the training data 304. Each outcome for the dependent variable can be placed at a percentile relative to all other outcomes for the dependent variable, and the count of these outcomes for the dependent variable below a certain percentile is the number that is placed at the corresponding position in the delayed percentile vector. Therefore, for example, in the case of date splitting by PO receipt date, with respect to the 10th percentile, (at 520) 159 of the results of the target variable in the test dataset are below the 10th percentile, and (at 521) 6 of the results in the training dataset are below the 10th percentile in the training dataset.

[0040] The embodiments are not limited to the percentiles described above. Other embodiments may calculate all 100 percentiles to determine the shift in the distribution, or use measurements such as central tendency and diffusion, mean, standard deviation, interquartile range, skewness, kurtosis, and Kullback-Leibler divergence.

[0041] At 408, the pairwise differences of the vector components and the pairwise mean (i.e., arithmetic mean or geometric mean) of the vector components are determined. In Figure 5, the pairwise differences are shown in row 530, and the pairwise mean is shown in row 531.

[0042] At 410, the pairwise decisions at 408 are normalized by dividing the pairwise difference by the pairwise mean (i.e., the ratio of difference to mean) and multiplying by a constant 100 to generate the percentage difference for each vector component. Row 532 in Figure 5 shows the ratio of mean difference to mean.

[0043] In 412, the difference score between the training dataset 307 and the test / validation dataset 305 is determined. In one embodiment, the difference score is determined using the Euclidean distance as SQRT (sum of squared differences). In another embodiment, the difference score is determined using the Manhattan distance as the absolute value of (the difference between pairs of all percentiles / the sum of the means between pairs). In Figure 5, an exemplary Manhattan distance is shown in 541 and an exemplary Euclidean distance is shown in 542.

[0044] In 414, the training / test split and date split with the shortest vector length (i.e., whichever is used is the smallest Manhattan distance or the smallest Euclidean distance) among all training / test splits and date splits is selected as the optimal training / test split and date split. In the simplified example in Figure 5, the PO approval date is selected along with the 75 / 25 split. Generally, a smaller difference score indicates a smaller shift in the distribution, and a larger difference indicates a larger shift.

[0045] In step 416, to generate an optimally trained model, the model is trained or retrained using the selected split and date column.

[0046] In one embodiment, if the scores selected in 414 exceed a predefined threshold, the data distribution shift may be considered so large that a stable model cannot be built. In one embodiment, the Euclidean distance threshold is 150. If all scores exceed the threshold, the training dataset 307 can be brought closer to the test / validation dataset 305 by making larger splits of the training dataset parts (e.g., 90 / 10) or, for example, 95 / 5 or 85 / 15, depending on the empirical proximity of the distributions, and the function in Figure 4 can be performed again. This may lead to improved scores.

[0047] Data analysis environment In one embodiment, an embodiment of the present invention is implemented as part of a cloud-based data analytics environment. Generally, data analytics enables computer-based investigation or analysis of large amounts of data to derive conclusions or other information from that data, while business intelligence tools provide business users of an organization with information that describes corporate data in a format that enables those business users to make strategic management decisions.

[0048] Examples of data analytics environments and business intelligence tools / servers include Oracle Business Intelligence Server ("OBIS"), Oracle Analytics Cloud ("OAC"), and Fusion Analytics Warehouse ("FAW"), which support functions and analytical applications such as data mining or analysis.

[0049] Figure 6 shows an exemplary data analysis environment according to an embodiment. The exemplary embodiment shown in Figure 6 is provided for the purpose of illustrating an example of a data analysis environment, and various embodiments described herein may be used in connection with these examples. According to other embodiments and examples, the methods described herein may be used in conjunction with other types of data analysis, database, or data warehouse environments. Components and processes shown in Figure 6 and further described herein in relation to various other embodiments may be provided, for example, as software or program code executable by a cloud computing system or other appropriately programmed computer system.

[0050] As shown in Figure 6, according to the embodiment, the data analysis environment 100 has computer hardware (e.g., processor, memory) 101 and includes one or more software components that act as a control plane 102 and a data plane 104, and can be provided by a computer system that provides access to a data warehouse, a data warehouse instance 160, a database 161, or other types of data sources, or can otherwise operate on such a computer system.

[0051] According to the embodiment, the control plane operates to provide control over cloud or other software products provided in a SaaS or cloud environment, such as an Oracle Analytics Cloud environment or other types of cloud environments. For example, according to the embodiment, the control plane may include a cloud environment having a console interface 110 and / or provisioning components 111 that enable access by customers (tenants).

[0052] According to the embodiment, the console interface can enable access by customers (tenants) operating a graphical user interface ("GUI") and / or a command-line interface ("CLI") or other interface, and / or may include an interface for use by a SaaS or cloud environment provider and its customers (tenants). For example, according to the embodiment, the console interface can provide an interface that enables customers to provision services for use within a SaaS environment and to configure those provisioned services.

[0053] According to the embodiment, a customer (tenant) can request the provisioning of a customer schema within a data warehouse. The customer can also provide several attributes associated with the data warehouse instance, including required attributes (e.g., login credentials) and optional attributes (e.g., size or speed), via a console interface. The provisioning component can then provision the requested data warehouse instance containing the customer schema for the data warehouse and populate the data warehouse instance with the appropriate information provided by the customer.

[0054] According to the embodiment, the provisioning component may also be used to update or edit extract, transform, and load ("ETL") processes operating on data warehouse instances and / or data planes, for example, by changing or updating the frequency of requested ETL process executions for a particular customer (tenant).

[0055] According to the embodiment, the data plane may include a data pipeline or process layer 120 and a data transformation layer 134, which together process operational or transactional data from the organization's enterprise software applications or data environment, such as business productivity software applications provisioned in a customer's (tenant's) SaaS environment. The data pipeline or process may include various functions to extract transactional data from business applications and databases provisioned in the SaaS environment and then load the transformed data into a data warehouse.

[0056] According to the embodiment, the data transformation layer may include a data model, such as a knowledge model ("KM" or other types of data model), which the system uses to transform transactional data received from business applications provisioned in a SaaS environment and corresponding transactional databases into a model format understandable by the data analytics environment. The model format may be provided in any data format suitable for storage in a data warehouse. According to the embodiment, the data plane may also include a data and configuration user interface, as well as a mapping and configuration database.

[0057] According to the embodiment, the data plane is responsible for performing ETL operations, including extracting transactional data from an organization's enterprise software applications or data environment, such as business productivity software applications and corresponding transactional databases provided in a SaaS environment, converting the extracted data into a model format, and loading the converted data into a customer schema in a data warehouse.

[0058] For example, according to one embodiment, each customer (tenant) in the environment can be associated with its own customer tenancy within the data warehouse, that is, it can be associated with its own customer schema and further provided with read-only access to the data analysis schema, which can be updated periodically or otherwise by a data pipeline or process, such as an ETL process.

[0059] According to the embodiment, the data pipeline or process may be scheduled to run at intervals (e.g., hourly / daily / weekly) to extract transactional data from an enterprise software application or data environment, such as a business productivity software application provisioned in a SaaS environment and a corresponding transactional database 106.

[0060] According to the embodiment, the extraction process 108 can extract transactional data, and the data pipeline or extraction process can then insert the extracted data into a data staging area, which can function as a temporary staging area for the extracted data. Data quality components and data protection components may be used to ensure the integrity of the extracted data. For example, according to the embodiment, the data quality component can perform validation on the extracted data while it is temporarily held in the data staging area.

[0061] According to one embodiment, when the extraction process completes the extraction, a data transformation layer may be used to initiate a transformation process to convert the extracted data into a model format that can be loaded into the customer schema of the data warehouse.

[0062] According to the embodiment, a data pipeline or process can operate in conjunction with a data transformation layer to transform data into a model format. A mapping and configuration database can store metadata and data mappings that define the data models used by the data transformation. A data and configuration user interface ("UI") can facilitate access to and modification of the mapping and configuration database.

[0063] According to the embodiment, the data transformation layer can transform the extracted data into a format suitable for loading into the customer schema of the data warehouse, for example, according to a data model. During the transformation, the data transformation can perform dimension generation, fact generation, and aggregation generation as needed. Dimension generation may include generating dimensions or fields for loading into the data warehouse instance.

[0064] According to one embodiment, after the extracted data has been transformed, the data pipeline or process can execute a warehouse loading procedure 150 to load the transformed data into the customer schema of the data warehouse instance. After loading the transformed data into the customer schema, the transformed data can be analyzed and used in various additional business intelligence processes.

[0065] Customers with different data analytics environments may have different requirements regarding how data is categorized, aggregated, or transformed for the purpose of providing data analytics or business intelligence data, or for developing software analytics applications. Depending on the embodiment, to support such different requirements, the semantic layer 180 may include data that defines a semantic model of the customer's data, which helps to help users understand and access that data using commonly understood business terminology, and can provide custom content to the presentation layer 190.

[0066] According to the embodiment, the semantic model can be defined, for example, in an Oracle environment, as a BI Repository ("RPD") file containing metadata, which defines the logical schema, physical schema, mapping between physical and logical, aggregate table navigation, and / or other constructs that implement various physical layers, business model and mapping layers, and presentation layers of the semantic model.

[0067] According to the embodiment, the customer may make changes to the data source model, for example by adding custom facts or dimensions associated with the data stored in the data warehouse instance, in order to support specific requirements, and the system may extend the semantic model accordingly.

[0068] According to the embodiment, the presentation layer can enable access to data content using, for example, software analytics applications, user interfaces, dashboards, key performance indicators ("KPIs"), or other types of reporting or interfaces that may be provided by products such as Oracle Analytics Cloud or Oracle Analytics for Applications.

[0069] According to the embodiment, the query engine 18 (e.g., OBIS) operates in a manner that is integrated to be useful for analytical queries via SQL, for example, within an Oracle Analytics Cloud environment, pushing its operation down to a supported database and translating business user queries into the appropriate database-specific query language (e.g., Oracle SQL, SQL Server SQL, DB2 SQL, or Essbase MDX). The query engine (e.g., OBIS) also supports internal execution of SQL operators that cannot be pushed down to the database.

[0070] According to the embodiment, a user / developer can interact with a client computer device 10 which includes computer hardware 11 (e.g., processor, storage, memory), a user interface 19, and an application 14. A query engine or business intelligence server, such as OBIS, typically processes inbound requests to a database model, such as SQL, and in response to the requests, constructs and executes one or more physical database queries, processes the data appropriately, and then returns the data.

[0071] To achieve this, according to the embodiment, the query engine or business intelligence server may include various components or functions, such as a logical model or business model or metadata describing data available as the subject area of ​​the query; a request generator that receives incoming queries and transforms them into physical queries for use with connected data sources; and a navigator that receives incoming queries, navigates the logical model, and generates physical queries that best return the data required for a particular query.

[0072] For example, according to one embodiment, the query engine or business intelligence server may employ a logical model mapped to data in a data warehouse by creating a simplified star schema business model across various data sources, so that users can query the data as if it originated from a single source. The information can then be returned to the presentation layer as a subject area according to the layer mapping rules of the business model.

[0073] According to the embodiments, a query engine (e.g., OBIS) can process a query against a database according to a query execution plan 56 which may include various child (leaf) nodes, typically called RqLists, in various embodiments of this specification, and which generates one or more diagnostic log entries. Each execution plan component (RqList) in the query execution plan represents a block of queries in the query execution plan, which is typically translated into a SELECT statement. An RqList may have nested child RqLists, similar to how a SELECT statement can select from nested SELECT statements.

[0074] According to the embodiment, during operation, the query engine or business intelligence server may create a query execution plan, which may then be further optimized, for example, to perform aggregation of data necessary to respond to the request. The data can be combined, and further calculations may be applied before the results are returned to the calling application, for example, via an ODBC interface.

[0075] According to one embodiment, a complex multipath request requiring multiple data sources may require a query engine or business intelligence server to decompose the query, determine which sources, multipath calculations, and aggregates may be used, and generate a logical query execution plan across multiple databases and physical SQL statements, the query engine or business intelligence server then returning the results which may be further combined or aggregated.

[0076] Figure 7 further illustrates an exemplary data analysis environment according to an embodiment. As shown in Figure 7, according to the embodiment, the provisioning component may also include a provisioning application programming interface ("API") 112, a number of workers 115, an instrumentation manager 116, and a data plane API 118, as will be further described below. The console interface can communicate with the provisioning API by making API calls when the console interface receives commands, instructions, or other inputs for provisioning services within a SaaS environment or for making configuration changes to provisioned services.

[0077] According to the embodiment, the data plane API can communicate with the data plane. For example, according to the embodiment, provisioning and configuration changes targeting services provided by the data plane can be communicated to the data plane via the data plane API.

[0078] According to the embodiment, the metering manager may include various functions for measuring services provisioned via the control plane and the usage of those services. For example, according to the embodiment, the metering manager may, for billing purposes, record the usage of processors provisioned via the control plane over time for a particular customer (tenant). Similarly, the metering manager may, for billing purposes, record the amount of storage space in a data warehouse partitioned for use by customers in a SaaS environment.

[0079] According to the embodiment, the data pipeline or process provided by the data plane may include a monitoring component 122, a data staging component 124, a data quality component 126, and a data projection component 128, as further described below.

[0080] According to one embodiment, the data transformation layer may include a dimension generation component 136, a fact generation component 138, and an aggregation generation component 140, as further described below. The data plane may also include a data and configuration user interface 130, as well as a mapping and configuration database 132.

[0081] According to one embodiment, the data warehouse may include a default data analysis schema (referred to herein, according to one embodiment, as the analysis warehouse schema) 162 and a customer schema 164 for each customer (tenant) of the system.

[0082] According to the embodiment, in order to support multiple tenants, the system can enable the use of multiple data warehouses or data warehouse instances. For example, according to the embodiment, a customer tenancy of a first warehouse in a first tenant may include a first database instance, a first staging area, and a first data warehouse instance of multiple data warehouses or data warehouse instances, while a second customer tenancy of a second tenant may include a second database instance, a second staging area, and a second data warehouse instance of multiple data warehouses or data warehouse instances.

[0083] According to one embodiment, based on a data model defined in a mapping and configuration database, the monitoring component can determine the dependencies between multiple different datasets to be transformed. Based on the determined dependencies, the monitoring component can determine which of the multiple different datasets should be transformed into the model format first.

[0084] For example, according to one embodiment, if the first model dataset does not have any dependencies on any other model datasets, and the second model dataset does have dependencies on the first model dataset, the monitoring component may decide to transform the first dataset before the second dataset to accommodate the second dataset's dependency on the first dataset.

[0085] For example, according to the embodiment, a dimension may include categories of data such as “Name,” “Address,” or “Age.” Fact generation involves generating values ​​or “measurements” that the data can receive. Facts may be associated with appropriate dimensions within the data warehouse instance. Aggregation generation involves creating data mappings that compute aggregates of transformed data into existing data in the customer schema of the data warehouse instance.

[0086] According to the embodiment, after any transformation is ready (as defined by the data model), the data pipeline or process can read the source data, apply the transformation, and then push the data to the data warehouse instance.

[0087] According to the embodiment, data transformations can be represented by rules, values ​​can be temporarily held in a staging area after the transformation is performed, and data quality components and data projection components can verify and check the integrity of the transformed data before the data is uploaded to the customer schema in the data warehouse instance. For example, monitoring can be provided when the extract, transform, and load processes are executed on multiple compute instances or virtual machines. Dependencies can also be maintained between the extract, transform, and load processes, and the data pipeline or process can accommodate such ordering decisions.

[0088] According to one embodiment, after the extracted data has been transformed, a data pipeline or process can execute a warehouse loading procedure to load the transformed data into a customer schema of a data warehouse instance. After loading the transformed data into the customer schema, the transformed data can be analyzed and used in various additional business intelligence processes.

[0089] Figure 8 further illustrates an exemplary data analysis environment according to the embodiment. As shown in Figure 8, according to the embodiment, data may be supplied, for example, from a customer's (tenant's) enterprise software application or data environment (106) using a data pipeline process, or as custom data 109 supplied from one or more customer-specific applications 107, and may be loaded into a data warehouse instance, including in some examples the use of object storage 105 for storing the data.

[0090] For example, in an implementation of an analytics environment such as Oracle Analytics Cloud ("OAC"), users can create datasets that use tables from different connections and schemas. The system then uses the relationships defined between these tables to create relationships or joins within the dataset.

[0091] According to the embodiment, for each customer (tenant), the system pre-populates the data warehouse instance for the customer based on an analysis of data within the customer's enterprise application environment and customer tenancy 117, using a data analysis schema maintained and updated by the system within the system / cloud tenancy 114. Thus, the data analysis schema maintained by the system enables data to be retrieved from the customer's environment by a data pipeline or process and loaded into the customer's data warehouse instance.

[0092] According to the embodiment, the system also provides a customer schema for each customer of the environment, which can be easily modified by the customer, enabling the customer to supplement and utilize data within its own data warehouse instance. For each customer, the resulting data warehouse instance operates as a database, its contents partially controlled by the customer and partially controlled by the environment (system).

[0093] For example, according to one embodiment, a data warehouse (e.g., ADW) may include data analysis schemas and customer schemas supplied from enterprise software applications or data environments on a per-customer / tenant basis. Data provisioned in a data warehouse tenancy (e.g., ADW cloud tenancy) is accessible only from that tenant, while simultaneously enabling access to various functions, such as ETL-related functions or other functions of the shared environment.

[0094] According to the embodiment, in order to support multiple customers / tenants, the system may enable the use of multiple data warehouse instances. For example, a first customer tenancy may include a first database instance, a first staging area, and a first data warehouse instance, while a second customer tenancy may include a second database instance, a second staging area, and a second data warehouse instance.

[0095] According to the embodiment, for a particular customer / tenant, when extracting customer / tenant data, the data pipeline or process can insert the extracted data into the tenant's data staging area, which can function as a temporary staging area for the extracted data. Data quality and data protection components may be used to ensure the integrity of the extracted data, for example, by performing validation on the data while it is temporarily held in the data staging area. When the extraction process completes the extraction, a data transformation layer may be used to initiate a transformation process to convert the extracted data into a model format that can be loaded into the customer schema of the data warehouse.

[0096] Figure 9 further illustrates an exemplary data analysis environment according to the embodiment. As shown in Figure 9, according to the embodiment, the process of extracting data from a customer's (tenant's) enterprise software application or data environment, or as custom data supplied from one or more customer-specific applications, using the data pipeline process as described above, loading the data into a data warehouse instance, or updating the data within the data warehouse, typically includes three broad stages performed by an ETP service 160 or process, including one or more extract services 163, a transformation service 165, and a load / publish service 167, which are performed by one or more compute instances 170.

[0097] For example, according to one embodiment, a list of view objects for extraction may be submitted to an Oracle BI Cloud Connector ("BICC") component, for example, via a ReST call. The extracted files may be uploaded to an object storage component, such as an Oracle Storage Service ("OSS") component, for data storage. The transformation process receives the data files from the object storage component (e.g., OSS) and applies business logic while loading those data files into a target data warehouse, such as an ADW database, which resides within a data pipeline or process and is not exposed to the customer (tenant). The load / publish service or process receives the data from, for example, the ADW database or warehouse and exposes it to a data warehouse instance accessible to the customer (tenant).

[0098] Figure 10 further illustrates an exemplary data analysis environment according to the embodiment. As shown in Figure 10, which illustrates the operation of a system including multiple tenants (customers) according to the embodiment, data may be supplied from each of the multiple customers' (tenants') enterprise software applications or data environments using, for example, the data pipeline process described above, and loaded into the data warehouse instance.

[0099] According to the embodiment, the data pipeline or process maintains a data analytics schema that is periodically updated by the system for each of multiple customers (tenants), for example, customer A180, customer B182, in accordance with best practices for specific analytical use cases.

[0100] According to the embodiment, for each of several customers (e.g., customers A and B), the system uses data analysis schemas 162A and 162B, maintained and updated by the system, to pre-populate data into data warehouse instances for the customer based on an analysis of data within the customer's enterprise application environments 106A and 106B and within each customer's tenancy (e.g., tenancy 181 for customer A, tenancy 183 for customer B), so that data can be retrieved from the customer's environment by a data pipeline or process and loaded into the customer's data warehouse instances 160A and 160B.

[0101] According to the embodiment, the data analysis environment also provides, for each of the multiple customers of the environment, a customer schema that can be easily modified by the customer (e.g., schema 164A for customer A, schema 164B for customer B), enabling the customer to supplement and utilize data within their own data warehouse instance.

[0102] As described above, according to the embodiment, for each of the multiple customers of the data analysis environment, the resulting data warehouse instance operates as a database whose contents are partially controlled by the customer and partially controlled by the data analysis environment (system), and this control includes the database appearing to be pre-populated with appropriate data extracted from the enterprise application environment to address various analytical use cases. When the extraction processes 108A, 108B for a particular customer have completed the extraction, a data transformation layer may be used to initiate a transformation process to convert the extracted data into a model format that will be loaded into the customer schema of the data warehouse.

[0103] According to one embodiment, the activation plan 186 may be used to control the operation of a data pipeline or process service for a customer with respect to a specific functional area, in order to address the specific needs of that customer (tenant).

[0104] For example, according to one embodiment, the activation plan can define a series of extraction, transformation, and loading (publishing) services or steps that are performed in a certain order, at a certain time, and within a certain time frame.

[0105] According to the embodiment, each customer may be associated with their own activation plan. For example, the activation plan of a first customer A may determine which tables to retrieve from the customer's enterprise software application environment (e.g., a Fusion Applications environment) or which services and service processes should be executed in sequence, while the activation plan of a second customer B may similarly determine which tables to retrieve from the customer's enterprise software application environment or which services and service processes should be executed in sequence.

[0106] As disclosed, embodiments optimize the training and testing of ML models using time-dependent data by creating vector markers, such as percentiles or different levels of statistical moments, for the distribution of each dataset. Embodiments find normalized vector differences using distances between vector markers, along with normalization factors along each vector dimension. Embodiments find the size of vector distances using different distance measures, such as Manhattan distance or Euclidean distance. Embodiments compare vector distances to find a variable that yields the minimum distance between the distributions of the target variable in training, testing, and validation, and subsequently train the model using selected date splits and training / test splits.

[0107] The embodiment automates the process of variable selection for time series training / test / validation splitting. The embodiment creates normalized distribution shift scores that work across all distributions in fields within a single customer's data and across all variable types, regardless of the scale or units of the variables.

[0108] The features, structures, or characteristics of the Disclosure described throughout this Specification may be combined in any suitable way in one or more embodiments. For example, the use of “one embodiment,” “some embodiments,” “certain embodiments,” “certain embodiments,” or other similar terms means throughout this Specification that certain features, structures, or characteristics described in relation to embodiments may be included in at least one embodiment of the Disclosure. Thus, the appearance of the phrases “one embodiment,” “some embodiments,” “certain embodiments,” “certain embodiments,” or other similar terms throughout this Specification does not necessarily all refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable way in one or more embodiments.

[0109] Those skilled in the art will readily understand that the embodiments described above can be practiced using steps in a different order than disclosed, and / or using elements in configurations different from those disclosed. Therefore, while this disclosure considers the outlined embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative structures are evident, while remaining within the spirit and scope of this disclosure. Accordingly, reference to the appended claims should be made to determine the boundaries and scope of this disclosure.

Claims

1. A method for generating a machine learning (ML) model that includes a target variable, wherein the method is: This includes receiving training data, the training data including time-dependent data and a plurality of dates corresponding to the time-dependent data, The aforementioned method, The training data is divided by date using two or more of the aforementioned multiple dates, and multiple date-divided training data sets are generated. For each of the aforementioned multiple date-partitioned training data sets, the date-partitioned training data is divided into a training dataset and a corresponding test dataset using one or more different ratios, thereby generating multiple training / test splits. For each of the aforementioned training / test splits, the difference in distribution between the training dataset and the corresponding test dataset is determined. Selecting the training / test split having the smallest difference in distribution, A method further comprising training and testing the ML model using the selected training / test split.

2. Determining the difference in the distribution between training / test splits is, The method according to claim 1, further comprising creating a delayed percentile vector for each training / test split, for each percentile, which includes the results of the target variable for the corresponding multiple training datasets and the results of the target variable for the multiple test datasets.

3. The method according to claim 2, further comprising determining the pairwise difference and pairwise mean of the delayed percentile vectors for each training / test split.

4. The method according to claim 3, further comprising determining a difference score for each training / test split based on the pairwise difference and the pairwise mean.

5. The method according to claim 4, wherein the difference score includes SQRT (sum of squared differences).

6. The method according to claim 4, wherein the difference score includes the absolute value of (the pairwise difference for all percentiles / the sum of the pairwise means).

7. The method according to claim 4, further comprising selecting an optimized split date and an optimized training / test split from among the plurality of dates based on the difference score, and training the ML model using the optimized split date and the optimized training / test split.

8. The method according to claim 1, wherein the training data includes order transactions, and the target variable includes the amount of delay in payment for the corresponding order transactions.

9. A computer-readable medium storing instructions, wherein, when an instruction is executed by one or more processors, it causes the processors to generate a machine learning (ML) model including an objective variable, and the generation is This includes receiving training data, the training data including time-dependent data and a plurality of dates corresponding to the time-dependent data, The training data is divided by date using two or more of the aforementioned multiple dates, and multiple date-divided training data sets are generated. For each of the aforementioned multiple date-partitioned training data sets, the date-partitioned training data is divided into a training dataset and a corresponding test dataset using one or more different ratios, thereby generating multiple training / test splits. For each of the aforementioned training / test splits, the difference in distribution between the training dataset and the corresponding test dataset is determined. Selecting the training / test split having the smallest difference in distribution, A computer-readable medium comprising training and testing the ML model using the selected training / test split.

10. Determining the difference in the distribution between training / test splits is, The computer-readable medium according to claim 9, comprising, for each of the training / test splits, creating a delayed percentile vector for each percentile that includes the results of the target variable for the corresponding multiple training datasets and the results of the target variable for the multiple test datasets.

11. The computer-readable medium according to claim 10, wherein the generation further comprises determining the pairwise difference and pairwise mean of the delayed percentile vectors for each training / test split.

12. The computer-readable medium according to claim 11, further comprising generating a difference score for each training / test split based on the pairwise difference and the pairwise mean.

13. The computer-readable medium according to claim 12, wherein the difference score includes SQRT (sum of squared differences).

14. The computer-readable medium according to claim 12, wherein the difference score includes the absolute value of (the pairwise difference for all percentiles / the sum of the pairwise means).

15. The computer-readable medium according to claim 12, wherein generating further comprises selecting an optimized split date and an optimized training / test split from among the plurality of dates based on the difference score, and training the ML model using the optimized split date and the optimized training / test split.

16. The computer-readable medium according to claim 9, wherein the training data includes order transactions, and the target variable includes the amount of delay in payment for the corresponding order transactions.

17. A cloud-based machine learning (ML) model generation system, wherein the ML model includes an objective variable, and the system is The system comprises one or more processors that execute instructions, and the one or more processors It is configured to receive training data, and the training data includes time-dependent data and a plurality of dates corresponding to the time-dependent data. The training data is divided by date using two or more of the aforementioned multiple dates, and multiple date-divided training data sets are generated. For each of the aforementioned multiple date-partitioned training data sets, the date-partitioned training data is divided into a training dataset and a corresponding test dataset using one or more different ratios, thereby generating multiple training / test splits. For each of the aforementioned training / test splits, the difference in distribution between the training dataset and the corresponding test dataset is determined. Selecting the training / test split having the smallest difference in distribution, A cloud-based machine learning (ML) model generation system configured to train and test the ML model using the selected training / test split.

18. Determining the difference in the distribution between training / test splits is, The cloud-based machine learning (ML) model generation system according to claim 17, comprising creating a delayed percentile vector for each percentile for each of the training / test splits, which includes the results of the target variable for the corresponding multiple training datasets and the results of the target variable for the multiple test datasets.

19. The cloud-based machine learning (ML) model generation system according to claim 18, wherein the processor is further configured to determine the pairwise difference and pairwise mean of the delayed percentile vectors for each training / test split.

20. The cloud-based machine learning (ML) model generation system according to claim 19, wherein the processor is further configured to determine a difference score for each training / test split based on the pairwise difference and the pairwise mean.

Images