Dynamic data selection for machine learning models

Abstract

Embodiments relate to dynamic data selection for machine learning models. A plurality of data predictions generated by a trained machine learning model may be accessed, wherein the data predictions include corresponding observation data. The accuracy of the machine learning model may be calculated based on the accessed plurality of data predictions and the corresponding observation data. A variable number of data predictions may be used for iterative access and calculation, wherein the variable number of data predictions may be adjusted based on actions taken during previous iterations, and when the calculated accuracy does not meet an accuracy criterion during a given iteration, training of the machine learning model may be triggered.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 458,924, filed on July 1, 2019, which claims priority to Indian Patent Application No. 201941003803, filed on January 30, 2019, the disclosures of which are incorporated herein by reference. Technical Field

[0003] Embodiments of the present disclosure generally relate to dynamic data selection for machine learning models. Background Art

[0004] The implementation of artificial intelligence (AI) has yielded tangible benefits, ranging from supervised learning-based data prediction to facial recognition. However, challenges surrounding AI remain, such as implementation-specific issues that often vary across different systems. Training AI models can be resource-intensive and sometimes disruptive. Furthermore, retraining AI models can impact model accuracy and system efficiency. Therefore, tools that balance resource usage with the performance of AI models can yield tangible benefits for their users. Summary of the Invention

[0005] Embodiments of the present disclosure generally relate to systems and methods for dynamic data selection for machine learning models, which significantly improve the related art.

[0006] A plurality of data predictions generated by a trained machine learning model may be accessed, wherein the data predictions include corresponding observation data. An accuracy of the machine learning model may be computed based on the accessed plurality of data predictions and the corresponding observation data. Access and computation may be iteratively performed using a variable number of data predictions, wherein the variable number of data predictions is adjusted based on actions taken during a previous iteration, and when the computed accuracy does not meet an accuracy criterion during a given iteration, retraining of the machine learning model may be triggered.

[0007] Features and advantages of the embodiments are set forth in the following description, or will be obvious from the description, or may be learned through practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Additional embodiments, details, advantages and modifications will become apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.

[0009] Figure 1 A system for dynamic data selection for machine learning models is shown according to an example embodiment.

[0010] Figure 2A block diagram of a computing device operatively coupled to a system is shown according to an example embodiment.

[0011] Figures 3A-3C A flow diagram of dynamic data selection for triggering training of a machine learning model is shown according to an example embodiment.

[0012] Figure 4 A flow diagram of dynamic data selection for a machine learning model is shown according to an example embodiment.

[0013] Figure 5 An integrated supplier, inventory, and logistics system according to an example embodiment is shown that includes improved planning and supply actions as disclosed herein. DETAILED DESCRIPTION

[0014] Embodiments perform dynamic data selection for machine learning models. In some embodiments, training data can be used to train a machine learning model (e.g., via supervised learning). The trained model can then be used to generate data predictions based on the input data. Sometimes, a trained machine learning model can be updated, for example, by retraining the model or updating the training.

[0015] In some embodiments, an accuracy metric can be used to trigger retraining or updating training. For example, some implementations of a trained machine learning model generate data predictions for data points observed at some future time. Thus, past data predictions can include corresponding observed data points. An accuracy metric based on these two values ​​can be used to trigger retraining or updating training. For example, the accuracy metric can be compared to an accuracy standard, and based on this comparison, training can be triggered.

[0016] In some embodiments, the number of past data predictions and corresponding observation data point pairs can affect the accuracy metric. For example, as observation data becomes available, a set of these data point pairs can be generated. The accuracy determination can depend on the number of these data point pairs selected. A large number of data point pairs may dilute the importance of any particular subset of these data points. For example, when using a month's worth of data point pairs to determine accuracy, the last week's worth of data point pairs would be diluted by the rest of the month. In another example, when using two weeks' worth of data point pairs to determine accuracy, the last week's worth of data point pairs would have a significant impact on the accuracy metric.

[0017] In some implementations, where the first number of data point pairs is less than the second number of data point pairs, the accuracy metric based on the first number is more likely to trigger retraining or updated training than the accuracy metric based on the second number. In other words, in some implementations, the smaller number of data point pairs is more likely to generate an accuracy metric that does not meet the accuracy criterion, thereby triggering retraining or updated training.

[0018] Training of machine learning models can be resource intensive, so selectively retraining or updating the training of machine learning models can balance accuracy with resource efficiency and practicality. Embodiments iteratively compute an accuracy metric using a variable number of data point pairs, and retraining or updating training can be triggered at a given iteration when the accuracy metric does not meet an accuracy criterion.

[0019] For example, at a given iteration, the number of data point pairs can be based on the number of previous data point pairs (e.g., used in the previous iteration), factor values ​​(e.g., dynamic values ​​that adjust with iteration), and the state of the last retraining or update training (e.g., whether the model was retrained or update training occurred in the previous iteration). Thus, as the number of data point pairs used to evaluate accuracy changes, the likelihood of triggering retraining or update training changes. This configured likelihood improves the implementation overall by balancing prediction accuracy with resource usage and system interruption.

[0020] For example, some traditional implementations offer the option of a fixed number of data point pairs for calculating accuracy. However, in these cases, the calculated accuracy is universal across implicit feedback evaluation cycles or iterations. As described in the embodiments, dynamic selection of points provides flexible and selective accuracy calculation, which leads to enhanced machine learning implementations. These improvements in accuracy calculation can generate enhanced training decisions and provide more efficient resource utilization.

[0021] Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. In the detailed description that follows, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments. Whenever possible, identical reference numerals will be used for identical elements.

[0022] Figure 1 A system for dynamic data selection for a machine learning model according to an example embodiment is shown. System 100 includes a machine learning model 102, training data 104, input data 106, predictions 108, and observation data 110. In some embodiments, machine learning model 102 can be a design model that includes one or more machine learning elements (e.g., a neural network, a support vector machine, a Bayesian network, etc.). Training data 104 can be any data set capable of training machine learning model 102 (e.g., a set of features with corresponding labels, such as labeled data for supervised learning). In some embodiments, training data 104 can be used to train machine learning model 102 to generate a trained machine learning model.

[0023] In some embodiments, input data 106 (e.g., an input feature set) can be input to a trained machine learning model to generate predictions 108. In some implementations, predictions 108 can represent observable data points. Thus, at a point in time, a given prediction within predictions 108 can include an observed data point corresponding to the given prediction. In other words, a given observed data point within observations 110 can correspond to the data that a given prediction within predictions 108 is attempting to predict.

[0024] For example, a trained machine learning model can be configured to predict the fuel level in a vehicle over time. Thus, the prediction generated by the trained model can include the fuel level at a future point in time. In the future, at a point in time corresponding to the prediction, the vehicle's actual fuel level can be observed (e.g., retrieved and stored). The predicted fuel level at that point in time can be compared to the observed fuel level at that point in time to obtain an accuracy calculation.

[0025] In embodiments, the predictions 108 and observations 110 may include pairs of data points representing past data predictions and corresponding observation data points. Embodiments perform dynamic selection of these pairs of data points when calculating the accuracy of the machine learning model 102. For example, a dynamic number of these points may be taken in various iterations to evaluate the accuracy of the machine learning model 102, and if the accuracy does not meet a criterion, retraining or update training of the machine learning model 102 may be triggered. For example, the observations 110 may be used to update the training data 104 as part of the retraining or update training. In embodiments, once retraining or update training of the machine learning model 102 occurs, the accuracy of the newly trained model may be iteratively evaluated.

[0026] Figure 2 is a block diagram of a computer server / system 200 according to an embodiment. All or part of the system 200 may be used to implement Figure 1 Any component shown in Figure 2 As shown in FIG, system 200 may include a bus device 212 and / or other communication mechanism(s) configured to transmit information between various components of system 200 (e.g., processor 222 and memory 214). Additionally, communication device 220 may support connectivity between processor 222 and other devices by encoding data to be sent from processor 222 to another device over a network (not shown) and decoding data received from another system for processor 222 over a network.

[0027] For example, the communication device 220 may include a network interface card configured to provide wireless network communications. Various wireless communication technologies may be used, including infrared, radio, Bluetooth Wi-Fi and / or cellular communications. Alternatively, the communication device 220 may be configured to provide wired network connection(s), such as an Ethernet connection.

[0028] Processor 222 may include one or more general or special purpose processors to perform computational and control functions of system 200. Processor 222 may include a single integrated circuit, such as a microprocessing device, or may include multiple integrated circuit devices and / or circuit boards that work in concert to implement the functions of processor 222. In addition, processor 222 may execute computer programs stored in memory 214, such as operating system 215, prediction module 216, and other applications 218.

[0029] The system 200 may include a memory 214 for storing information and instructions executed by the processor 222. The memory 214 may contain various components for retrieving, presenting, modifying, and storing data. For example, the memory 214 may store software modules that provide functionality when executed by the processor 222. The modules may include an operating system 215 that provides operating system functionality for the system 200. These modules may include an operating system 215, an accuracy module 216 that implements dynamic data selection for machine learning models and all other functionality disclosed herein, and other application modules 218. The operating system 215 provides operating system functionality for the system 200. In some cases, the accuracy module 216 may be implemented as an in-memory configuration. In some implementations, when the system 200 performs the functionality of the accuracy module 216, it implements a non-conventional, special-purpose computer system that performs the functionality disclosed herein.

[0030] The non-volatile memory 214 may include various computer-readable media accessible by the processor 222. For example, the memory 214 may include any combination of random access memory ("RAM"), dynamic RAM ("DRAM"), static RAM ("SRAM"), read-only memory ("ROM"), flash memory, cache memory, and / or any other type of non-transitory computer-readable media. The processor 222 is also coupled to a display 224, such as a liquid crystal display ("LCD"), via the bus 212. A keyboard 226 and a cursor control device 228 (e.g., a computer mouse) are also coupled to the communication device 212 to enable a user to interact with the system 200.

[0031] In some embodiments, the system 200 may be part of a larger system. Thus, the system 200 may include one or more additional function modules 218 to include additional functionality. For example, the other application modules 218 may include Cloud infrastructure, Cloud platform, Various modules of cloud applications. The accuracy module 216, other application modules 218, and any other suitable components of the system 200 may include the following various modules: analyze, Data Science Cloud, Customer Experience Cloud, Enterprise Resource Planning Cloud, Manufacturing cloud, other adaptive intelligence Application, any suitable A data science toolkit for a product or service, or any other suitable product or service.

[0032] The database 217 is coupled to the bus 212 to provide centralized storage for the modules 216 and 218 and to store data received, for example, from the planning module 216 or other data sources. The database 217 can store data in an integrated collection of logically related records or files. The database 217 can be an operational database, an analytical database, a data warehouse, a distributed database, an end-user database, an external database, a navigational database, an in-memory database, a document-oriented database, a real-time database, a relational database, an object-oriented database, a non-relational database, a NoSQL database, Distributed File System ("HFDS") or any other database known in the art.

[0033] Although shown as a single system, the functionality of system 200 may be implemented as a distributed system. For example, memory 214 and processor 222 may be distributed across multiple different computers that collectively represent system 200. In one embodiment, system 200 may be part of a device (e.g., a smartphone, tablet, computer, etc.). In embodiments, system 200 may be separate from the device and the disclosed functionality may be provided remotely to the device. Additionally, one or more components of system 200 may not be included. For example, for functionality as a user or consumer device, system 200 may be a smartphone or other wireless device that includes a processor, memory, and display, but does not include a processor, memory, and display. Figure 2 One or more of the other components shown in , and including Figure 2 Additional components not shown in the diagram may include antennas, transceivers, or any other suitable wireless device components. Furthermore, when implemented to perform the functions disclosed herein, system 200 is a dedicated computer specifically designed to provide demand forecasts.

[0034] Embodiments relate to the dynamic selection of multiple data point pairs for computing the accuracy of a trained machine learning model. For example, an implicit feedback evaluator (e.g., such as A module of a data science toolkit system such as the Data Science Toolkit system can periodically evaluate model performance and, based on accuracy, can trigger training if model performance does not meet a criterion (e.g., has an accuracy metric that does not meet a criterion). In some embodiments, the algorithm uses an initial number of data point pairs and then increases / decreases that number based on the action taken in the previous iteration (e.g., based on whether the previous iteration triggered retraining).

[0035] For example, sometimes data science toolkits (e.g., from IoT analytics) is used to automatically take corrective actions when the trained model performance does not meet standards (e.g., has accuracy that does not meet standards). Some data science toolkit providers use accuracy calculation mechanisms to measure model performance. However, this accuracy is based on the number of data point pairs used to calculate the accuracy. An embodiment of an implicit feedback evaluator (e.g., a module in the Data Science Toolbox system) periodically evaluates the trained model performance and automatically takes appropriate actions (e.g., retraining / updating the machine learning model, etc.) whenever the performance exceeds certain tolerance levels. Data point selection can play a role in this evaluation.

[0036] The selection of fixed data points provides a static opportunity for retraining or updating training in the implicit feedback evaluation iteration, since the accuracy calculation depends on the number of data points considered. Based on the dynamic selection of data points, embodiments include some iterations with reduced retraining opportunities, while other iterations may have increased retraining opportunities. Prediction training can be a highly resource-intensive operation, so various embodiments help optimize the use of computing / cluster resources.

[0037] To illustrate how the selection of multiple data points affects the accuracy calculation, consider the following scenario. A car rental company plans to use an Internet of Things (“IoT”) asset monitoring application (e.g., An IoT asset monitoring application) can be used to manage its cars. The car can be equipped with a sensor (e.g., an IoT device) that can provide a pulse (e.g., a measurement) of the car's current at specified intervals.

[0038] In an embodiment, a car rental company, as a user of an asset management application, can create an asset type as a car with certain equipment models. For example, consider an equipment model created for a fuel tank with the following attributes: fuel level (fuelLevel), temperature, etc. In an embodiment, the company may want to track the fuel level of the car every hour and create a metric / key performance indicator ("KPI"): "Tracking AVG(fuelLevel) every hour."

[0039] In some embodiments, the company may also want to predict future fuel level readings and may create a prediction by “predicting AVG(fuelLevel) for the next day.” For example, a machine learning model may be trained to predict AVG(fuelLevel) for the next day. In some embodiments, after training / deploying the machine learning model for seven days, an accuracy check may be used to determine the accuracy of the trained machine learning model. For example, the accuracy calculation may be 1-mean relative error, where the mean relative error is:

[0040]

[0041] Here, "xi" can be the actual value of the quantity being predicted, "yi" can be the prediction, and "n" can be the number of different times the variable is predicted. Any other suitable accuracy measure can be used. Consider that there are five data point pairs that can be used for accuracy evaluation, but three are selected for accuracy evaluation:

[0042]

[0043] 3 sampling points -

[0044]

[0045] In an embodiment, using the above accuracy formula, the three data points will have errors abs(10-15) / 10=0.5, abs(15-10) / 15=0.33, and abs(20-24) / 20=0.2, summed and divided by "n" accordingly to obtain (0.5+0.33+0.2) / 3=0.34, and 1-absolute relative error reaches 1-0.34=0.66.

[0046] Consider whether to sample four data points instead of three to calculate the algorithm accuracy.

[0047]

[0048] In an embodiment, using the above average relative error formula, the four data points will have errors abs(10-15) / 10=0.5, abs(15-10) / 15=0.33, abs(20-24) / 20=0.2, and abs(20-30) / 20=0.5, summed and divided by "n" accordingly to give (0.5+0.33+0.2+0.5) / 4=0.38, and 1-absolute relative error to give 1-0.38=0.62. In some embodiments, rather than being paired, the data points may include a larger number of values ​​for error calculation.

[0049] This scenario demonstrates how the accuracy calculation depends on the choice of the number of data points. Choosing this number dynamically (for example, from time to time) reduces the risk of frequent retraining and the risk of retraining every time a drastic change occurs, among other benefits.

[0050] Figures 3A-3C A flow chart of dynamic data selection for triggering training of a machine learning model according to an example embodiment is shown. In some embodiments, Figures 3A-3C and the following Figure 4 The functions of the processor are implemented by software stored in a memory or other computer-readable or tangible medium and executed by a processor. In other embodiments, each function may be performed by hardware (e.g., by using an application-specific integrated circuit (ASIC), a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.) or any combination of hardware and software. In an embodiment, Figures 3A-3C and Figure 4 The function can be Figure 2 Executed by one or more elements of system 200.

[0051] In some embodiments, a machine learning model can be designed, trained, and stored. Figure 3A At 302A, you can train machine learning models. Return to reference Figure 1 , the training data 104 can be used to train the machine learning model 102.

[0052] The design of the machine learning model 102 can include any suitable machine learning model components (e.g., neural networks, support vector machines, specialized regression models, etc.). For example, a neural network can be implemented with a given cost function (e.g., for training / gradient calculations). The neural network can include any number of hidden layers (e.g., 0, 1, 2, 3, or more) and can include feedforward neural networks, recursive neural networks, convolutional neural networks, modular neural networks, and any other suitable types. In some embodiments, the neural network can be configured for deep learning, for example, based on the number of hidden layers implemented. In some examples, a Bayesian network or other type of supervised learning model can be similarly implemented.

[0053] For example, in some cases, a support vector machine can be implemented with one or more kernels (e.g., a Gaussian kernel, a linear kernel, etc.). In some embodiments, the machine learning model 102 can be a stack of multiple models, such as where the output of a first model is fed into the input of a second model. Some implementations can include multiple layers of predictive models. In some embodiments, features of the machine learning model 102 can also be determined. For example, feature engineering can be used to generate a set of features implemented by one or more machine learning models.

[0054] In some embodiments, the design of the machine learning model 102 can be tuned during training, retraining, and / or update training. For example, tuning can include adjusting the number of hidden layers in a neural network, adjusting the kernel computations used to implement a support vector machine, etc. Such tuning can also include adjusting / selecting features used by the machine learning model. Embodiments include implementing various tuning configurations (e.g., different versions of the machine learning model and features) while training in order to arrive at a configuration of the machine learning model 102 that achieves the desired performance when trained (e.g., performs predictions at a desired level of accuracy, runs according to a desired resource utilization / time metric, etc.).

[0055] In some embodiments, retraining and updating the training of a machine learning model may include training the model with updated training data. For example, the training data may be updated to incorporate observational data, or data that has been labeled in other ways (e.g., for supervised learning).

[0056] At 304A, the trained model and corresponding metadata can be stored. For example, in some implementations, the machine learning model 102 can be trained, and the resulting trained machine learning model can be stored along with metadata about the model. For example, the stored metadata can be the number of points used for accuracy calculation (e.g., a default value, or a number generated by performing iterations), a factor value (e.g., a default value such as 2, or a value calculated after performing iterations), a model accuracy (modelAccuracy), such as the accuracy calculated by the disclosed embodiments, and the like.

[0057] In some embodiments, a feedback loop or evaluation determination for a trained machine learning model can be performed after a period (e.g., an iteration). For example, the period can be a predetermined amount, such as a period of time (e.g., an hour, a day, a week, two weeks, a month, a quarter, etc.) or a predetermined amount of observation data (e.g., 30 accessible data point pairs, 50 points, hundreds of points, thousands of points, etc. for evaluation). The period for performing algorithm iterations can depend on the specific implementation, the rate at which data points are observed, the rate at which accuracy degrades, and other factors.

[0058] Figure 3B The first iteration of the evaluation loop is shown. At 302B, a factor value can be determined. For example, a default factor value (e.g., 2, 3, or any other suitable number) can be determined. In some implementations, the default factor value can depend on implementation details, and any suitable default factor value can be implemented.

[0059] At 304B, the number of data points can be selected. For example, an initial selection of a default number can be determined from the available data point pairs (e.g., 50% of the data points available from the trained model creation date). The default number can be any other suitable percentage or number, and in some embodiments can also depend on the implementation. In an embodiment, the default number of data points can be determined using the following formula: number of data points = 0.5 (current time - trained model creation time) / minimum KPI frequency (0.5 (current time - trained model creation time) / minimum KPI frequency). For example, if the key performance indicator ("KPI") is hourly and the period is 7 days, the number of data points can be 0.5 (7 days / 1 hour) = 0.5 168 = 84.

[0060] At 306B, a determination may be made as to whether the number of data points meets a criterion, such as a threshold number of data points. For example, the threshold number of data points may be 30, and in some cases, 30 data points may not have been generated based on data observed since the model was trained (e.g., when the period is defined in terms of time, the number of data points generated may be variable). Any suitable criterion or threshold number of data points may be used. At 308B, when the number of data points does not meet the criterion (e.g., is less than or equal to a minimum number of data points), retraining or updating training is not triggered during the iteration.

[0061] At 310B, when the number of data points does meet the criteria (eg, is greater than or equal to a minimum number of data points), an accuracy metric may be calculated. Return to Reference Figure 1 , the accuracy metric can be based on a comparison between a predicted data point from prediction 108 and a corresponding observed data point from observation 110. For example, consider that a trained machine learning model can generate data predictions and then a corresponding data point can be observed, thereby generating data point pairs that can be used for accuracy calculations. Example accuracy metrics can include mean relative error, 1-mean relative error, mean absolute percentage error, mean absolute error, root mean square deviation, and the like.

[0062] In some embodiments, the trained model can be a logical model, and the accuracy can be based on more than one element. Accuracy metrics such as F1 scores can be implemented. For example, F1 can be a function of the accuracy (e.g., positive predictive value) and recall (e.g., sensitivity) of the model. Accuracy can be the ratio of the true positives of the model (e.g., predicted positives by the model corresponding to the labeled positives) to the total positives of the model (e.g., the true positives + false positives of the model). Recall can be the ratio of true positives to the total labeled positives (e.g., the true positives + false negatives of the model). Any other suitable definition of accuracy and recall can be implemented similarly. The F1 score function can be F1=2×(accuracy×recall) / (precision+recall). Any other suitable definition of F1 score can be implemented. Some embodiments implement other suitable logical model accuracy metrics.

[0063] At 312B, a determination may be made as to whether the calculated accuracy metric satisfies an accuracy criterion. For example, a determination may be made as to whether the accuracy calculated for the trained machine learning model is greater than a threshold accuracy (e.g., 25%, 50%, 75%, etc.). Some embodiments may calculate the accuracy metric as an error metric, and thus, the error metric may be compared to an error criterion to determine whether the error is greater than the threshold.

[0064] At 308B, when the accuracy metric meets the accuracy standard (e.g., the accuracy metric is greater than the accuracy threshold or the error metric is less than the error threshold), retraining or updating training is not triggered in the iteration. At 312B, when the accuracy metric does not meet the accuracy standard (e.g., the accuracy metric is less than the accuracy threshold or the error metric is greater than the error threshold), retraining or updating training is triggered. For example, the training data 104 can be updated with the observation data 110, and the machine learning model 102 can be retrained or update training can be performed. In some embodiments, if retraining is triggered, the machine learning model 102 is trained and stored. In some implementations, the factor values ​​and the number of points can also be saved with the trained model information (e.g., the trained model and metadata). In some embodiments, implicit feedback will also be recorded in the feedback persistent storage device.

[0065] Figure 3C The "n"th iteration of the evaluation cycle is shown. At 302C, a factor value can be determined. For example, the factor value can be based on the factor value of the "n-1"th iteration. In some embodiments, the "n-1"th factor value can be multiplied by a static value (e.g., 0.95, 0.90, 0.85, etc.) to determine the factor value for the "n"th iteration. Any other suitable factor value determination can be used.

[0066] At 304C, the number of data points may be selected. For example, the number of data points determined may be based on the factor value of the "n"th iteration and the action taken at the "n-1"th iteration (e.g., retraining or not retraining). In some embodiments, when retraining is triggered at the "n-1"th iteration, the number of data points in the "n-1"th iteration may be multiplied by the factor value, and when retraining is not triggered at the "n-1"th iteration, the number of data points in the "n-1"th iteration may be divided by the factor value. In other words, the number of points may be increased when there was training in the previous iteration, and the number of points may be decreased when there was no training in the previous iteration.

[0067] Thus, for the current iteration, the number of points used in the previous iteration can be increased or decreased based on the action taken last time. In some embodiments, when retraining was performed in the previous iteration, the number of data points can be increased by multiplying it by a factor value, and the number of data points can be decreased by dividing it by a factor value.

[0068] In some embodiments, the functionality of 306C, 308C, 310C, 312C, and 314C can be similar to the functionality of 306B, 308B, 310B, 312B, and 314B. For example, at 306C, it can be determined whether the number of data points meets a criterion, such as a threshold number of data points. At 308C, when the number of data points does not meet the criterion (e.g., is less than or equal to a minimum number of data points), retraining or updating training is not triggered during the iteration.

[0069] At 310C, when the number of data points does meet the criteria (e.g., greater than or equal to the minimum number of data points), an accuracy metric may be calculated. At 312C, a determination may be made as to whether the calculated accuracy metric meets the accuracy criteria. At 308C, when the accuracy metric meets the accuracy criteria (e.g., the accuracy metric is greater than an accuracy threshold or the error metric is less than an error threshold), retraining or updating training is not triggered in the iteration. At 312C, when the accuracy metric does not meet the accuracy criteria (e.g., the accuracy metric is less than an accuracy threshold or the error metric is greater than an error threshold), retraining or updating training is triggered.

[0070] In some embodiments, the factor value can be initialized to a default value (e.g., such as a value of 2, as shown in 302B of FIG. 3 b ) and can be decreased by a fixed value during the feedback iteration (e.g., as shown in 302C). In some embodiments, the dynamic selection of points can end when the factor value reaches an end value (e.g., 1 or any other suitable value). When the factor value reaches the end value, the number of data points in the previous iteration can be used in each upcoming evaluation cycle / feedback iteration.

[0071] In some embodiments, a variable number of data points and feedback iterations can be used to arrive at a number of data points suitable for evaluating the accuracy of a particular model / machine learning implementation (e.g., when a factor value reaches 1). Determining the number of data points after performing an iteration is advantageous over selecting a static number of data points without such an iteration cycle, at least because the iterations adjust the number of data points determined to be appropriate for the particular model / implementation. For example, the number of data points determined by the iterations can be different for different models / implementations because the specifics of each model can be suitable for different end values.

[0072] Thus, embodiments adjust the number of data points used to evaluate model accuracy to suit a particular model / implementation, thereby improving performance. This improved performance can include better resource utilization, for example due to improved retraining decisions. Retraining of a model can be triggered based on the estimated accuracy of the model, so improving the number of data points for accuracy assessment also improves retraining decisions based on that accuracy assessment.

[0073] The following pseudo-code represents an example implementation of an embodiment:

[0074]

[0075]

[0076]

[0077] Return Reference Figure 1 , the machine learning model 102 can be designed and trained to perform any type of appropriate data prediction. For example, the generated data predictions can relate to values ​​measured by Internet of Things ("IoT") connected devices, such as physical measurements (e.g., speed, acceleration, position, physical attributes of physical things, quantity, or any other suitable physical measurement), measurements in a computing environment or network, and the like. In some examples, the generated data predictions can relate to values ​​for an enterprise organization, such as retail sales, inventory, returns, any of these elements within various sub-channels (e.g., different locations, markets, delivery / return channels, such as direct to consumer, in-store, third party, etc.), or any other suitable data for the enterprise organization.

[0078] In some embodiments, shipments can be performed based on machine predictions. For example, the machine predictions can be demand estimates, inventory forecasts, etc., and based on the estimates / forecasts, shipments can be performed between warehouses and stores.

[0079] Figure 4A flow chart for implementing a machine learning prediction model with dynamic data point selection according to an example embodiment is shown. At 402, a plurality of data predictions generated by a trained machine learning model may be accessed, where the data predictions correspond to observed data. For example, a data prediction may be generated for a data point observed at a later point in time. The accessed plurality of data predictions may have corresponding observed data such that they generate data point pairs.

[0080] At 404, the accuracy of the machine learning model can be calculated based on the accessed plurality of data predictions and corresponding observation data. For example, the accuracy can be an accuracy metric based on a comparison between the data predictions and their corresponding observation data.

[0081] At 406, a variable number of data predictions may be used to iteratively access and compute, wherein the variable number of data predictions is adjusted based on actions taken during previous iterations, and when the computed accuracy does not meet an accuracy criterion during a given iteration, training of the machine learning model is triggered. For example, multiple iterations may be performed over a period of time depending on the iteration period.

[0082] In some embodiments, a variable number of data predictions for a current iteration may be adjusted based on whether training was triggered during a previous iteration. In some embodiments, the previous iteration is the iteration immediately preceding the current iteration.

[0083] In some embodiments, when training was triggered during a previous iteration, the number of data predictions for the current iteration is increased, and when training was not triggered during the previous iteration, the number of data predictions is decreased. For example, when training was triggered during the previous iteration, the number of data predictions for the current iteration is multiplied by a factor, and when training was not triggered during the previous iteration, the number of data predictions is divided by the factor. In some embodiments, the value of the factor decreases over multiple iterations. In an example, the value of the factor is initialized and has a value greater than or equal to 1.

[0084] In some embodiments, the triggered training can be a retraining or update training of a trained machine learning model. In some embodiments, when training is triggered, the next iteration of access and calculation uses the data predictions generated by the machine learning model generated by the triggered training.

[0085] In some embodiments, iterations are performed according to a predetermined period. For example, the predetermined period may be a predetermined time period or a data forecast with a predetermined amount of corresponding observation data.

[0086] At 408, access and computation using a variable number of data predictions may be terminated after performing multiple iterations such that the number of configured data predictions is determined based on the iterations, wherein after the iterations, the accuracy of the machine learning model is calculated using the configured number of data predictions, and the accuracy is used to trigger training of the machine learning model.

[0087] Embodiments may improve upon traditional approaches to forecasting solutions by providing businesses (e.g., retailers) with a mechanism for using improved machine learning models for forecast accuracy in demand forecasting. In many cases, demand forecasts are the driving force of the supply chain, and if demand forecasts are inaccurate, allocation, planning, and replenishment will perform poorly, resulting in inefficient use of resources. Generally, if the forecast is high, items will remain unsold; they will either need to be sold at a discount or waste will increase. If the forecast is low, demand will not be met, and the retailer will experience lost sales and low customer satisfaction. Both of these situations result in inefficient use of resources. Embodiments provide techniques for achieving reliable demand forecasts, thereby improving resource utilization efficiency in the supply chain and across the enterprise.

[0088] Figure 5 An integrated supplier, inventory, and logistics system including the inventory management disclosed herein is shown according to an example embodiment. Figure 5 As shown in FIG, system 500 may include an enterprise business system 570 that executes code to manage product inventory at enterprise locations 501-504 using a warehouse 580 and to ship products directly to consumers from the warehouse 580. The enterprise business system 570 communicates with one or more inventory systems 520 via a cloud network 550 or other type of communication network. In some embodiments, the planning and forecasting software of the enterprise business system 570 may generate forecasts that provide forecast inventory levels for various products. The inventory system 520 and the warehouse 580 may execute shipments to and from the enterprise locations 501-504 based on these forecast inventory levels. In some embodiments, the use of a trained machine learning model may provide improvements to these forecast inventory levels, resulting in a more efficient shipping process.

[0089] Inventory system 520 stores inventory and provides transportation logistics to deliver items to business locations 501-504 and consumer locations (e.g., consumer homes) using trucks 510-513 or some other transportation mechanism. In one embodiment, inventory system 520 implements an enterprise resource planning (ERP) dedicated computer system or a dedicated inventory control system that uses input from enterprise business system 510 (e.g., seasonal plans generated by planning and forecasting software) to determine inventory levels and the amount and timing of product deliveries to business locations 501-504.

[0090] Warehouse 580 may be a fulfillment warehouse that supplies one or more products to business locations 501-504 based on inventory system 520 and ships the products to consumer locations (e.g., consumers' homes). In one embodiment, warehouse 580 implements an ERP-specific computer system or a dedicated supplier system that uses input from enterprise business system 510 (e.g., a seasonal plan generated by planning and forecasting software) to determine the quantity and timing of product shipments to inventory system 520 and / or business locations 501-504. In some embodiments, warehouse 580 may receive shipments from business locations 501-504, for example, based on returns to business locations 501-504, to ensure that the business locations are not oversupplied and have sufficient product variety.

[0091] Embodiments perform dynamic data selection for machine learning models. In some embodiments, training data can be used to train a machine learning model (e.g., through supervised learning). The trained model can then be used to generate data predictions based on the input data. Sometimes, a trained machine learning model can be updated, for example, by retraining the model or updating the training.

[0092] In some embodiments, an accuracy metric can be used to trigger retraining or updating training. For example, some implementations of a trained machine learning model generate data predictions for data points observed at a certain time in the future. Thus, past data predictions can include the corresponding observed data points. An accuracy metric based on these two values ​​can be used to trigger retraining or updating training. For example, the accuracy metric can be compared to an accuracy standard, and based on this comparison, training can be triggered.

[0093] In some embodiments, the number of past data predictions and corresponding observed data point pairs can influence the accuracy metric. For example, as observed data becomes available, a set of these data point pairs can be generated. The accuracy determination can depend on the number of these data point pairs selected. A large number of data point pairs may dilute the importance of any particular subset of these data points. For example, when using a month's worth of data point pairs to determine accuracy, the last week's worth of data point pairs may be diluted by the rest of the month. In another example, when using two weeks' worth of data point pairs to determine accuracy, the last week's worth of data point pairs may be significant to the accuracy metric.

[0094] In some implementations where the first number of data point pairs is less than the second number of data point pairs, the accuracy metric based on the first number is more likely to trigger retraining or updated training than the accuracy metric based on the second number. In other words, in some implementations, a smaller number of data point pairs is more likely to generate an accuracy metric that does not meet the accuracy criterion, thereby triggering retraining or updated training.

[0095] Training of machine learning models can be resource intensive, so selectively retraining or updating the training of machine learning models can balance accuracy with resource efficiency and practicality. Embodiments use a variable number of data point pairs to iteratively compute an accuracy metric, and when the accuracy metric does not meet the accuracy criteria, retraining or updating training can be triggered at a given iteration.

[0096] For example, at a given iteration, the number of data point pairs can be based on the number of previous data point pairs (e.g., used in the previous iteration), factor values ​​(e.g., dynamic values ​​adjusted by iteration), and the state of the last retraining or update training (e.g., whether retraining or update training occurred for the model in the previous iteration). Thus, as the number of data point pairs used to evaluate accuracy changes, the likelihood of triggering retraining or update training changes. This configured likelihood improves the implementation overall by balancing prediction accuracy with resource usage and system interruption.

[0097] In one or more embodiments, the features, structures, or characteristics of the present disclosure described throughout this specification may be combined in any suitable manner. For example, the phrases "one embodiment," "some embodiments," "an embodiment," "certain embodiments," or other similar language used in this specification refer to the fact that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the present disclosure. Thus, throughout this specification, appearances of the phrases "one embodiment," "some embodiments," "an embodiment," "certain embodiments," or other similar language are not necessarily all referring to the same set of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0098] Those skilled in the art will readily appreciate that the embodiments discussed above may be implemented with steps in a different order and / or with elements in configurations different from those disclosed. Thus, while the present disclosure contemplates the embodiments outlined, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will be apparent while remaining within the spirit and scope of the present disclosure. Therefore, reference should be made to the appended claims to determine the metes and bounds of the present disclosure.

Claims

1. A method for implementing a machine learning prediction model through dynamic data selection, the method comprising: Access to a variable number of data point pairs, each data point pair consisting of past data predictions generated by a trained machine learning model deployed at deployment time and the corresponding observations; calculating the accuracy of the machine learning model based on the variable number of data point pairs; The accessing and computing are iterated by adjusting the variable number of data point pairs, wherein, adjusting the variable number of data point pairs based on whether the machine learning model was retrained during a previous iteration, the variable number of data point pairs being adjusted between successive iterations by an incremental value calculated by applying a factor value; The factor value changes as the number of iterations increases relative to the deployment time, and the change in the delta value with iterations is based at least in part on the change in the factor value as the number of iterations increases; and When the calculated accuracy does not meet the accuracy criterion during a given iteration, triggering training for the machine learning model; wherein when training was triggered during an iteration prior to a first iteration, the variable number of data point pairs for the first iteration is increased, and when training was not triggered during an iteration prior to the first iteration, the variable number of data point pairs is decreased; and After performing a number of iterations such that a configured number of data point pairs is determined based on the iterations, terminating the accessing and computing using the variable number of data point pairs, wherein The termination occurs after the factor value reaches a standard value, and After the iteration, a prediction accuracy of the machine learning model is calculated using the configured number of data point pairs, and the prediction accuracy is used to trigger training of the machine learning model.

2. The method according to claim 1, wherein The iteration preceding the first iteration includes the iteration immediately preceding the first iteration.

3. The method according to claim 1, wherein The triggered training includes retraining or update training for the trained machine learning model.

4. The method according to claim 1, wherein When training is triggered, subsequent iterations of the access and computation use data predictions generated by the machine learning model generated by the triggered training.

5. The method according to claim 1, wherein The iterations are performed according to a predetermined period.

6. The method according to claim 5, wherein: The predetermined period is a predetermined time period or a data forecast having a predetermined amount of corresponding observation data.

7. The method of claim 1, wherein: The variable number of data point pairs for the first iteration is multiplied by a factor when training was triggered during a previous iteration, and the variable number of data point pairs is divided by the factor when training was not triggered during a previous iteration.

8. The method of claim 7, wherein: The factor comprises a value greater than or equal to one.

9. The method of claim 8, wherein: The value of the factor decreases with each iteration.

10. A system for implementing a machine learning prediction model through dynamic data selection, the system comprising: processor; as well as a memory storing instructions executed by the processor, the instructions configuring the processor to: Access to a variable number of data point pairs, each data point pair consisting of past data predictions generated by a trained machine learning model deployed at deployment time and the corresponding observations; calculating an accuracy of the machine learning model based on the variable number of data point pairs; The accessing and computing are iterated by adjusting the variable number of data point pairs, wherein, adjusting the variable number of data point pairs based on whether the machine learning model was retrained during a previous iteration, the variable number of data point pairs being adjusted between successive iterations by an incremental value calculated by applying a factor value; The factor value changes as the number of iterations increases relative to the deployment time, and the change in the delta value with iterations is based at least in part on the change in the factor value as the number of iterations increases; wherein when training was triggered during an iteration prior to a first iteration, the variable number of data point pairs for the first iteration is increased, and when training was not triggered during an iteration prior to the first iteration, the variable number of data point pairs is decreased; and triggering training of the machine learning model when the calculated accuracy does not meet an accuracy criterion during a given iteration; and After performing a number of iterations such that a configured number of data point pairs is determined based on the iterations, terminating the accessing and computing using the variable number of data point pairs, wherein The termination occurs after the factor value reaches a standard value, and After the iteration, a prediction accuracy of the machine learning model is calculated using the configured number of data point pairs, and the prediction accuracy is used to trigger training of the machine learning model.

11. The system of claim 10, wherein: The iteration preceding the first iteration includes the iteration immediately preceding the first iteration.

12. The system of claim 11, wherein: The iterations are performed according to a predetermined period, and the predetermined period is a predetermined time period or a data forecast with a predetermined amount of corresponding observation data.

13. The system of claim 10, wherein: The variable number of data point pairs for the first iteration is multiplied by a factor when training was triggered during an iteration prior to the first iteration, and the variable number of data point pairs is divided by the factor when training was not triggered during an iteration prior to the first iteration.

14. The system of claim 13, wherein: The factor comprises a value greater than or equal to 1, and the value of the factor decreases with a plurality of iterations.

15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to implement a machine learning predictive model through dynamic data selection, wherein: The instructions, when executed, cause the processor to: Access to a variable number of data point pairs, each data point pair consisting of past data predictions generated by a trained machine learning model deployed at deployment time and the corresponding observations; calculating an accuracy of the machine learning model based on the variable number of data point pairs; The accessing and computing are iterated by adjusting the variable number of data point pairs, wherein, adjusting the variable number of data point pairs based on whether the machine learning model was retrained during a previous iteration, the variable number of data point pairs being adjusted between successive iterations by an incremental value calculated by applying a factor value; The factor value changes as the number of iterations increases relative to the deployment time, and the change in the delta value with iterations is based at least in part on the change in the factor value as the number of iterations increases; wherein when training was triggered during an iteration prior to a first iteration, the variable number of data point pairs for the first iteration is increased, and when training was not triggered during an iteration prior to the first iteration, the variable number of data point pairs is decreased; and triggering training of the machine learning model when the calculated accuracy does not meet an accuracy criterion during a given iteration; and After performing a number of iterations such that a configured number of data point pairs is determined based on the iterations, terminating the accessing and computing using the variable number of data point pairs, wherein The termination occurs after the factor value reaches a standard value, and After the iteration, a prediction accuracy of the machine learning model is calculated using the configured number of data point pairs, and the prediction accuracy is used to trigger training of the machine learning model.

Images