Interactive Machine Learning

Abstract

A computer-implemented method of interactive machine learning in which a user is given predicted results from a trained machine learning model, and the user can take the predicted results and either i) adjust the predicted results and input the adjusted results as new data, or ii) adjust the predicted data to retrain the model.

Description

[Technical field]

[0001] The present disclosure is directed generally to machine learning based predictions, and more particularly to systems and methods for user interaction with machine learning predictions. [Background technology]

[0002] Current attempts to apply machine learning to various industries rarely involve direct involvement by end users. Furthermore, the predictions of machine learning models are highly dependent on the quality of the data on which the machine learning models are trained. However, most users view such models as "black boxes" that users do not understand or intuitively trust. Furthermore, users do not have the opportunity to know how they can directly influence the predicted outcomes of the machine learning models. Summary of the Invention [Means for solving the problem]

[0003] Disclosed herein are systems and methods that allow user interaction to affect machine learning results by either modifying input data or by retraining the model. In other words, human feedback is used to retrain the model to improve the accuracy of predictions regarding supply chain information.

[0004] To enable users to build intuitive trust in machine learning models, the present disclosure provides methods and systems that allow users to influence the model itself based on user interactions. Such influence can be done in two ways: 1) by changing the information used to make predictions, and 2) by retraining the model itself.

[0005] In one aspect, a system includes a processor and a memory storing instructions that, when executed by the processor, configure the system to: preprocess data, by a machine learning module; train, by the machine learning module, a plurality of machine learning models with the preprocessed data; select, by the machine learning module, a trained machine learning model from the plurality of machine learning models; predict, by the machine learning module, an outcome based on the trained machine learning model; output, by the machine learning module, a prediction for a user to a user interface; correct the prediction by the user via the user interface to provide a corrected prediction; preprocess, by the machine learning module, data associated with the corrected prediction; and predict, by the machine learning module, a new outcome based on the corrected prediction and data associated with the trained machine learning model.

[0006] In some embodiments, the user interface may be a graphical user interface. In some embodiments, the results may be output to a device and a user may correct the prediction by moving one or more objects on the screen of the device.

[0007] In some embodiments, a user may correct a prediction by correcting a data file associated with the prediction.

[0008] In some embodiments, the machine learning model may be selected from the group consisting of nearest neighbor, Naive Bayes, decision trees, linear regression, Support Vector Machines (SVMs), neural networks, and any combination thereof.

[0009] In some examples, the instructions may further configure the system to, after predicting an outcome based on the trained machine learning model, optimize the outcome with respect to one or more features associated with the data, where correcting the prediction includes constraining a subset of the one or more features of the data.

[0010] In another aspect, a computer-implemented method of interactive machine learning, the method including: preprocessing data, by a machine learning module; training, by the machine learning module, a plurality of machine learning models with the preprocessed data; selecting, by the machine learning module, a trained machine learning model from the plurality of machine learning models; predicting, by the machine learning module, an outcome based on the trained machine learning model; outputting, by the machine learning module, the prediction for a user to a user interface; correcting, by the user via the user interface, the prediction to provide a corrected prediction; preprocessing, by the machine learning module, data associated with the corrected prediction; and predicting, by the machine learning module, a new outcome based on the corrected prediction and data associated with the trained machine learning model.

[0011] In some embodiments, the user interface may be a graphical user interface. In some embodiments, the results may be output to a device and a user may correct the prediction by moving one or more objects on the screen of the device.

[0012] In some embodiments, a user may correct a prediction by correcting a data file associated with the prediction.

[0013] In some embodiments, the machine learning model may be selected from the group consisting of nearest neighbor, naive Bayes, decision tree, linear regression, support vector machine (SVM), neural network, and any combination thereof.

[0014] In some examples, the method may further include, after predicting the outcome based on the trained machine learning model, optimizing, by the machine learning module, the outcome for one or more features associated with the data, where correcting the prediction includes constraining a subset of the one or more features of the data.

[0015] In another aspect, a non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: preprocess, by a machine learning module, data; train, by the machine learning module, a plurality of machine learning models with the preprocessed data; select, by the machine learning module, a trained machine learning model from the plurality of machine learning models; predict, by the machine learning module, an outcome based on the trained machine learning model; output, by the machine learning module, a prediction for a user to a user interface; correcting, by the user via the user interface, the prediction to provide a corrected prediction; preprocess, by the machine learning module, data associated with the corrected prediction; and predict, by the machine learning module, a new outcome based on the corrected prediction and the data associated with the trained machine learning model.

[0016] In some embodiments, the user interface may be a graphical user interface. In some embodiments, the results may be output to a device and a user may correct the prediction by moving one or more objects on the screen of the device.

[0017] In some embodiments, a user may correct a prediction by correcting a data file associated with the prediction.

[0018] In some embodiments, the machine learning model may be selected from the group consisting of nearest neighbor, naive Bayes, decision tree, linear regression, support vector machine (SVM), neural network, and any combination thereof.

[0019] In some examples, the instructions may further configure the computer to, after predicting an outcome based on the trained machine learning model, optimize the outcome with respect to one or more features associated with the data, where correcting the prediction includes constraining a subset of the one or more features of the data.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0021] Like reference numbers and designations in the various drawings indicate like elements.

[0022] To easily identify the description of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced. [Brief description of the drawings]

[0023] [Figure 1A] FIG. 1 illustrates an overview of one embodiment of interactive machine learning. [Figure 1B] FIG. 1 illustrates an overview of one embodiment of interactive machine learning. [Diagram 2] FIG. 1 illustrates a block diagram of an embodiment of interactive machine learning. [Diagram 3] FIG. 1 illustrates a flowchart according to one embodiment of interactive machine learning. [Figure 4]FIG. 1 illustrates a flowchart according to one embodiment of interactive machine learning. [Diagram 5] FIG. 1 illustrates a block diagram of an embodiment of interactive machine learning. [Figure 6] FIG. 1 illustrates a flowchart according to one embodiment of interactive machine learning. [Figure 7] FIG. 1 illustrates phase 1 of promotion optimization according to one embodiment of interactive machine learning. [Figure 8] FIG. 8 illustrates an optimized prediction 800 according to the embodiment shown in FIG. [Figure 9] FIG. 9 illustrates an interactive phase of a promotion according to the embodiment shown in FIG. 8. [Figure 10] FIG. 10 illustrates the final phase of promotion optimization according to the embodiment illustrated in FIG. [Figure 11] FIG. 1 illustrates a block diagram of an embodiment of interactive machine learning. [Figure 12] FIG. 13 is a diagram showing a cluster grouping diagram according to one embodiment of interactive machine learning. [Figure 13] FIG. 13 shows the cluster grouping diagram of FIG. 12 after moving one element. [Figure 14] This shows the cluster grouping diagram of Figure 13 after adding a new element. [Figure 15] FIG. 13 shows a series of diagrams of cluster groupings after adding, moving, and adding an element according to one embodiment of interactive machine learning. [Figure 16A] FIG. 1 illustrates a flowchart according to one embodiment of interactive machine learning. [Figure 16B] FIG. 1 illustrates a flowchart according to one embodiment of interactive machine learning. [Figure 17] FIG. 1 illustrates a system according to one embodiment of interactive machine learning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In this document, any embodiment or implementation of the subject matter described herein serves as an example, instance, or illustration and is not necessarily to be construed as preferred or advantageous over other embodiments.

[0025] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are described in detail below. It is not intended, however, to limit the disclosure to the particular forms disclosed, but to the contrary, the disclosure may be modified in any manner that satisfies the principles of the present disclosure. Example It should be understood that the invention is intended to cover all modifications, equivalents, and alternatives that fall within its scope.

[0026] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion such that a setup, device or method that includes a list of components or steps does not include only those components or steps, but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceding "comprises" does not, without further constraints, preclude the presence of other or additional elements in the system or apparatus.

[0027] In the following detailed description of embodiments of the present disclosure, reference is made to the accompanying drawings, which form a part of this application, and in which are shown by way of illustration specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, but it should be understood that other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. The following description is therefore not to be taken in a limiting sense.

[0028] Figure 1 A and Figure 1B 1 shows an overview 100 of one embodiment of interactive machine learning.

[0029] Figure 1 AIn the conventional process 110 , a simple overview of conventional machine learning is given a database 102 to train a machine learning model 104 , and the trained model provides a prediction 106 .

[0030] Figure 1 B In, an overview of interactive machine learning 112 shows the insertion of user interaction 108 between the transmission of the database 102 and the machine learning model 104. That is, the user directly intervenes to manipulate the data prior to its use by the machine learning model.

[0031] FIG. 2 illustrates a block diagram 200 according to one embodiment of interactive machine learning.

[0032] Raw data is first sent to a machine learning module in step 204. The data is pre-processed in step 206, after which the data is sent to train one or more machine learning models in step 208. Each trained model is evaluated and a trained model is selected in step 210 to make a prediction. The trained models are used to predict outcomes in step 212. The results are then presented to the user with the option to correct the predicted results in step 218. The corrected results are then used as new or updated data.

[0033] If the machine learning model provides a regression analysis, the corrected results are treated as preprocessed data in step 222. The preprocessed data is then used by the trained model selected in step 210 to make a prediction in step 212 in step 224. The user can look at the new prediction again and decide whether to keep the result (at which point the process ends in step 216) or continue in the loop 230. This loop can be performed as many times as necessary until the user is satisfied with the results, and the process ends in step 216.

[0034] If the machine learning model does not provide a regression analysis (e.g., a clustering analysis), the corrected results are treated as data to be preprocessed in step 206, and steps 208 to 212 are re-executed. If the user is satisfied with the new predictions, the process ends in step 216. Otherwise, loop 228 may be executed as many times as necessary until the user is satisfied with the results, and the process ends in step 216.

[0035] FIG. 3 illustrates a flow chart 300 according to one embodiment of interactive machine learning.

[0036] In Fig. 3, a user 304 requests 310 results from a server 306. Based on this initial request, the server 306 can calculate what is needed for the requested result. The server 306 sends a query or requests 312 data to a data store 308, which then provides a transmission 314 of the data to the server 306 hosting a machine learning module. This module pre-processes the data that is fed into a machine learning model that makes predictions and provides a transmission 316 of the results to the user 304, who can then modify the results via user corrections 302.

[0037] Pre-processing of the data may include transforming, validating, modifying the data, or any combination thereof.

[0038] Validating data simply means determining whether there are potential errors in the incoming data. For example, validation can include identifying missing data, null data, row count discrepancies, and data inconsistencies. In some embodiments, the data validation module may use machine learning algorithms along with z-score thresholds to identify anomalous data values.

[0039] Data correction involves modifying or recalculating data that indicates an error. For example, a missing or erroneous value may be replaced using one or more existing values, an arithmetic average of the existing data, or data interpolated from an average of the existing data. In some embodiments, data correction may use a predictive model to replace data that indicates an error.

[0040] A transmission 318 of the corrected results / data from the user 304 to the server 306 occurs. At this point, data preprocessing 326 occurs. For example, the corrected data is preprocessed (as in step 206 or step 222 in FIG. 2). If the machine learning model provides a regression type analysis, data preprocessing 326 includes reapplication of the trained model to make new or updated predictions (as in loop 230 of FIG. 2). If the machine learning model does not involve regression analysis (e.g., clustering analysis), retraining 328 may include a step in loop 228 of FIG. 2 to make new or updated predictions.

[0041] The updated prediction is provided to the user 304 via Send Updated Results 322. At this point, the user 304 can either accept the updated results or ask to correct the results via optional loop 324, which once again involves sending 318 the corrected results / data to the server 306 along with pre-processing of the data 326, for further computation of a new prediction and sending 322 the updated results back to the user 304. The optional loop 324 continues until the user 304 is satisfied with the results. The final corrected data set is sent to the data store 308 for storage via Send Corrected Data 320.

[0042] In some embodiments, user correction 302 may occur as follows: The prediction sent to user 304 may be in any format that includes predicted data (e.g., in a spreadsheet or as a data set in a JSON document). User 304 may purchase a prediction. In that case, user 304 determines that the prediction is somewhat incorrect for one group of items and corrects the prediction by correcting the file containing the predicted data set (e.g., by correcting the spreadsheet or JSON document with the corrected prediction). User 304 sends the corrected data file (e.g., via an HTTP request for the spreadsheet or JSON document) to server 306, which accepts it. Server 306 corrects the preprocessed data set with the corrected data set in data preprocessing 326.

[0043] FIG. 4 illustrates a flow chart 400 according to one embodiment of interactive machine learning.

[0044] In Figure 4, a user requests 310 results from a server 306. Based on this initial request, the server 306 can calculate what is needed for the requested result. The server 306 queries or requests 312 data from a data store 308, which then transmits 314 the data to the server 306 hosting a machine learning module. This module pre-processes the data, which is fed into a machine learning model, which makes predictions, and transmits 316 the results to the user 304, who can then modify the results via user corrections 302.

[0045] A transmission 318 of the corrected results / data from the user 304 to the server 306 occurs. At this point, data preprocessing 326 occurs. For example, the corrected data is preprocessed (as in step 206 or step 222 in FIG. 2). If the machine learning model includes a regression analysis, data preprocessing 326 includes reapplication of the trained model to make new or updated predictions (as in loop 230 of FIG. 2). If the machine learning model does not provide a regression analysis (e.g., clustering analysis), retraining 328 may include a step in loop 228 of FIG. 2 to make new or updated predictions.

[0046] The updated prediction is provided to the user 304 via Send Updated Results 322. At this point, the user 304 can either accept the updated results or ask to have the results corrected via optional loop 324, which once again involves sending corrected results / data 318 to the server 306 along with data pre-processing 326, for further computation of a new prediction and sending updated results 322 back to the user 304. The optional loop 324 continues until the user 304 is satisfied with the results. The final corrected data set is sent to the data store 308 for storage via Send Corrected Data 320.

[0047] 4, rather than directly correcting the file containing the predicted data, a user interface 402 is used by the user 304 to correct the data. The user interface 402 can be any type of interface that allows the user to obtain the predicted results, correct them, and send them back to the user 304. In some embodiments, the user interface 402 is a graphical user interface (GUI) where the results are analyzed by the GUI, thereby allowing the user 304 to correct / constrain / confirm the results, which are then sent back to the server 306, after which further pre-processing 326 of the data takes place before sending the new predictions.

[0048] FIG. 5 illustrates a block diagram 500 according to one embodiment of interactive machine learning.

[0049] Raw data is first sent to a machine learning module in step 504. The data is pre-processed in step 506, after which the data is sent to train one or more machine learning models in step 508. Each trained model is evaluated and one trained model is selected in step 510 to make a prediction. The trained models are used to predict outcomes in step 512. The results are then presented to the user with the option to correct the predicted results in step 518. The corrected results are then used as new or updated data.

[0050] The corrected results are treated as preprocessed data in step 520. The preprocessed data is then used by the trained model selected in step 510 to make a prediction in step 512 in step 522. The user can look at the new prediction again and decide whether to keep the result (at which point the process ends in step 516) or continue once more in loop 524, which can be performed as many times as necessary until the user is satisfied with the results, and the process ends in step 516.

[0051] Example: Machine Learning with User-Based Constraints Figure 6 shows a flow chart 600 according to one embodiment of interactive machine learning, in which interactive machine learning is used for optimization with user-based constraints on a machine learning model.

[0052] The first step is step 604, where a prediction of the output is made using the previously trained and selected machine learning model. The output can then be optimized (using an optimization algorithm) in step 606 for one or more subsets of the data. In step 608, the user introduces constraints in one or more subsets, resulting in one or more constraints being imposed on the original data set. This corrected result is treated as a new data set, which is then pre-processed in step 610 before being processed by the same trained model (in step 604) to make a new prediction in step 612, thereby obtaining a new machine learning model prediction in step 612. At this stage, the user may either approve the result and therefore terminate the program (in step 616), or decide to introduce further constraints and run the loop 618 again until a satisfactory result is obtained in step 616.

[0053] An embodiment using the flowchart 600 is shown in FIGS.

[0054] Example: Interactive machine learning for promotion optimization Promotion optimization is a crucial problem in many industries as well as in supply chain management and demand forecasting.

[0055] Machine learning can be used to optimize the dates of promotions to maximize product sales. In a single run, the algorithm performs multiple (e.g., hundreds, thousands, etc.) optimization iterations. In each iteration, the promotion dates are moved to different dates, a forecast is run, and the resulting sales are recorded. Eventually, a maximum sales value is reached and the promotion dates are moved back. The original and optimized results are then displayed to the user. The user has the option to move the promotions and to "anchor" them for future runs. Once a promotion is "anchored," it cannot be moved or optimized. This is to simulate constraints in the real world. For example, a company may have already pre-scheduled a promotion on a date and cannot move it. This example demonstrates interactive machine learning by allowing user feedback to influence future runs of the algorithm.

[0056] In such embodiments, the machine learning model is used along with a module for optimization and a user interface for user feedback. Examples include LightGBM (a forecast / prediction algorithm), DiscreteOnePlusOne (a common algorithm in Facebook's Nevergrad library for optimization), and Plotly Dash (for the user interface).

[0057] Experiment details (promotion optimization) In the first phase, it is possible to increase sales by optimizing promotions using a suitable optimization algorithm: by experimenting with as many algorithms as possible for a selected subset of products, an optimal model can be found.

[0058] First, we used a publicly available data set. Second, we built a machine learning model to give predictions based on the data. Then, we performed an exhaustive search of numerous optimization algorithms in Facebook's Nevergrad library. An optimal set of model hyperparameters as well as the best optimization algorithm were found. This boosted sales by a significant amount (between 5-15% for a single optimization).

[0059] First, we selected a forecast / prediction model called LightGBM to fit the dataset. We chose LightGBM based on past experience because it has worked extremely well to fit other data. We treated the promotion dates as features (inputs) for the model. The goal was to move the promotion to the date that would maximize the boost in sales (i.e., find the optimal promotion date).

[0060] We selected products in the dataset that had high transaction volume and good fit for the model. This is important because any optimization attempt can be inaccurate if the model does not fit well from the start. We then ran an exhaustive search of the best algorithms in Facebook AI's Nevergrad library. The Nevergrad library is an open source gradient-free optimization platform. That is, the total number of searches was (number of algorithms in Nevergrad) x (number of products selected), where each algorithm in the library was tested for each product and averaged. We selected the algorithm that performed best. In this example, DiscreteOnePlusOne was the model that gave the best results. After running one optimization on the model, we shuffled the promotions to new dates and sales were boosted by about 5-15% by the product.

[0061] The second phase was the anchoring of some promotions, i.e. these promotions were not optimized. This was to mimic the situation as if human decision-making was involved (a human gives some requirements for the promotion dates and the algorithm has to optimize keeping those requirements in mind). The anchoring was performed using Plotly Dash for the user interface.

[0062] In addition, we experimented with different types of promotions, including in-store displays, flyers, and discounts. Overall, our methodology required systematic research and analysis to test our hypotheses.

[0063] Figure 7 illustrates phase 1 of promotion optimization 700 according to one embodiment of interactive machine learning. In Figure 7, a machine learning model is used to predict sales of pretzels 714 with different promotions scheduled.

[0064] In FIG. 7, there are three types of promotions: a first type promotion 704 (i.e., display promotions), a second type promotion 706 (i.e., feature promotions), and a third type promotion 708 (i.e., discount promotions), with each promotion scheduled as shown. On July 13, 2017, there is one instance of the third type promotion 708. On July 20, 2011, there is one instance of the first type promotion 704 and one instance of the second type promotion 706. On July 27, 2011, there is one instance of the first type promotion 704 and one instance of the second type promotion 706. On November 2, 2011, there is one instance of the first type promotion 704 and one instance of the second type promotion 706. 2There is one instance of the second type of promotion, 706, on January 7, 2011. 2 On month 14, there is one instance of the second type promotion 706. In total, there are three instances of the first type promotion 704, five instances of the second type promotion 706, and one instance of the third type promotion 708. A forecast 712 predicts that 2474 units will be sold.

[0065] FIG. 8 illustrates an optimized forecast 800 according to the embodiment shown in FIG.

[0066] In Figure 8, promotions are moved to optimize the forecast. The optimized forecast sum 802 is shown against the forecasted sales 702 from Figure 7 (where promotions were not optimized).

[0067] As a result, three types of promotions have been moved such that the optimized forecast 806 predicts a jump in sales to 2778 units, a jump of 304 units or 12.3% from forecast 712.

[0068] Two of the three instances of the first type of promotion 704 have moved (from their original dates of July 27, 2011 and November 2, 2011) to June 8, 2011 and June 29, 2011. Only one instance of the first type of promotion 704 remains on July 20, 2011.

[0069] Four of the five instances of the second type promotion 706 have moved (from their original dates of July 20, 2011, July 27, 2011, November 2, 2011, and December 7, 2011) to November 9, 2011, November 23, 2011, December 21, 2011, and January 4, 2012. Only one instance of the second type promotion 706 remains on December 14, 2011. Meanwhile, the only instance of the third type promotion 708 has moved from its original date of July 13, 2011 to July 20, 2012. 2 It is moving on the 7th.

[0070] FIG. 9 illustrates an interactive phase of a promotion 900 according to the embodiment shown in FIG. 8. In FIG. 9, a user interacts with the machine learning model by pinning one or more of the promotion dates through a user interface. In this embodiment, the user interface is a graphical user interface, by which the user can move the promotions on the screen. In FIG. 9, the user has pinned two instances of the second type of promotion 706, namely pinned promotion 902 and pinned promotion 904, respectively indicated by black dots. That is, the user has chosen not to use the fully optimized result (shown in FIG. 8) and instead determined that there should be one promotion on September 7, 2011 and another promotion on September 14, 2011. The user, however, has not placed any constraints on the remaining promotions.

[0071] FIG. 10 illustrates the final phase of promotion optimization 1000 according to the embodiment illustrated in FIG.

[0072] In FIG. 10, with two fixed promotions (fixed promotion 902 and fixed promotion 904), the machine learning model is re-optimized for the remaining promotions. The result is a new interactive optimized forecast interactive optimized forecast sum 1002. The forecasted total units is 2734 units, which is a 10.5% increase compared to the original forecast 712 of 2474 units. The optimized forecast sum 802 (without fixed dates for the promotions) is 2778 units.

[0073] The new optimized promotion dates (excluding fixed promotions 902 and 904) are as follows: Promotions of the first type 704 are now scheduled for June 15, 2011, July 6, 2011, and July 20, 2011. Promotions of the second type 706 are now scheduled for October 5, 2011, November 2, 2011, and November 30, 2011, in addition to the two fixed dates of September 7, 2011, and September 14, 2011.

[0074] At this stage, the user can decide whether a slight drop in predicted sales volume (from 2778 units with no promotion constraints to 2734 units using two fixed promotion dates) is acceptable. The user can go back and consider other fixed dates (i.e., move before or after the fixed dates) to see how the machine learning model predicts sales volume based on the user's requirements.

[0075] Example: Machine Learning (Clustering) FIG. 11 shows a block diagram 1100 according to one embodiment of interactive machine learning.

[0076] Raw data is first sent to a machine learning module in step 1104. The data is pre-processed in step 1106, after which the data is sent to train one or more machine learning models in step 1108. Each trained model is evaluated and one trained model is selected to make a prediction in step 1110. The trained models are used to predict outcomes in step 1112.

[0077] The results are then presented to the user with the option to correct the predicted results in step 1116. The corrected results are then used as new or updated data to execute loop 1120 (comprised of steps 1106, 1108, 1110, and 1112). Loop 1120 can be executed as many times as necessary until the user is satisfied with the results, and the process ends in step 1118.

[0078] Alternatively, after pre-processing the data in step 1106, a loop 1122 may be used that simply re-trains the selected model rather than training and evaluating the machine learning model (steps 1108 and 1110). The re-trained model is used to predict new outcomes in step 1112.

[0079] Example: Interactive Machine Learning for New Product Introduction New product introduction is a huge problem in many industries as well as in supply chain management and demand forecasting. This problem is broadly applicable to any organization that deals with selling or purchasing products.

[0080] In some embodiments, existing items are first grouped based on semantic similarity using unsupervised machine learning techniques. When a new item is introduced, the new item is automatically added to the cluster to which it is most similar, or 2) if it is completely different from any of the existing items, the machine learning model creates a separate cluster from the new item. The problem with these divisions obtained from the clustering of items is that they may be far from the user's preferences. The present disclosure provides systems and methods that allow a user to move a desired item, which affects the position of all items added after it. To implement such a scenario, the model can be retrained after each user interaction to capture user feedback and allow the model to learn from the feedback.

[0081] In some embodiments, this may be accomplished by first converting the item descriptions / names into word vectors of multiple dimensions (in some embodiments, 300 dimensions). These items may be presented to the user on a user interface (in some embodiments, the user interface is a GUI) by projecting them onto a 2D plane. A neural network may be trained on the word vectors and their predicted 2D coordinates, which is then used to predict the location of the items shown on the user interface. Whenever changes are made to the predicted results shown on the user interface, the neural network is retrained with the feedback and then produces results that are acceptable to the user.

[0082] Experiment details (new product introduction) A search was conducted to determine which models or algorithms could both cluster existing items (given the system constraints) and allow for retraining the model with user input without changing the coordinates of existing items on the graph. This required several experiments with different types of techniques for dimensionality reduction, such as t-Distributed Stochastic Neighbor Embedding (t-SNE) and Principal Component Analysis (PCA), clustering algorithms like K-means, Fuzzy K-means and Denclue, correlation algorithms like Spearman's rank, Pearson correlation coefficient and Cramer's V statistic. Another uncertainty was whether moving one or more items would generate enough information to meaningfully retrain the model to produce the desired results.

[0083] We evaluated a number of approaches and hypotheses. First, we hypothesized that it would be possible to cluster existing items using an appropriate set of algorithms. We created and tested several algorithms to allow for meaningful retraining of the machine learning model, to allow the model to learn from user feedback.

[0084] To conduct the experiments, a publicly available data set was used. Secondly, a model was built for product similarity prediction. Then, another model was built for the projection of products from a higher dimension onto a two-dimensional screen to allow visualizing the clustered products, which is able to learn from user interactions such as moving products (changing coordinates) and adding new products. An optimal set of model hyperparameters was found, as well as the best algorithms for capturing and learning from user feedback. This allowed the creation of a model that can effectively learn from unlabeled user feedback to better classify products.

[0085] First, we chose Word2Vec to learn word similarities in English to create a product similarity matrix from the dataset. Then, we fitted a neural network called a Multi-Layer Perceptron (MLP) regressor to the word vectors and the 2D projection output from t-SNE. We treated the product descriptions / names as features (inputs) for the model. The goal is to correctly classify products by their semantic meaning and allow the user to override this prediction in a way that the model learns from.

[0086] We selected all the products from the dataset. We ran a pre-trained Word2Vec model on all the product descriptions to find the word vectors for every product description. We projected these word vectors onto a 2D plane using t-SNE. This prediction was learned using a neural network that was used to retrain whenever a product was moved to a different location or a new product was added by the user. Thus, the neural network learns from human interactions and makes better predictions over time.

[0087] FIG. 12 shows a diagram of cluster grouping 1200 according to one embodiment of interactive machine learning.

[0088] In Figure 12, a set of items (characterized by words or phrases) have been grouped by a machine learning program into approximately four clusters: Cluster 1 1202 corresponds to oral hygiene products, Cluster 2 1204 corresponds to pretzel related products, Cluster 3 1206 corresponds to pizza related products, and Cluster 4 1208 corresponds to cereal related products. The user notes that "Braided Honey Wheat Pretzels" (item 1210) has been grouped with Cluster 4 1208 by the machine learning program. The user believes this to be an incorrect placement.

[0089] Figure 1 3 illustrates the cluster grouping diagram 1200 of FIG. 12 after moving item 1210 ("Braided Honey Wheat Pretzels") from cluster 4 1208 (cereal-related products) to cluster 2 1204 (pretzel-related products). In doing so, the use retrained the machine learning model with a revised data set in which Braided Honey Wheat Pretzels were classified as part of the pretzel-related group and not as part of the cereal-related group.

[0090] FIG. 13 therefore illustrates interactive machine learning being used to retrain a machine learning model.

[0091] FIG. 14 shows a diagram of the cluster grouping 1200 of FIG. 13 after adding some new elements.

[0092] For example, a new term "electric toothbrush" (item 1404) is added by the machine learning program to cluster 1 1202 (oral hygiene products). A new term "enchilada" (item 1406) is added by the machine learning program to cluster 3 1206 (pizza products). However, two terms "lawn mower" (item 1408) and "grass trimmer" (item 1410) are placed in a new cluster 5 1402 by the machine learning model. The addition of new items in interactive machine learning does not actually have any impact on the user's machine learning process.

[0093] FIG. 15 shows a series of diagrams of cluster groupings after adding, moving, and adding an element according to one embodiment of interactive machine learning.

[0094] FIG. 15 provides a close-up of cluster 3 1206 (a pizza-related product) from FIG. 12, which undergoes a series of changes, which are described as follows.

[0095] In snapshot 1502, a new item 1512 ("Sourdough Bread") is input into the machine learning model, which places it in cluster 3 1206, as shown in snapshot 1504. The user notes that item 1514 ("Sourdough Nibblers") is closely related to item 1512 ("Sourdough Bread") and decides that these two items should form their own separate group; i.e., the user wishes to override the machine learning classification, as shown in snapshot 1504.

[0096] To do so, the user deletes the newly added item 1512 ("Sourdough Bread"), as shown in snapshot 1506. The user then moves item 1514 ("Sourdough Nibblers") from its original cluster 3 1206 (pizza-related products) to its own cluster 6 1518, as shown in snapshot 1508. In doing so, the user interaction retrains the machine learning model to create new cluster 6 1518.

[0097] The user then adds item 1512 ("Sourdough Bread"), as placed by the machine learning model at that time, to cluster 6 1518 (sourdough related products) along with item 1516 ("Sourdough Nibblers"), as shown in snapshot 1510.

[0098] Thus, Figure 15 shows interactive machine learning.

[0099] Figure 16 A and Figure 16B 16 shows a flowchart 1600 according to one embodiment of interactive machine learning.

[0100] figure 16 A and Figure 16B The illustrated embodiment shows a flowchart 1600 used in the clustering with user interaction example shown in FIGS.

[0101] In particular, block 1602 illustrates a case where a machine learning model is trained and provides cluster results, and then a user interacts with the results by moving one entity in one cluster to a different or completely new cluster. This process includes the following steps: Historical data 1606 undergoes pre-processing 1608 and is then used to train 1610 a machine learning model. The trained model makes predictions 1612 and provides cluster results 1614 to the user. This can, for example, describe the process used to obtain clusters of groups, as shown in FIG. 12.

[0102] At this point, the user may disagree with the clustering of items given by the machine learning model and may "move" (i.e., reclassify) one or more items to a different cluster or start a new cluster. When an item undergoes reclassification 1616, through interaction by the user, the trained model is retrained (arrow 1622) to reflect the new classification of the "moved" items.

[0103] Block 1602 illustrates, for example, the ongoing transition from the classification shown in FIG. 12 to the classification shown in FIG. 13. In both figures, the same number of items are classified. In FIG. 12, the results (i.e., results 1614) show that item 1210 ("braided honey wheat pretzels") is placed in cluster 4 1208, which groups together honey related items. However, the user is not satisfied with the placement of item 1210 and decides that item 1210 would be better placed in cluster 2 1204, which groups together pretzel related items. The user moves item 1210 ("braided honey wheat pretzels") to cluster 2 1204, as shown in FIG. 13. This reclassification of item 1210 retrains the machine learning model before the next run of the machine learning model. This is illustrated in block 1602 by arrow 1622 as reclassification 1616.

[0104] Block 1604 shows a case where a machine learning model is trained and gives a classification result. Then, one or more new items are added and their classification is based on the trained model. The new items can be added to a pre-existing cluster by the trained model or can become part of a newly defined cluster. Once new items are added and put into a cluster by the trained model, a user can interact with the results by moving an item in one cluster to a different or completely new cluster.

[0105] The process in block 1604 includes the following steps: Historical data 1606 undergoes preprocessing 1608 and is then used to train 1610 a machine learning model. The trained model makes a prediction 1612 and provides cluster results 1614 to the user. The user then adds a new item 1618, whose placement is given by the prediction 1612 shown in the results 1614. At this point, the user may disagree with the results 1614 and choose to move one item to a different cluster or to a new cluster entirely. At this point, reclassification 1620 occurs (arrow 1622) and the model is retrained.

[0106] For example, the situation shown in Figure 14 can be described by the steps of block 1604. After the initial clustering of groups (shown in Figure 12), there is then a movement (in Figure 13) of item 1210 to cluster 2 1204. This movement, or interaction, results in a reclassification of item 1210, which retrains the model. In Figure 14, a new item 1406 is added and placed in cluster 3 1206 by the once retrained model. Next, item 1404 is added and placed in cluster 1 1202 by the once retrained model. After that, a new item 1408 is added and placed in new cluster 5 1402 by the once retrained model. Finally, a new item 1410 is added and placed in new cluster 5 1402 by the once retrained model. The sequence of events does not involve any movement (or reclassification) of items by the user. The situation depicted in FIG. 14 is a simple prediction of the classification of a new item based on a trained model: adding a new item 1618, followed by a prediction 1612 of the new item 1618, followed by a sequence of results 1614.

[0107] Returning to block 1604, once the classification of the new item is predicted (prediction 1612), the user may disagree with the classification outcome after placement of the new item. Similar to block 1602, the user may “move” (i.e., reclassify) one or more items to a different cluster or start a new cluster. In some embodiments, the user may choose to move the new item to a different cluster, a cluster that may already exist, or to create an entirely new cluster grouping. In some embodiments, the user may choose to move (or reclassify) one previously classified item (i.e., not a new item) while maintaining the classification of the newly classified item. When an item undergoes reclassification 1620, due to interaction by the user, the trained model is retrained (arrow 1622) to reflect the new classification of the “moved” item.

[0108] Block 1604, for example, illustrates the transition shown in FIG. 15. In snapshot 1502, a new item 1512 ("Sourdough Bread") is input into the machine learning model, which places the new item 1512 ("Sourdough Bread") into cluster 3 1206 as shown in snapshot 1504. This is a straightforward prediction (prediction 1612) by the trained model after a new item 1618 is input. The user notes that item 1514 ("Sourdough Nibblers") is closely related to item 1512 ("Sourdough Bread") and decides that these two items should form their own separate group. That is, the user wishes to override the machine learning classification as shown in snapshot 1504.

[0109] To do so, the user deletes the newly added item 1512 ("Sourdough Bread"), as shown in snapshot 1506. The user then moves item 1514 ("Sourdough Nibblers") from its original cluster 3 1206 (pizza-related products) to its own cluster 6 1518, as shown in snapshot 1508. In so doing, the user interaction retrains the machine learning model to create new cluster 6 1518. That is, the user reclassifies item 1514 (corresponding to reclassification 1620 in block 1604), thus leading to a retraining of the model (arrow 1622).

[0110] The user then inserts item 1512 ("Sourdough Bread"), which is then placed by the retrained machine learning model into cluster 6 1518 (sourdough related products) together with item 1516 ("Sourdough Nibblers"), as shown in snapshot 1510. This result 1614, which was deemed satisfactory by the user, remains as is (i.e., the arrangement is not subject to reclassification 1620).

[0111] FIG. 17 illustrates a system 1700 according to one embodiment of interactive machine learning.

[0112] The system 1700 includes a system server 1704, a machine learning storage 1712, a client data source 1722, and one or more devices 1706, 1710, and 1708. The system server 1704 can include a memory 1716, a disk 1718, a processor 1714, and a network interface 1720. Although one processor 1714 is shown, the system server 1704 can include one or more processors. In some embodiments, the memory 1716 can be volatile memory, as compared to the disk 1718, which can be non-volatile memory. In some embodiments, the system server 1704 can communicate with the machine learning storage 1712, the client data source 1722, and one or more external devices 1710, 1706, and 1708 via the network 1702. Although the machine learning storage 1712 is shown as separate from the system server 1704, the machine learning storage 1712 may also be integrated into the system server 1704 either as a separate component within the system server 1704 or as part of at least one of the memory 1716 and the disk 1718.

[0113] System 1700 may also include additional features and / or functionality. For example, system 1700 may also include additional storage (removable and / or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in FIG. 17 by memory 1716 and disk 1718. Storage media may include volatile and nonvolatile media, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Memory 1716 and disk 1718 are examples of non-transitory computer-readable storage media. Non-transitory computer readable media also include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory and / or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs), and / or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, and / or any other media that may be used to store desired information and that may be accessed by system 1700. Any such non-transitory computer readable storage media may be part of system 1700.

[0114] Communications between the system server 1704, the machine learning storage 1712, and the one or more external devices 1710, 1706, and 1708 via the network 1702 may occur via a variety of network types. In some embodiments, the processor 1714 may be disposed in communication with the network 1702 via a network interface 1720. The network interface 1720 may communicate with the network 1702. The network interface 1720 may employ connection protocols including, but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 40 / 400 Base T), transmission control protocol / internet protocol (TCP), Token Ring, IEEE 802.11a / b / g / n / x, and the like. Non-limiting exemplary network types may include Fibre Channel, small computer system interface (SCSI), Bluetooth, Ethernet, Wi-fi, Infrared Data Association (IrDA), Local Area Networks (LAN), Wireless Local Area Networks (WLAN), Internet, serial, and wide area networks (WAN), such as universal serial bus (USB). In general, communication between various components of system 1700 may occur via hardwired, cellular, Wi-Fi or Bluetooth networked components, and the like. In some embodiments, one or more electronic devices of system 1700 may include cloud-based capabilities, such as cloud-based memory storage.

[0115] The machine learning storage 1712 may implement an "in-memory" database where volatile (e.g., non-disk-based) storage (e.g., random access memory) is used both for cache memory and to store the entire database during operation, and persistent storage (e.g., one or more fixed disks) is used for offline persistence and for maintaining database snapshots. Alternatively, persistent storage may store the entire database, but volatile storage may be used as cache memory to store recently used data.

[0116] The machine learning storage 1712 may store metadata about the structure, relationships, and meaning of the data. This information may include data defining the schema of the database tables in which the data is stored. The database table schema may specify the names of the database tables, the columns of the database tables, the data types associated with each column, and other information associated with the database tables. The machine learning storage 1712 may also, or alternatively, support multi-tenancy by providing multiple logical database systems programmatically separated from one another. Moreover, the data may be indexed and / or selectively replicated in the index to enable fast searching and retrieval of the data. Additionally, the machine learning storage 1712 may store a number of machine learning models accessed by the system server 1704. Several ML models may be used.

[0117] In some implementations where machine learning is used, gradient boosted trees, ensembles of trees, and support vector regression may be used. In some implementations of machine learning, one or more clustering algorithms may be used. Non-limiting examples include hierarchical clustering, k-means, mixture models, density-based spatial clustering of applications with noise, and ordering point to identify clustering structures.

[0118] In some implementations of machine learning, one or more anomaly detection algorithms may be used, non-limiting examples of which include local outlier factor methods.

[0119] In some embodiments of machine learning, neural networks may be used.

[0120] The client data sources 1722 may provide various raw data from the user including, but not limited to, point of sales data showing sales records of all the client's products at all locations, inventory history of all the client's products at all locations, promotional campaign details for all the products at all locations, and important / relevant events for the sales of the client's products at all locations.

[0121] Using the network interface 1720 and the network 1702, the system server 1704 may communicate with one or more devices 1710, 1706, and 1708. These devices 1710, 1706, and 1708 may include, but are not limited to, one or more personal computers, one or more servers, cellular phones, smartphones (e.g., Apple iPhone®, BlackBerry, Android®-based phones, etc.), tablet computers, e-book readers (e.g., Amazon Kindle, Nook, etc.), laptop computers, notebooks, gaming consoles (e.g., Microsoft Xbox, Nintendo DS, Sony PlayStation, etc.), and various mobile devices.

[0122] Using the network 1702, the system server 1704 can retrieve data from the machine learning storage 1712 and the client data sources 1722. The retrieved data can be stored in memory 1716 or 1708. In some embodiments, the system server 1704 can also include a web server and format resources into a format suitable for display on a web browser.

[0123] Once the provisional machine learning results are provided to any of the one or more devices, a user may correct the results, which are resent to machine learning storage 1712 for further execution. The results may then be corrected by interaction with one or more data files sent to 1714, or through a user interface at one or more of devices 1710, 1706, and 1708. For example, at 1720, a user may correct the results using a graphical user interface.

[0124] The algorithms described above include those algorithms described separately with respect to the flow charts above, but it should be understood that any two or more algorithms disclosed herein may be combined in any combination. Any of the methods, modules, algorithms, implementations, or procedures described herein may include machine-readable instructions for execution by (a) a processor, (b) a controller, and / or (c) any other suitable processing device. Those skilled in the art will readily appreciate that any algorithm, software, or method disclosed herein may be implemented in software stored on a non-transitory tangible medium, such as, for example, a flash memory, a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), or other memory device, although the algorithm, in its entirety and / or portions thereof, may alternatively be executed by devices other than a controller and / or implemented in firmware or dedicated hardware in a well-known manner (e.g., it may be implemented by an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable logic device (FPLD), discrete logic, etc.). Furthermore, while certain algorithms have been described with reference to flow charts shown herein, those skilled in the art will readily appreciate that many other ways of implementing the example machine-readable instructions may alternatively be used. For example, the order of execution of the blocks may be changed and / or some of the described blocks may be changed, eliminated, or combined.

[0125] It should be noted that the algorithms shown and described herein as having various modules perform specific functions and interact with each other. It should be understood that these modules are only separated based on their functionality for the purposes of explanation and represent computer hardware and / or executable software code stored on a computer readable medium for execution on suitable computing hardware. The various functions of the different modules and units may be combined or separated in any manner as described above as modules, as hardware, and / or as software stored on a non-transitory computer readable medium, and may be used separately or in combination.

[0126] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A processor; a display in communication with the processor; A memory for storing instructions; wherein the instructions, when executed by the processor, The machine learning module preprocesses the data and training, by the machine learning module, a plurality of machine learning models using the pre-processed data; selecting, by the machine learning module, a trained machine learning model from the plurality of machine learning models; predicting, by the machine learning module, an outcome based on the selected trained machine learning model; optimizing, by the processor, the results with respect to one or more features associated with the pre-processed data; outputting the optimized result by the display; A loop, constraining a subset of the one or more features of the data into a new data set in response to input received via a graphical user interface; pre-processing the new data set by the machine learning module; predicting new outcomes based on the pre-processed new data set with the selected trained machine learning model; outputting, by said display, on said graphical user interface, said optimized result and said new result simultaneously; receiving user input indicating whether the new result is accepted or rejected; Start a loop that contains Using the selected trained machine learning model, the system re-runs the loop until the new result is accepted.

2. The system described in claim 1, wherein the input received via the graphical user interface to constrain the subset of the one or more features includes movement of one or more objects on the display.

3. The system of claim 1, wherein the input received to constrain the subset of the one or more features includes a selection of the subset in a data file associated with the optimized result.

4. 2. The system of claim 1, wherein the machine learning model is selected from the group consisting of nearest neighbor, naive Bayes, decision tree, linear regression, support vector machine (SVM), neural network, and any combination thereof.

5. The system of claim 1, wherein the one or more features include a promotion type and a promotion date, and the result is a sales amount.

6. 1. A computer-implemented method for interactive machine learning, the method comprising: pre-processing the data by a machine learning module; training, by the machine learning module, a plurality of machine learning models using the pre-processed data; selecting, by the machine learning module, a trained machine learning model from the plurality of machine learning models; predicting, by the machine learning module, an outcome based on the selected trained machine learning model; optimizing, by a processor, the results with respect to one or more features associated with the pre-processed data; A loop, constraining a subset of the one or more features of the data into a new data set in response to input received via a graphical user interface; pre-processing the new data set with the machine learning model; predicting new outcomes based on the pre-processed new data set with the selected trained machine learning model; outputting, by said display, on said graphical user interface, said optimized result and said new result simultaneously; receiving user input indicating whether the new result is accepted or rejected; starting a loop including: using the selected trained machine learning model to re-run the loop until the new result is confirmed; A method comprising:

7. The computer-implemented method of claim 6, wherein the input received via the graphical user interface to constrain the subset of the one or more features includes movement of one or more objects on the display.

8. The computer-implemented method of claim 6, wherein the input received to constrain the subset of the one or more features includes a selection of the subset in a data file associated with the optimized result.

9. 7. The computer-implemented method of claim 6, wherein the machine learning model is selected from the group consisting of nearest neighbor, naive Bayes, decision tree, linear regression, support vector machine (SVM), neural network, and any combination thereof.

10. The computer-implemented method of claim 6, wherein the one or more features include a promotion type and a promotion date, and the result is a sales amount.

11. A non-transitory computer-readable storage medium, the computer-readable storage medium, when executed by a computer, causing the computer to: The machine learning module preprocesses the data, training a plurality of machine learning models using the pre-processed data by the machine learning module; selecting, by the machine learning module, a trained machine learning model from the plurality of machine learning models; causing the machine learning module to predict an outcome based on the selected trained machine learning model; optimizing, by a processor, the results with respect to one or more features associated with the pre-processed data; outputting the optimized result by a display; A loop, constraining a subset of the one or more features of the data into a new data set in response to input received via a graphical user interface; pre-processing the new data set by the machine learning module; predicting new outcomes based on the pre-processed new data set with the selected trained machine learning model; outputting, by said display, on said graphical user interface, said optimized result and said new result simultaneously; receiving user input indicating whether the new result is accepted or rejected; Start a loop that contains Using the selected trained machine learning model, re-run the loop until the new result is confirmed. A non-transitory computer-readable storage medium containing instructions.

12. The computer-readable storage medium of claim 11, wherein the input received via the graphical user interface to constrain the subset of the one or more features includes movement of one or more objects on the display.

13. The computer-readable storage medium of claim 11, wherein the input received to constrain the subset of the one or more features includes a selection of the subset in a data file associated with the optimized result.

14. 12. The computer-readable storage medium of claim 11, wherein the machine learning model is selected from the group consisting of nearest neighbor, naive Bayes, decision tree, linear regression, support vector machine (SVM), neural network, and any combination thereof.

15. The computer-readable storage medium of claim 11, wherein the one or more features include a promotion type and a promotion date, and the result is a sales amount.

Images