Systems and methods for machine learning interpretability

Abstract

A method and system for providing machine learning interpretability. The SHAP values ​​of the historical data and the predicted data, along with features of both, are used to provide a measure of the influence of the training data point on the prediction. After removal of an individual training data point from the training data set, the resulting prediction is compared to the prediction obtained with the full training data set, also providing a measure of the influence of the individual training data point on the prediction.

Description

[Technical field]

[0001] Machine learning provides powerful predictive tools, but users are often left wondering how the training data (used to train the machine learning model) relates to the predictions provided by the trained model. [Background technology]

[0002] This phenomenon is often called "black box" machine learning models. One way to provide users with an interpretation of machine learning prediction results based on tabular data is through charts. There are also some interpretability methods specific to image or text data. However, there are no methods that are applicable to time series forecasting. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure addresses the problem of visually illustrating example-based machine learning interpretability explanations of time series forecasts from black box machine learning models. A method and system are disclosed that associates a similarity measure between selected forecast points in a forecast and the training data used to train the model, shown by a visualization suitable for interpreting the time series data. The method solves the above-mentioned problem, since it makes it clear from a plot of the time series data that one or more points of the training data explain the predicted value of the selected forecast. The method can include using SHapley Additive exPlanations (SHAP), a unified method for explaining the output of machine learning models. SHAP may be used by the model to calculate feature importance per instance. These feature importances and feature values ​​are used as a vector to calculate the similarity between the training data and the forecast. The method can not only show how the model weights the importance of features for explaining a particular instance, but can also explain why based on related examples from the past. [Means for solving the problem]

[0004] In one aspect, the method includes training, by a processor, a regression machine learning model with training data; predicting, by the processor, a prediction based on the trained model; receiving, by a machine learning interpretability module, the training data, the trained model, and the prediction; and comparing, by the machine learning interpretability module, properties of the training data and the prediction.

[0005] In some embodiments of the method, the comparing step includes visualizing the training data, the predictions, and the characteristics of the training data and the predictions.

[0006] In some embodiments of the method, the step of comparing the characteristics includes: Heuristic Functions determining a value of Heuristic Functions incorporates the SHAP value of each training data point, the SHAP value of the predicted data point, the feature values ​​of the training data points, and the feature values ​​of the predicted data points. Heuristic Functions may comprise a combination of SHAP distance and feature distance, where the SHAP distance is the Euclidean distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted data points, the feature distance is the Euclidean distance between the feature vectors of the training data points and the feature vectors of the predicted data points, the SHAP vector is an ordered sequence of the SHAP values ​​of the data points, and the feature vector is an ordered sequence of the feature values ​​of the data points.

[0007] In some embodiments of the method, comparing the characteristics includes determining, by the machine learning interpretability module, a SHAP value for one or more points of the prediction, determining, by the machine learning interpretability module, a SHAP value for one or more points of the training data, and determining, by the machine learning interpretability module, for each of the one or more points of the prediction, a difference between the SHAP value of the prediction point and the SHAP value of each of the one or more points of the training data. The difference can be a Euclidean distance between the SHAP vector of the prediction point and the SHAP vector of each of the one or more points of the training data.

[0008] In some embodiments of the method, comparing the characteristics includes removing, by the machine learning interpretability module, training data points from the training data to form a corrected training data set; retraining, by the machine learning interpretability module, the trained model on the corrected training data set; predicting, by the machine learning interpretability module, based on the corrected training data set and providing a corrected prediction; comparing, by the machine learning interpretability module, a difference between the prediction and the corrected prediction; and assigning, by the machine learning interpretability module, a measure of influence to the removed training data points based on the difference.

[0009] In another aspect, a system includes a processor and a memory storing instructions that, when executed by the processor, configure the system to: train, by the processor, a regression machine learning model with training data; predict, by the processor, a prediction based on the trained model; receive, by a machine learning interpretability module, the training data, the trained model, and the prediction; and compare, by the machine learning interpretability module, properties of the training data and the prediction.

[0010] In some embodiments, the system is further configured to provide visualizations of the training data, the predictions, and properties of the training data and the predictions.

[0011] In some embodiments, the system performs a machine learning interpretability module to interpret the Heuristic Functions and the prediction comprises a plurality of predicted data points; Heuristic Functions incorporates the SHAP value of each training data point, the SHAP value of the predicted data point, the feature values ​​of the training data points, and the feature values ​​of the predicted data points. 11. Heuristic Functions may comprise a combination of SHAP distance and feature distance, where the SHAP distance is the Euclidean distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted data points, the feature distance is the Euclidean distance between the feature vectors of the training data points and the feature vectors of the predicted data points, the SHAP vector is an ordered sequence of the SHAP values ​​of the data points, and the feature vector is an ordered sequence of the feature values ​​of the data points.

[0012] In some embodiments, the system is further configured to determine, by the machine learning interpretability module, a SHAP value for one or more points of the prediction, determine by the machine learning interpretability module a SHAP value for one or more points of the training data, and determine, by the machine learning interpretability module, for each of the one or more points of the prediction, a difference between the SHAP value of the prediction point and the SHAP value of each of the one or more points of the training data. The difference can be a Euclidean distance between the SHAP vector of the prediction point and the SHAP vector of each of the one or more points of the training data.

[0013] In some embodiments, the system is further configured to remove, by the machine learning interpretability module, training data points from the training data to form a corrected training data set, by the machine learning interpretability module, retraining the trained model on the corrected training data set, by the machine learning interpretability module, predicting based on the corrected training data set and providing the corrected prediction, by the machine learning interpretability module, comparing a difference between the prediction and the corrected prediction, and assigning, by the machine learning interpretability module, a measure of influence to the removed training data points based on the difference.

[0014] In yet another aspect, a non-transitory computer readable storage medium includes instructions that, when executed by a computer, cause the computer to: train, by a processor, a regression machine learning model with training data; predict, by the processor, a prediction based on the trained model; receive, by a machine learning interpretability module, the training data, the trained model, and the prediction; and compare, by the machine learning interpretability module, properties of the training data and the prediction.

[0015] In some embodiments of the non-transitory computer-readable storage medium, the instructions when executed by a computer further cause the computer to provide training data, predictions, and visualizations of properties of the training data and the predictions.

[0016] In some embodiments of the non-transitory computer-readable storage medium, the instructions when executed by a computer further cause the computer to, by a machine learning interpretability module, Heuristic Functions determining a value of Heuristic Functions incorporates the SHAP value of each training data point, the SHAP value of the predicted data point, the feature values ​​of the training data points, and the feature values ​​of the predicted data points. Heuristic Functionsmay comprise a combination of SHAP distance and feature distance, where the SHAP distance is the Euclidean distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted data points, the feature distance is the Euclidean distance between the feature vectors of the training data points and the feature vectors of the predicted data points, the SHAP vector is an ordered sequence of the SHAP values ​​of the data points, and the feature vector is an ordered sequence of the feature values ​​of the data points.

[0017] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by the computer, further cause the machine learning interpretability module to determine a SHAP value for one or more points of the prediction, the machine learning interpretability module to determine a SHAP value for one or more points of the training data, and the machine learning interpretability module to determine, for each of the one or more points of the prediction, a difference between the SHAP value of the prediction point and the SHAP value of each of the one or more points of the training data. The difference can be a Euclidean distance between the SHAP vector of the prediction point and the SHAP vector of each of the one or more points of the training data.

[0018] In some embodiments of the non-transitory computer-readable storage medium, the instructions when executed by a computer further cause the computer to: remove, by the machine learning interpretability module, a training data point from the training data to form a corrected training data set; retrain, by the machine learning interpretability module, the trained model on the corrected training data set; predict, by the machine learning interpretability module, based on the corrected training data set and provide a corrected prediction; compare, by the machine learning interpretability module, a difference between the prediction and the corrected prediction; and assign, by the machine learning interpretability module, a measure of influence to the removed training data point based on the difference.

[0019] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0020] Like reference numbers and designations in the various drawings refer to like elements.

[0021] To easily identify the discussion of any particular element or operation, the most significant number or digits of a reference number refer to the figure number in which that element is first introduced. [Brief description of the drawings]

[0022] [Figure 1] 1 is a flow chart according to one embodiment. [Diagram 2] 1 is a flow chart of a machine learning interpretability module according to one embodiment. [Figure 3A] FIG. 2 illustrates an example of a heuristic function according to one embodiment. [Figure 3B] 3B illustrates further aspects of the example heuristic function shown in FIG. 3A. [Figure 3C] 3B illustrates further aspects of the example heuristic function shown in FIG. 3A. [Figure 4] FIG. 1 illustrates an example according to one embodiment. [Diagram 5] FIG. 1 illustrates an example according to one embodiment. [Figure 6] 1 is a flow chart according to one embodiment. [Figure 7] FIG. 1 illustrates an example according to one embodiment. [Figure 8] FIG. 1 illustrates a system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In this disclosure, any embodiment or implementation of the subject matter described herein serves as an example, example, or illustration, and should not necessarily be construed as preferred or advantageous over other embodiments.

[0024] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments of the invention are shown by way of example in the drawings and are described in detail below. It is to be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents and alternatives falling within the spirit and scope of the disclosure.

[0025] The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a setup, device, or method comprising a list of components or steps does not include only those components or steps, but may include other components or steps that are not explicitly listed or that are inherent to such setup or device or method. In other words, one or more elements in a system or apparatus "comprising" does not, without more constraints, exclude the presence of other or additional elements in the system or apparatus.

[0026] In the following detailed description of the disclosed embodiments, reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, it being understood that other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. Therefore, the following description is not to be taken in a limiting sense.

[0027] FIG. 1 shows a flow chart 100 according to one embodiment.

[0028] The flowchart 100 comprises two stages: a first stage 102 and a second stage 104 .

[0029] In a first stage 102, training data 106 is used by a machine learning algorithm 108 to provide a trained model 110. The machine learning algorithm 108 uses the trained model 110 to provide one or more predictions 112 of future data.

[0030] In a second stage 104, the training data 106, the trained model 110 and the predictions 112 are then input into a machine learning interpretability module 114 to provide an explanatory output 116. The explanatory output 116 can be output visually and may include a graphical user interface 118 so that a user can interact with the explanatory output 116.

[0031] 2 illustrates a flow chart 200 of a Machine Learning Interpretability (MLI) module according to one embodiment. That is, FIG. 2 illustrates one embodiment of the Machine Learning Interpretability module 114.

[0032] The machine learning interpretability module 114 may operate in two stages: The first stage may comprise the calculation of past SHAP values ​​202 based on the training data 106 and the trained model 110, and future SHAP values ​​204 based on the trained model 110 and the predictions 112.

[0033] Once the past SHAP values ​​202 and future SHAP values ​​204 have been calculated, they are used in a second stage to calculate a similarity measure 206 between the past SHAP values ​​202 and the future SHAP values ​​204 .

[0034] The similarity measure 206 can then be output as an explanatory output 116 for the user. The explanatory output 116 can be visual and may include a graphical user interface 118 so that the user can interact with the results.

[0035] In some embodiments, Heuristic Functions can be used in the calculation of the similarity measure 206 by including a combination of both the difference between past SHAP values ​​202 and future SHAP values ​​204 and the difference between past feature values ​​and future feature values.

[0036] In some embodiments, each point (either past or predicted) is given a feature vector and a SHAP vector. A feature vector is simply an ordered sequence of values ​​assigned to predefined features of a data point. Similarly, a SHAP vector is simply an ordered sequence of values ​​assigned to predefined SHAP characteristics of a data point.

[0037] In some embodiments, the similarity measure can refer to the similarity between the predicted data points and the training data points, as measured by the distance between the vectors associated with each point. For example, a feature similarity measure can be obtained by calculating the distance between the feature vectors of the training data points and the feature vectors of the predicted points. Similarly, a SHAP similarity measure can be obtained by calculating the distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted points.

[0038] In some embodiments, Heuristic Functions can be a combination of feature distance and SHAP distance.

[0039] Heuristic Functions Examples of

[0040] In a time series, each training data point can have the following features: year, month, week of year, day of week, season, etc. For season, a numerical value can be assigned to the season (e.g., '0' for winter, '1' for summer or '0' for winter, '1' for spring, '2' for summer and '3' for fall). The feature vector did not provide information about attributes or values ​​in the data points. For example, for a lead time series, the feature vector does not provide information about the lead time of any given data point, it only provides information about the features of that data point.

[0041] Given expected point 'P F 'For'P F The feature vector of 'P F ' are obtained based on the features of each training data point 'H i Each training data point 'Hi ' and expected point 'P F ' feature similarity can be computed by standard techniques for computing Euclidean distance between vectors.

[0042] Similarly, the predicted point 'P F 'For'P F The SHAP vector for each training data point 'H i A SHAP vector for 'H' is also calculated. Unlike a feature vector, a SHAP vector contains information about the attributes or values ​​associated with a data point. For example, if a lead time is predicted, the SHAP vector contains information about the lead time for the data point in question. For each training data point 'H i ' and expected point 'P F The SHAP similarity to ' can be calculated by standard techniques for computing Euclidean distance between vectors.

[0043] A simple method including both feature distance and SHAP distance Heuristic Functions The HF can be formulated as follows:

[0044]

number

[0045] The value of 'a' can be adjusted between 0 and 1. When a=0, Heuristic Functions provides only feature similarity. When a=1, Heuristic Functions provides only SHAP similarity.

[0046] 3A, 3B and 3C are diagrams according to one embodiment. Heuristic Functions 300. In each of these figures, historical lead time data 318 is shown from approximately September 1, 2016 to approximately November 30, 2017, while forecasted lead times 320 are shown from approximately December 1, 2016 to approximately November 30, 2018.

[0047] 3A, 3B, and 3C each show a SHAP scale 322, which varies from a minimum value of '0' (as shown in FIG. 3A) to a maximum value of '100' (as shown in FIG. 3C). The value of the SHAP scale 322 is equal to the value of 'a' x 100, where 'a' is defined in Equation 1. That is, if a=1, the SHAP scale value is 100, if 'a'=0.5, the SHAP scale value is equal to 50, etc. That is, the SHAP scale value is equal to the value of 'a' x 100, where 'a' is defined in Equation 1 above. Heuristic Functions Represents the sliding value of the SHAP distance.

[0048] 3A, 3B, and 3C each show a prediction point scale 328 that indicates various points in the predicted lead time 320. In the figures, the prediction point scale 328 is set to '151', which corresponds to the prediction point 308.

[0049] SHAP and feature similarity are shown for the training data points versus predicted points 308 in each of Figures 3A, 3B, and 3C. Additionally, each figure shows a gradient key (gradient key 310 in Figure 3A, gradient key 312 in Figure 3B, and gradient key 314 in Figure 3C). In each figure, the darker the training data point is shaded by the gradient key, the greater its influence or weight of the training data point versus predicted points 308. Although the figures are shown in grayscale, it is understood that the graphic display is in color.

[0050] 3A shows the case where the value of the SHAP scale 322 is equal to zero, i.e., 'a'=0 in equation (1), which means that Heuristic Functionsis meant to represent only the feature similarity plot 302. The resulting feature similarity plot 302 shows that the darkest points in the historical lead time data 318 occur between training data points in the range of March 1, 2017 to July 1, 2017 for the predicted point 308 (close to May 15, 2018). That is, those points with the darkest slopes indicate that the greatest similarity occurs between training data points in the range of March 1, 2017 to July 1, 2017 for the predicted point 308. This is not surprising since these are training data points with similar dates (i.e., features) to the predicted point 308. Lead time has no bearing on feature similarity.

[0051] FIG. 3B shows a case where the value of the SHAP scale 322 is equal to 50, i.e., 'a'=0.5 in equation (1), which means that Heuristic Functions is meant to represent a half feature, half SHAP plot 304. The resulting half feature, half SHAP plot 304 shows that the greatest similarity occurs between training data points in the range of April 15, 2017 to June 15, 2017 for the predicted point 308 (close to May 15, 2018), as estimated by the point with the darkest slope. Note how the similarity range narrows from the range of March 1, 2017 to July 1, 2017 shown in FIG. 3A (with feature similarity only) to April 15, 2017 to June 15, 2017 in FIG. 3B (with half feature, half SHAP similarity).

[0052] FIG. 3C shows the case where the value of the SHAP scale 322 is equal to 100. That is, in equation (1), 'a'=1.0, which means that Heuristic Functionsrepresents a SHAP similarity plot 306. The resulting SHAP similarity plot 306 shows that the maximum SHAP similarity occurs between training data points around May 1, 2017 for predicted point 308 (close to May 15, 2018). Note how the similarity ranges in FIG. 3C narrow successively from the feature similarity plot 302 shown in FIG. 3A and from the half feature, half SHAP plot 304 shown in FIG. 3B.

[0053] FIG. 3C also shows the SHAP value 316 of the forecast point 308, which indicates that the most significant feature in the historical lead time data 318 for forecast point 308 is when the day of the week is equal to 1, which reduces the predicted lead time to 7.6 days (as opposed to other days of the week). Looking at the training data, based on SHAP similarity, one training data point around May 1, 2017 has a similar lead time to that of forecast point 308. Looking at this point in the past can provide some explanation as to why this predicted point (i.e., forecast point 308) was given a lower predicted lead time than the forecast points beside it. For the forecast points next to forecast point 308, the day of the week has a value different than '1', which has the least impact on the forecast according to the SHAP value 316. Therefore, any points adjacent to forecast point 308 do not show a reduction in lead time to the extent indicated by forecast point 308.

[0054] The next most significant feature in the historical lead time data 318 for the forecast point 308 is month equal to 5 (ie, the month of May).

[0055] FIG. 4 illustrates an example 400 according to one embodiment.

[0056] In Figure 4, the difference between past and future SHAP values ​​is shown for two adjacent predicted points 308 and 404. The SHAP similarity plot 306 and the SHAP values ​​316 are identical to the corresponding plot shown in Figure 3C.

[0057] Prediction point 404 is one day after prediction point 308.

[0058] For forecast point 308, the greatest impact in lowering the forecast lead time to 7.6 days is when the day of the week is 1, as shown by SHAP value 316. For forecast point 404, the forecast lead time jumps to 22, as shown by SHAP value 406. Furthermore, the day of the week has no impact in lowering the forecast lead time. Unlike forecast point 308, the week of the year set to 19 has the highest impact for forecast point 404. Although the figures are shown in grayscale, it will be understood that the graphic display is in color.

[0059] FIG. 5 illustrates an example 500 according to one embodiment.

[0060] Graph 502 shows an example of lead time versus date, showing both historical data 504 and forecast 506. In Figure 5, the forecast point 508 (around July 5th, indicated by the arrow) is highlighted. In example 500, the features are year, month of year, week of year, day of year, and season (e.g., '0' for winter, '1' for summer).

[0061] The SHAP value 510 of the forecast point 508 indicates that the forecast point 508 has a predicted lead time of 1.00 (the output value). Week of the year, with a value of 28, has the greatest influence on the forecast, while year (2018) has the next greatest influence. Day of the week is next in terms of influence on the forecast, with the resulting prediction of lead time being higher if the day of the week is other than 5. Season (with a value of '1') has the least influence on the forecast point 508.

[0062] The influence of each training data point on the predicted point 508 includes a combination of the past SHAP vector distance and the feature vector distance, as described above. Heuristic Functions 5, the SHAP scale 322 value is 50, which corresponds to '1'=0.5 in equation (1). Although the drawings are shown in greyscale, it will be understood that the graphic display is in colour.

[0063] In FIG. 5, a sliding scale value of 50 (out of 100) (represented by the SHAP scale 322) Heuristic Functions This is used in the evaluation of the feature vector distance and the past SHAP vector distance. Heuristic Functions This means that the evaluation of

[0064] FIG. 6 illustrates a flow chart 600 according to one embodiment.

[0065] Flowchart 600 illustrates another example of machine learning interpretability in which the impact of a training data point (on a prediction) is provided. The impact is not measured by the SHAP property, but instead affects how removing that training data point affects the prediction.

[0066] At block 604, the training data is used to train a machine learning model. At block 606, a prediction is made using the model. To obtain a measure of the influence of each training data point on the prediction, each training data point is individually removed (at block 608), forming a modified or new training data set (at block 610), the model is retrained on the new data set (at block 612), and a new prediction is made (at block 614). At block 616, the results of the prediction (made at block 614) are compared to the results of the prediction using the full training data set (made at block 606). The comparison may be made in any number of ways known in the art. The removed points are then added back into the training data set at block 618 along with a measure of the influence of the removed data point. Examples of influence measures are described below.

[0067] If this is not the last data point sampled for removal (decision block 620), then a new training data point is removed, at block 622, and the procedure is repeated, at block 610, with the new training data set.

[0068] On the other hand, if there are no more data points to sample for removal, then at block 624 the method ends and provides a measure of influence for each training data point.

[0069] If the removal of a particular training data point does not lead to a change in the resulting corrected data forecast, then that particular training data point does not affect the prediction. The greater the change in the corrected data forecast from the perfect data forecast, the greater the impact of the particular training data point on the forecast.

[0070] The impact measure can be provided to the user in any suitable manner known in the art. In some embodiments, the impact measure for each training data point is visually presented in graphical form. In some embodiments, the impact measure for each training data point is visually presented in tabular form.

[0071] 7 shows an example 700 according to one embodiment of machine learning interpretability. Flowchart 600 was used to obtain the example 700 shown.

[0072] Historical data 702 for lead times (indicated by black circles) from about September 1, 2016 to about January 7, 2018 was used to train a machine model and derive a perfect data forecast 704.

[0073] 7, a past data point 712 (from about March 25, 2017) is removed from the training data set. The revised prediction (based on removing the past data point 712) is shown as a corrected data forecast 706, which is in most cases lower than the full data forecast 704 over the entire forecast range from about January 8, 2018 to about January 8, 2019. The difference between the full data forecast 704 and the corrected data forecast 706 can be assessed by means known in the art, and the difference is given a difference value for the past data point 712.

[0074] In Figure 7, all of the remaining training data points (i.e., past data 702 except for past data point 712) have been subjected to the procedure described above for past data point 712 and have already been given difference values. This is indicated by the shading of the various points of past data 702. Although the figure is shown in greyscale, it will be understood that the graphic display is in color.

[0075] 7, the gradient key 714 is used as a measure to indicate that the lighter the training data point, the less its influence on the prediction. For example, data point 710, which is mostly white according to gradient key 714, has minimal influence on the prediction. Meanwhile, group 708 of data points (around August 1, 2017) is dark according to gradient key 714 and has a large influence on the prediction.

[0076] If the removal of a particular training data point does not lead to a change in the resulting corrected data forecast, then that particular training data point does not affect the prediction. The greater the change in the corrected data forecast from the perfect data forecast, the greater the impact of the particular training data point on the forecast.

[0077] A user can glean further information from the color gradient of the historical data 702 by looking for patterns of high or low impact data points. This can be accomplished via a graphical user interface that allows the user to select different data points according to the historical data 702 and see how the resulting corrected data forecast 706 changes relative to the complete data forecast 704.

[0078] FIG. 8 illustrates a system 800 according to one embodiment of machine learning interpretability.

[0079] The system server 802 includes machine learning algorithms, machine learning interpretability modules, and other modules and / or algorithms, and includes access to a library of SHAP algorithms. The machine learning storage device 812 can include training data used to train the machine learning algorithms.

[0080] The system 800 includes a system server 802, a machine learning storage device 812, a client data source 822, and one or more devices 814, 816, and 818. The system server 802 can include a memory 808, a disk 804, a processor 806, and a network interface 820. Although one processor 806 is shown, the system server 802 can include one or more processors. In some embodiments, the memory 808 can be a volatile memory, as compared to the disk 804, which can be a non-volatile memory. In some embodiments, the system server 802 can communicate with the machine learning storage device 812, the client data source 822, and one or more external devices 814, 816, and 818 via a network 810. Although the machine learning storage device 812 is shown as separate from the system server 802, the machine learning storage device 812 can also be incorporated within the system server 802 as a separate component within the system server 802 or as part of at least one of the memory 808 and the disk 804.

[0081] System 800 may also include additional features and / or functionality. For example, system 800 may also include additional storage (removable and / or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in FIG. 8 by memory 808 and disks 804. Storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Memory 808 and disks 804 are examples of non-transitory computer-readable storage media. Non-transitory computer readable media also include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory and / or other memory technologies, compact disk read only memory (CD-ROM), digital versatile disk (DVD) and / or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices and / or any other medium that can be used to store the desired information and that can be accessed by system 800. Any such non-transitory computer readable storage media can be part of system 800.

[0082] Communications between the system server 802, the machine learning storage device 812, and one or more external devices 814, 91, and 818 over the network 810 can be over a variety of network types. In some embodiments, the processor 806 may be arranged to communicate with the network 810 through a network interface 820. The network interface 820 may communicate with the network 810. The network interface 820 may use connection protocols including, but not limited to, direct bond, Ethernet (e.g., twisted pair 10 / 40 / 400 base-T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, etc. Non-limiting example network types may include Fibre Channel, SCSI, Bluetooth, Ethernet, Wi-Fi, Infrared Data Access (IrDA), Local Area Networks (LAN), Wireless Local Area Networks (WLAN), Wide Area Networks (WAN), such as the Internet, Serial, and Universal Serial Bus (USB). In general, communication between the various components of system 800 may occur through hardwire connections, cellular, Wi-Fi, or Bluetooth network components, and the like. In some embodiments, one or more electronic devices of system 800 may include cloud-based features, such as cloud-based memory storage.

[0083] The machine learning storage 812 may implement an "in-memory" database in which volatile (e.g., non-disk based) storage (e.g., random access memory) is used both for cache memory and to store the entire database during operation, and persistent storage (e.g., one or more fixed disks) is used for offline persistence and maintaining database snapshots. Alternatively, the volatile storage may be used as cache memory to store recently used data, and the persistent storage stores the entire database.

[0084] The machine learning storage 812 may store metadata about the structure, relationships, and meaning of the data. This information may include data defining the schema of the database tables in which the data is stored. The database table schema may identify the names of the database tables, the columns of the database tables, the data types associated with each column, and other information associated with the database tables. The machine learning storage 812 may additionally or alternatively support multitenancy by providing multiple logical database systems that are programmatically separated from each other. Furthermore, the data may be indexed and / or selectively replicated with indexes to enable rapid exploration and retrieval thereof. Additionally, the machine learning storage 812 may store many machine learning models that are accessed by the system server 802. Many ML models may be used.

[0085] In some embodiments where machine learning is used, gradient boosting trees, ensemble trees, and support vector regression can be used. In some embodiments of machine learning, one or more clustering algorithms can be used. Non-limiting examples include hierarchical clustering, k-means, mixture models, density-based spatial clustering for noisy applications, and ordering points to identify the clustering structure.

[0086] In some implementations of machine learning, one or more anomaly detection algorithms can be used. Non-limiting examples include local outlier factor methods.

[0087] In some implementations of machine learning, neural networks can be used.

[0088] The client data source 822 may provide various raw data from the user including, but not limited to, point of sale data indicating all sales records of the client's products at all locations, all inventory history of the client's products at all locations, promotional campaign details for all products at all locations, and events significant / related to the sales of the client's products at all locations.

[0089] Using the network interface 820 and the network 810, the system server 802 may communicate with one or more devices 814, 816, and 818. These devices 814, 816, and 818 may include, but are not limited to, personal computers, servers, various mobile devices, such as mobile phones, smartphones (e.g., Apple iPhone, Blackberry, Android based phones, etc.), tablet computers, e-readers (Amazon Kindle, Nook, etc.), laptop computers, notebooks, game consoles (Microsoft Xbox, Nintendo DS, Sony PlayStation, etc.), and the like.

[0090] Using the network 810, the system server 802 can retrieve data from the machine learning storage 812 and the client data sources 822. The retrieved data can be stored in the memory 808 or on the disk 804. In some embodiments, the system server 802 also includes a web server and can format resources into a format suitable for display in a web browser.

[0091] Once the preliminary machine learning results are provided to any one or more devices, a user can correct the results, which are then resent to machine learning storage 812 for further execution. The results can be corrected by interaction with one or more data files that are then sent to machine learning storage 812, or through a user interface at one or more of devices 814, 816, and 818. For example, at device 816, a user can correct the results using a graphical user interface.

[0092] Although the algorithms described above, including those with respect to the flow charts described above, are described separately, it is to be understood that any two or more of the algorithms disclosed herein may be combined in any combination. Any of the methods, modules, algorithms, implementations or procedures described herein may include machine-readable instructions for execution by (a) a processor, (b) a controller, and / or (c) any other suitable processing device. While any algorithm, software or method disclosed herein may be implemented in software stored on a non-transitory tangible medium, such as, for example, a flash memory, a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD) or other memory device, those skilled in the art will readily recognize that the entire algorithm and / or parts thereof may alternatively be executed by devices other than a controller and / or may be implemented in firmware or dedicated hardware in a well-known manner (e.g., it may be implemented by an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable logic device (FPLD), discrete logic, etc.). Additionally, although particular algorithms have been described with reference to the flowcharts set forth herein, those skilled in the art will readily recognize that many other ways of implementing the exemplary machine-readable instructions may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be modified, eliminated, or combined.

[0093] It should be noted that the algorithms shown and discussed herein have various modules that perform specific functions and interact with each other. It should be understood that these modules are separated based on their functionality merely for the purposes of explanation and also represent computer hardware and / or executable software code stored on a computer readable medium for execution on suitable computing hardware. The various functions of the different modules and units can be combined or separated as hardware and / or software stored on a non-transitory computer readable medium as described above as modules in any manner and can be used separately or in combination.

[0094] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequence shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method comprising: training, by a processor, a regression machine learning model using the training data; predicting, by the processor, a prediction based on the trained model; receiving, by a machine learning interpretability module, the training data, the trained model, and the predictions; comparing, by the machine learning interpretability module, properties of the training data and the predictions; Including, The step of comparing the characteristics includes: removing, by the machine learning interpretability module, training data points from the training data to form a corrected training data set; retraining, by the machine learning interpretability module, the trained model on the corrected training data set; predicting, by the machine learning interpretability module, based on the corrected training data set and providing a corrected prediction; comparing, by the machine learning interpretability module, the difference between the prediction and the corrected prediction; assigning, by the machine learning interpretability module, a measure of influence to the removed training data points based on the difference; A method comprising:

2. The method of claim 1 , wherein comparing characteristics comprises visualizing the training data, the predictions, and the characteristics of the training data and the predictions.

3. The step of comparing the characteristics includes: determining, by the machine learning interpretability module, a value of a heuristic function for each training data point; the prediction comprises a plurality of predicted data points; the heuristic function incorporates a SHAP (Shapley Additive exPlanations) value for each training data point, a SHAP value for the predicted data point, feature values ​​for the training data points, and feature values ​​for the predicted data points; The method according to claim 1 or 2.

4. the heuristic function comprises a combination of SHAP distance and feature distance; The SHAP distance is the Euclidean distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted data points; the feature distance is the Euclidean distance between the feature vectors of the training data points and the feature vectors of the predicted data points; The SHAP vector is a sequence of the SHAP values ​​of the data points; The method of claim 3 , wherein the feature vector is an ordered sequence of feature values ​​of data points.

5. The step of comparing the characteristics includes: determining, by the machine learning interpretability module, a SHAP value for one or more points of the prediction; determining, by the machine learning interpretability module, a SHAP value for one or more points of the training data; determining, by the machine learning interpretability module, for each of the one or more points of the prediction, a difference between the SHAP value of the prediction point and the SHAP value of each of the one or more points of the training data; The method of claim 1 or 2, comprising:

6. The method of claim 5 , wherein the difference is the Euclidean distance between the SHAP vector of the prediction point and the SHAP vector of each of the one or more points of the training data.

7. 1. A system comprising: A processor; A memory for storing instructions; the instructions, when executed by the processor, cause the system to: training, by a processor, a regression machine learning model using the training data; predicting, by the processor, a prediction based on the trained model; receiving, by a machine learning interpretability module, the training data, the trained model, and the predictions; comparing properties of the training data and the predictions by the machine learning interpretability module; removing training data points from the training data by the machine learning interpretability module to form a corrected training data set; retraining, by the machine learning interpretability module, the trained model on the corrected training data set; predicting, by the machine learning interpretability module, based on the corrected training data set and providing a corrected prediction; comparing the difference between the prediction and the corrected prediction by the machine learning interpretability module; The machine learning interpretability module assigns a measure of influence to the removed training data points based on the difference. A system configured as follows:

8. The system of claim 7 , further configured to provide a visualization of the training data, the predictions, and the characteristics of the training data and the predictions.

9. The machine learning interpretability module is further configured to determine a value of a heuristic function for each training data point; the prediction comprises a plurality of predicted data points; 9. The system of claim 7 or 8, wherein the heuristic function incorporates a SHAP value for each training data point, a SHAP value for the predicted data point, feature values ​​for the training data points, and feature values ​​for the predicted data points.

10. the heuristic function comprises a combination of SHAP distance and feature distance; The SHAP distance is the Euclidean distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted data points; the feature distance is the Euclidean distance between the feature vectors of the training data points and the feature vectors of the predicted data points; The SHAP vector is a sequence of the SHAP values ​​of the data points; The system of claim 9 , wherein the feature vector is an ordered sequence of feature values ​​of data points.

11. determining, by the machine learning interpretability module, a SHAP value for one or more points of the prediction; determining, by the machine learning interpretability module, a SHAP value for one or more points of the training data; determining, by the machine learning interpretability module, for each of the one or more points of the prediction, a difference between the SHAP value of the prediction point and the SHAP value of each of the one or more points of the training data; The system according to claim 7 or 8, further configured to:

12. The system of claim 11 , wherein the difference is a Euclidean distance between a SHAP vector of the prediction point and a SHAP vector of each of the one or more points of the training data.

13. A non-transitory computer-readable storage medium containing instructions that, when executed by a computer, cause the computer to: training, by a processor, a regression machine learning model using the training data; causing the processor to predict a prediction based on the trained model; receiving, by a machine learning interpretability module, the training data, the trained model, and the predictions; causing the machine learning interpretability module to compare properties of the training data and the predictions; removing training data points from the training data by the machine learning interpretability module to form a corrected training data set; retraining the trained model on the corrected training data set by the machine learning interpretability module; causing the machine learning interpretability module to predict based on the corrected training data set and provide a corrected prediction; causing the machine learning interpretability module to compare the difference between the prediction and the corrected prediction; causing the machine learning interpretability module to assign a measure of influence to the removed training data points based on the difference. A non-transitory computer-readable storage medium.

14. 14. The computer-readable storage medium of claim 13, wherein the instructions when executed by a computer further cause the computer to provide a visualization of the training data, the prediction, and the characteristics of the training data and the prediction.

15. The instructions, when executed by a computer, further cause the computer to: causing the machine learning interpretability module to determine a value of a heuristic function for each training data point; the prediction comprises a plurality of predicted data points; 15. The computer-readable storage medium of claim 13, wherein the heuristic function incorporates a SHAP value for each training data point, a SHAP value for the predicted data point, feature values ​​for the training data points, and feature values ​​for the predicted data points.

16. the heuristic function comprises a combination of SHAP distance and feature distance; The SHAP distance is the Euclidean distance between the SHAP vectors of the training data points and the SHAP vectors of the predicted data points; the feature distance is the Euclidean distance between the feature vectors of the training data points and the feature vectors of the predicted data points; The SHAP vector is a sequence of the SHAP values ​​of the data points; The computer-readable storage medium of claim 15 , wherein the feature vector is an ordered sequence of feature values ​​of data points.

17. The instructions, when executed by a computer, further cause the computer to: causing the machine learning interpretability module to determine a SHAP value for one or more points of the prediction; causing the machine learning interpretability module to determine a SHAP value for one or more points of the training data; 15. The computer-readable storage medium of claim 13 or 14, further comprising: the machine learning interpretability module determining, for each of the one or more points of the prediction, a difference between the SHAP value of the predicted point and the SHAP value of each of the one or more points of the training data.

18. 20. The computer-readable storage medium of claim 17, wherein the difference is a Euclidean distance between a SHAP vector of the prediction point and a SHAP vector of each of the one or more points of the training data.

Images