Intelligent training and intelligent deployment of machine learning models

Abstract

Systems / techniques are provided that facilitate intelligent training and intelligent deployment of machine learning models. In various implementations, a system can access a first set of data candidates that are available for training of a machine learning model. In various aspects, the system can estimate at least one feature distribution of the first set of data candidates. In various cases, the system can identify a strict subset of data candidates in the first set of data candidates, where at least one feature distribution of the strict subset of data candidates matches the at least one feature distribution of the first set of data candidates. In various cases, the system can train the machine learning model on the strict subset of data candidates.

Description

Technical Field

[0001] This topic covers machine learning models in general, and more specifically, the intelligent training and deployment of machine learning models. Background Technology

[0002] The machine learning model is implemented in two phases: a training phase and a deployment / inference phase. During the training phase, the machine learning model is iteratively trained to accurately perform its desired function, transformation, and / or determination. During the deployment / inference phase, the machine learning model is executed on real-world data after training, so that the desired function, transformation, and / or determination are applied to such real-world data.

[0003] Unfortunately, existing technologies for training and deploying machine learning models suffer from various drawbacks. Specifically, when implementing the training phase of a machine learning model using existing technologies, there is a significant possibility that the model will be overtrained and / or overfitted. Furthermore, when implementing the deployment / inference phase using existing technologies, there is a significant possibility that the machine learning model, despite being trained, may still be unable to accurately analyze certain real-world data.

[0004] Therefore, systems and / or technologies that can solve one or more of these technical problems may be desirable. Summary of the Invention

[0005] The following summary is presented to provide a basic understanding of one or more embodiments of the invention. This summary is not intended to identify key or essential elements, nor is it intended to depict any scope of the specific embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to the more detailed description that follows. In one or more embodiments described herein, devices, systems, computer-implemented methods, apparatuses, and / or computer program products are described that facilitate intelligent training and / or intelligent deployment of machine learning models.

[0006] According to one or more embodiments, a system is provided. The system may include a computer-readable storage device capable of storing computer-executable components. The system may also include a processor operatively coupled to the computer-readable storage device and capable of executing the computer-executable components stored in the computer-readable storage device. In various embodiments, the computer-executable components may include a receiver component. In various cases, the receiver component may access a first set of data candidates that can be used to train a machine learning model. In various aspects, the computer-executable components may also include a feature component. In various cases, the feature component may estimate at least one feature distribution of the first set of data candidates. In various cases, the computer-executable components may also include a selection component. In various cases, the selection component may identify a strict subset of data candidates in the first set of data candidates, wherein at least one feature distribution of the strict subset of data candidates matches at least one feature distribution of the first set of data candidates. In various aspects, the computer-executable components may also include a training component. In various cases, the training component may train a machine learning model on the strict subset of data candidates.

[0007] According to one or more implementation schemes, the above system can be implemented as a computer-implemented method and / or computer program product. Attached Figure Description

[0008] Figure 1 A block diagram of an exemplary, non-limiting system for facilitating intelligent training and / or intelligent deployment of machine learning models according to one or more embodiments described herein is shown.

[0009] Figure 2 An exemplary, non-limiting block diagram of a set of training data candidates according to one or more embodiments described herein is shown.

[0010] Figure 3 A block diagram of an exemplary, non-limiting system comprising a set of training feature distributions according to one or more embodiments described herein is shown, which facilitates intelligent training and / or intelligent deployment of machine learning models.

[0011] Figure 4 An exemplary, non-limiting block diagram illustrating how a set of training feature distributions can be computed according to one or more embodiments described herein.

[0012] Figure 5 A block diagram of an exemplary, non-limiting system comprising a set of selected training data candidates and a set of selected training feature distributions according to one or more embodiments described herein is shown, which facilitates intelligent training and / or intelligent deployment of machine learning models.

[0013] Figure 6An exemplary, non-limiting block diagram of a set of selected training data candidates according to one or more embodiments described herein is shown.

[0014] Figure 7 An exemplary, non-limiting block diagram illustrating how a set of selected training feature distributions can be computed according to one or more embodiments described herein.

[0015] Figure 8 An exemplary, non-limiting block diagram is shown illustrating how a set of training feature distributions can be compared to a set of selected training feature distributions, according to one or more embodiments described herein.

[0016] Figure 9 A flowchart is shown of an exemplary, non-limiting computer implementation of a method for facilitating intelligent training of a machine learning model according to one or more embodiments described herein.

[0017] Figure 10 A block diagram is shown of one or more embodiments of a proposed exemplary, non-limiting system comprising a set of deployed data candidates, a set of deployed feature distributions, and a system that facilitates intelligent training and / or intelligent deployment of machine learning models.

[0018] Figure 11 An exemplary, non-limiting block diagram of a set of deployed data candidates according to one or more embodiments described herein is shown.

[0019] Figure 12 An exemplary, non-limiting block diagram illustrating how a set of deployed feature distributions can be computed according to one or more embodiments described herein.

[0020] Figure 13 An exemplary, non-limiting block diagram is shown illustrating how a set of training feature distributions can be compared with a set of deployed feature distributions according to one or more embodiments described herein.

[0021] Figure 14 A flowchart is shown of an exemplary, non-limiting computer implementation of a method for facilitating the intelligent deployment of machine learning models according to one or more embodiments described herein.

[0022] Figure 15 A block diagram of an exemplary, non-limiting system including a confidence score according to one or more embodiments described herein is shown, which facilitates intelligent training and / or intelligent deployment of machine learning models.

[0023] Figure 16An exemplary, non-limiting block diagram illustrating how confidence scores can be calculated according to one or more embodiments described herein is shown.

[0024] Figure 17 A flowchart is shown of an exemplary, non-limiting computer implementation of a method for facilitating intelligent training and / or intelligent deployment of machine learning models according to one or more embodiments described herein.

[0025] Figure 18 A block diagram is shown illustrating an exemplary, non-limiting operating environment in which one or more embodiments described herein may be facilitated.

[0026] Figure 19 An exemplary networking environment operable to perform the various specific implementations described herein is shown. Detailed Implementation

[0027] The following specific embodiments are merely exemplary and are not intended to limit the implementation and / or application or use of the embodiments. Furthermore, they are not intended to be construed as being bound by any express or implied information set forth in the foregoing "Background Art" or "Summary of the Invention" or "Detailed Description" sections.

[0028] One or more embodiments will now be described with reference to the accompanying drawings, wherein the same reference numerals are always used to denote the same elements. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent that, in various cases, one or more embodiments may be practiced without these specific details.

[0029] Machine learning models (e.g., neural networks, support vector machines, Naive Bayes models, linear and / or logistic regression models, decision tree models) can be implemented in two phases: a training phase and a deployment / inference phase. During the training phase, the machine learning model can be iteratively trained (e.g., via backpropagation for neural networks, via sample segmentation for decision trees) to accurately perform its desired function, transformation, and / or determination (e.g., in some cases, the machine learning model may be expected to classify the input data; in others, it may be expected to segment the input data; in still others, it may be expected to synthesize / predict output data based on the input data). During the deployment / inference phase (e.g., which occurs after training), the machine learning model can be executed on real-world data such that the desired function, transformation, and / or determination can be applied to such real-world data (e.g., enabling such real-world data to be classified, enabling such real-world data to be segmented, and / or enabling output data to be synthesized / predicted based on such real-world data).

[0030] Unfortunately, existing techniques for training and / or deploying machine learning models suffer from various drawbacks. Specifically, when implementing the training phase of a machine learning model via existing techniques, there is a significant possibility that the model will be overtrained and / or overfitted. In other words, when in a state of overtraining / overfitting, the machine learning model can be viewed as attempting to describe the random errors exhibited by the data on which it has been trained, rather than attempting to describe the more general relationships between the variables of that data. Therefore, such overtraining and / or overfitting can significantly reduce the generalizability of the machine learning model, which is undesirable.

[0031] Furthermore, when implementing the deployment / inference phase using existing technologies, there is a significant possibility that machine learning models, despite being trained, may still fail to accurately analyze certain real-world data. Specifically, even without overtraining / overfitting, a machine learning model can only execute accurately on real-world data that is sufficiently "similar" to the data on which the model was trained. If a machine learning model executes on a given piece of data that is not sufficiently "similar" to the data on which it was trained, any results produced by the model based on such a given piece of data cannot be considered reliable. Unfortunately, existing technologies do not take any steps to prevent the execution of machine learning on such insufficiently "similar" data, which is undesirable.

[0032] Therefore, systems and / or technologies that can solve one or more of these technical problems may be desirable.

[0033] The various innovative implementations of this topic can solve one or more of these technical problems. One or more implementations described herein include systems, computer-implemented methods, apparatuses, and / or computer program products that can facilitate the intelligent training and deployment of machine learning models.

[0034] In some respects, the inventors of the various embodiments described herein recognize that existing techniques for training machine learning models can lead to overtraining / overfitting, specifically because such existing techniques utilize excessive redundant training data. Therefore, the inventors realize that overtraining / overfitting can be mitigated and / or improved by preventing machine learning models from being trained on such redundant training data. More specifically, given the entire set of training data on which a machine learning model is intended to be trained, the entire set of such training data may exhibit a specific feature distribution. The inventors realize that there may exist subsets of the entire set of such training data, wherein the subset may include less training data than the entire set (e.g., the subset may be a strict subset and / or an appropriate subset), and wherein the feature distribution exhibited by the subset may match the specific feature distribution of the entire set (e.g., may not be statistically significantly different from that specific feature distribution). Therefore, the machine learning model can be trained on such subsets instead of the entire set. Since the subset can be strict / appropriate (e.g., may contain less data than the entire set), the likelihood of overtraining / overfitting can be reduced by training the machine learning model on the subset instead of the entire set. Furthermore, since subgroups can have feature distributions that are statistically significantly similar to the entire group, a machine learning model trained on subgroups rather than the entire group can be considered to have no omissions in substantial learning. In some cases, subgroups can be considered as the minimum required training set. In various respects, training a machine learning model on such subgroups rather than the entire group can be referred to as "intelligent training."

[0035] Furthermore, the inventors recognize that prior art for deploying / inferring machine learning models can be susceptible to unreliability, specifically because such prior art does not take into account the feature distribution exhibited by the deployment data (e.g., by the data on which the trained machine learning model is expected to be executed). Therefore, the inventors realize that the unreliability during the deployment / inference of machine learning models can be mitigated and / or improved by considering the feature distribution of the deployment data. More specifically, given a first dataset on which a machine learning model has been trained, such a first dataset may exhibit a first feature distribution. Furthermore, given a second dataset on which the trained machine learning model is expected to be executed, such a second dataset may exhibit a second feature distribution. If the second feature distribution matches the first feature distribution (e.g., it is not statistically significantly different from the first feature distribution), it can be concluded that the machine learning model can be reliably executed on the second dataset (e.g., it can be concluded that the second dataset is sufficiently "similar" to the first dataset). On the other hand, if the second feature distribution fails to match the first feature distribution (e.g., it is statistically significantly different from the first feature distribution), it can be concluded that the machine learning model cannot be reliably executed on the second dataset (e.g., it can be concluded that the second dataset is not sufficiently "similar" to the first dataset). In various cases, deploying / inferring a machine learning model only on deployment data that exhibits a feature distribution that matches the feature distribution of the data on which the machine learning model was trained can be referred to as "intelligent deployment".

[0036] In various respects, the various implementations described herein can be viewed as computerized tools (e.g., any suitable combination of computer-executable hardware and / or computer-executable software) that can facilitate intelligent training and intelligent deployment of machine learning models. In other words, a computerized tool can be viewed as an automated platform operating in any suitable computational foreground and / or any suitable computational background (e.g., stealth operations) that enables any suitable machine learning model to undergo intelligent training and / or intelligent deployment. In various cases, the computerized tool may include receiver components, feature components, selection components, training components, and / or deployment components.

[0037] In various implementations, a complete set of training data candidates may exist. In various aspects, the complete set of training data candidates may include any suitable number of training data candidates. In various cases, each training data candidate may be any suitable electronic information having any suitable format and / or dimension as needed (e.g., a training data candidate may be a collection of time series data, a collection of waveform data, a two-dimensional pixel array and / or a three-dimensional voxel array). In any case, each training data candidate may exhibit any suitable number of features and / or be otherwise associated with it (e.g., for time series data, features may be average time series value, maximum time series value, and / or minimum time series value; for waveform data, features may be spectral values, maximum amplitude, and / or minimum amplitude; for image data, features may be average pixel / voxel intensity value, maximum pixel / voxel intensity value, and / or minimum pixel / voxel intensity value). In some cases, the complete set of training data candidates may be annotated. In other cases, the complete set of training data candidates may be unannotated.

[0038] In various implementations, machine learning models may be present. In various aspects, the machine learning model may exhibit any suitable artificial intelligence architecture as needed (e.g., neural networks, support vector machines, Naive Bayes, linear and / or logistic regression, decision trees). In various cases, the machine learning model may be configured to receive any one of the entire set of training data candidates as input and produce some determination, prediction, and / or result based on the input training data candidates as output (e.g., in some cases, the machine learning model may be configured to classify the input data candidates; in others, it may be configured to segment the input data candidates; in still others, it may be configured to synthesize and / or predict new data based on the input data candidates).

[0039] In any case, it is desirable to train the machine learning model using the entire set of training data candidates. That is, if the entire set of training data candidates is annotated, it is desirable to train the machine learning model in a supervised manner using the entire set of training data candidates. Conversely, if the entire set of training data candidates is unannotated, it is desirable to train the machine learning model in an unsupervised and / or reinforcement learning manner using the entire set of training data candidates. In any case, it is desirable to train the machine learning model based on the entire set of training data candidates, thereby reducing the possibility of overtraining and / or overfitting of the machine learning model. In various cases, the computational tools described herein can facilitate such training.

[0040] In various implementations, the receiver component of the computerized tool can electronically receive and / or otherwise electronically access all groups and / or machine learning models of training data candidates. In some aspects, the receiver component can electronically retrieve all groups and / or machine learning models of training data candidates from any suitable centralized and / or decentralized data structure (e.g., graph data structure, relational data structure, hybrid data structure), whether retrieved remotely and / or locally on the receiver component. In any case, the receiver component can electronically acquire and / or access all groups and / or machine learning models of training data candidates, enabling other components of the computerized tool to electronically interact with (e.g., read, write, edit, copy, manipulate) all groups and / or machine learning models of training data candidates.

[0041] In various implementations, the feature component of the computerized tool can electronically estimate one or more training feature distributions associated with the entire set of training data candidates. As mentioned above, each training data candidate can include any suitable number of features and / or associated with them. For ease of illustration, it is assumed that each training data candidate includes... q One characteristic, q For any suitable positive integer. Therefore, the feature component can be computed for all groups of candidates in the training data. q There are 1 training feature distribution, one training feature distribution for each feature. More specifically, for any suitable positive integer... r ,in The feature component can estimate the first r The training feature distribution, where the th training feature distribution is... r The distribution of the training features describes how the th training feature distribution ...... r The value of the feature is distributed across all candidate groups in the training data. In some cases, if the feature's value is distributed across all candidate groups in the training data... r If the nth characteristic distribution is a normal distribution, then this characteristic distribution can be derived from the nth characteristic distribution. r The mean, the first r The variance and / or the first r The median value is used to represent this. In other cases, if the first... r If the nth characteristic distribution is a gamma distribution, then this characteristic distribution can be derived from the nth characteristic distribution. r The shape parameter, the first r The rate parameter and / or the first r The median value is used to represent this. In any case, the feature component can generate one or more training feature distributions for all groups of training data candidates.

[0042] In various implementations, the selection component of the computerized tool can electronically identify a selected subgroup of training data candidates within and / or from all groups of training data candidates. In various cases, the selected subgroup may contain fewer training data candidates than the total number of groups (e.g., the selected subgroup may be a strict subgroup and / or an appropriate subgroup). Furthermore, in various aspects, the selection component can electronically estimate one or more selected training feature distributions associated with the selected subgroup. Note that if one or more training feature distributions calculated by the feature component include... q If there are multiple distributions (e.g., each distribution describes how to distribute the corresponding features across all groups of training data candidates), then one or more selected training feature distributions computed by the selection component may also include... q Each distribution (e.g., each distribution describes how the corresponding features are distributed across the selected subgroups of training data candidates). In various cases, the selection component can pick the selected subgroups such that one or more selected training feature distributions match one or more training feature distributions respectively (e.g., are not statistically significantly different from the training feature distributions respectively).

[0043] More specifically, the selection component can iteratively identify the selected subgroup as follows: First, the selection component can create an initial empty group to serve as the selected subgroup. Next, the selection component can iteratively insert training data candidates that are not yet in the selected subgroup from all groups of training data candidates. That is, the size of the selected subgroup can increase with each iteration, meaning that the selected training feature distribution group can change with each iteration. In various cases, at each iteration, the selection component can estimate one or more selected training feature distributions of the selected subgroup, and the selection component can compare one or more selected training feature distributions with one or more training feature distributions estimated by the feature component, respectively. In various aspects, such comparisons can be facilitated via statistical hypothesis testing (such as z-tests, t-tests, and / or Kolmogorov-Smirnov tests). For example, if there exists q The distribution of training features and q Given the chosen training feature distribution, for each positive integer... s ,in The selected component can be determined through statistical hypothesis testing. s The distribution of the training features and the... s The selected training feature distributions are compared. If the selection component determines that one or more of the selected training feature distributions have not yet matched one or more training feature distributions estimated by the feature component (e.g., if the selection component determines that for at least one positive integer...), the selection component will compare the selected training feature distributions with .... t ,in , No. tThe distribution of the training features and the... t If the selected training feature distributions are statistically significantly different, the selection component can proceed to the next iteration (e.g., insert another training data candidate into the selected subgroup, re-estimate one or more selected training feature distributions, and compare the one or more re-estimated selected training feature distributions with one or more training feature distributions). In contrast, if the selection component alternatively determines that one or more selected training feature distributions now match one or more training feature distributions calculated by the feature component (e.g., if the selection component determines that for all positive integers...), the selection component can proceed to the next iteration (e.g., insert another training data candidate into the selected subgroup, re-estimate one or more selected training feature distributions, and compare the re-estimated selected training feature distributions with one or more training feature distributions calculated by the feature component). u ,in , No. u The distribution of the training feature is not related to the first training feature. u If the distributions of the selected training features are statistically significantly different, then the selection of the component can stop the iteration, and the selected subgroup can be considered complete and / or identified.

[0044] In various implementations, once the selection component identifies the selected subgroup, the training component of the computerized tool can electronically train a machine learning model on the selected subgroup. As those skilled in the art will understand, if the training data candidates are annotated, the training component can perform supervised training of the machine learning model on the selected subgroup. On the other hand, if the training data candidates are unannotated, the training component can perform unsupervised training and / or reinforcement learning of the machine learning model on the selected subgroup. In either case, the machine learning model can be trained on the selected subgroup of the training data candidates rather than on all groups of training data candidates. Because the selected subgroup may have fewer training data candidates than the entire group, training the machine learning model on the selected subgroup may have a lower probability of leading to overtraining and / or overfitting of the machine learning model. Furthermore, because one or more selected training feature distributions may match one or more training feature distributions of the entire group of training data candidates (e.g., may not be statistically significantly different from such training feature distributions), the selected subgroup can be considered substantially equivalent to the entire group for training purposes. Therefore, it may be beneficial to train the machine learning model on a selected subgroup rather than on all groups (e.g., to reduce the chance of overtraining / overfitting without diminishing the substantial learning achieved by the machine learning model). As mentioned above, this can be referred to as “intelligent training” of the machine learning model.

[0045] In this way, computerized tools can be viewed as automated platforms used to facilitate the intelligent training of machine learning models. In some cases, computerized tools can visually present any suitable results associated with such intelligent training on any suitable computer monitor / screen / monitor. For example, in some cases, the selection component can visually present the results of any statistical hypothesis tests it performs (e.g., the calculated p-value can be presented / displayed at each iteration). Similarly, in some cases, the selection component can visually present any selected training feature distribution from the selected set of training feature distributions (e.g., as a bar chart and / or violin plot). As yet another example, in some cases, the feature component can visually present any training feature distribution from the set of training feature distributions (e.g., as a bar chart and / or violin plot). Therefore, in various aspects, computerized tools can be viewed as creating visual dashboards depicting any suitable information about the intelligent training of machine learning models.

[0046] Now that intelligent training through computerized tools has been described, consider how computerized tools can perform intelligent deployment.

[0047] In various implementations, the receiver component may also access a set of deployed data candidates on which it intends to perform, deploy, and / or otherwise infer a trained machine learning model. In various cases, the deployed data candidates may have the same format and / or dimensions as the training data candidates, as described above.

[0048] In various implementations, the deployment component of the computerized tool can electronically estimate one or more deployed feature distributions of the set of deployed data candidates. Note that if one or more training feature distributions computed by the feature component include... q Each distribution (e.g., each distribution describes how to distribute the corresponding features across all groups of training data candidates), and / or if one or more of the selected training feature distributions computed by the selection component include... q If there are multiple distributions (e.g., each distribution describes how to distribute the corresponding features across selected subgroups of training data candidates), then one or more deployed feature distributions computed by the deployment component may also include... q Each distribution (e.g., each distribution describes how the corresponding features are distributed across the set of deployed data candidates).

[0049] In various scenarios, the deployment component can compare one or more deployed feature distributions with one or more training feature distributions (e.g., and / or with one or more selected training feature distributions) individually (e.g., via statistical hypothesis testing, such as z-tests, t-tests, and / or Kolmogorov-Smirnov tests). For example, for all positive integers... v ,in Selecting components allows for the execution of statistical hypothesis tests to determine the first... v Does the feature distribution of the deployed element correspond to the first element? v The distribution of the training features is statistically significantly different (and / or different from) the distribution of the training features. v (The distribution of the selected training features is different). If for all positive integers... w ,in , No. w The feature distribution of the deployed element is not the same as that of the first element. w The distribution of the training features is statistically significantly different (and / or different from) the distribution of the training features. w If the selected training feature distributions are different, then the deployment component can determine that the deployed set of data candidates is sufficiently "similar" to the entire set of training data candidates (e.g., and / or to the selected sub-sets of the training data candidates). That is, the deployment component can determine that the machine learning model can reliably execute on the deployed set of data candidates. On the other hand, if for at least one positive integer... w ,in , No. w The feature distribution of the deployed units and the first w The distribution of the training features is statistically significantly different (and / or different from) the distribution of the training features. w If the selected training feature distributions are different, then the deployment component can determine that the deployed data candidate set is not sufficiently "similar" to the entire set of training data candidates (e.g., and / or to a selected sub-set of training data candidates). That is, the deployment component can determine that the machine learning model cannot reliably execute on this set of deployed data candidates. In various aspects, determining whether a machine learning model can reliably execute on this set of deployed data candidates in this manner can be referred to as the "intelligent deployment" of the machine learning model.

[0050] In this way, computerized tools can be viewed as automated platforms for facilitating the intelligent deployment of machine learning models. In some cases, computerized tools can visually present any suitable results associated with such intelligent deployment on any suitable computer monitor / screen / monitor. For example, in some cases, the deployment component can visually present the results of any statistical hypothesis tests it performs (e.g., it can present / display the calculated p-values). Similarly, in some cases, the deployment component can visually present any deployed feature distribution within the set of deployed feature distributions (e.g., as a bar chart and / or violin plot). Therefore, in various aspects, computerized tools can be viewed as creating visual dashboards depicting any suitable information about the intelligent deployment of machine learning models.

[0051] Therefore, the various implementations described herein can include computerized tools that can facilitate intelligent training and / or intelligent deployment of machine learning models. As described herein, intelligent training can involve reducing the likelihood of overtraining / overfitting by removing substantially redundant data from the entire set of available training data. Also as described herein, intelligent deployment can involve suggesting and / or allowing the execution of a machine learning model on a set of deployment data only when that set is determined to be sufficiently "similar" to a set of training data. In any case, the computerized tools described herein can improve how machine learning models are trained and / or deployed.

[0052] Various implementations of the innovations in this topic can be used to solve inherently highly technical problems (e.g., to facilitate the intelligent training and / or intelligent deployment of machine learning models) using hardware and / or software. These problems are not abstract and cannot be performed as a set of human mental behaviors. Furthermore, some of the processes performed can be executed by a dedicated computer (e.g., a machine learning model, such as a neural network, support vector machine, or decision tree model) for performing defined tasks related to intelligent training and / or intelligent deployment. For example, such defined tasks may include: accessing a first set of data candidates operatively coupled to a processor for training a machine learning model; computing at least one feature distribution of the first set of data candidates by the device; identifying, by the device, a strict subset of data candidates within the first set of data candidates, wherein at least one feature distribution of the strict subset of data candidates matches at least one feature distribution of the first set of data candidates; and training a machine learning model on the strict subset of data candidates by the device. Such defined tasks may also include: the device accessing a second set of data candidates that can be used for inference of a machine learning model; the device determining whether at least one feature distribution of the second set of data candidates matches at least one feature distribution of the first set of data candidates; and when at least one feature distribution of the second set of data candidates fails to match at least one feature distribution of the first set of data candidates, the device suggesting that the machine learning model cannot be reliably executed on the second set of data candidates.

[0053] These defined tasks are not performed manually by humans. In fact, neither human thought nor human use of pen and paper can electronically receive training data, electronically identify the minimum required subsets of such training data, and electronically train a machine learning model on such minimum required subsets. Furthermore, neither human thought nor human use of pen and paper can electronically receive a set of deployment data, electronically compare the feature distribution of such deployment data with the feature distribution of the training data via statistical hypothesis testing, and electronically suggest whether a machine learning model can reliably perform on the deployment data based on such comparison. Conversely, the various implementations of the innovations in this subject matter inherently and inevitably rely on computer technology and cannot be realized outside of a computing environment (for example, machine learning models such as decision tree models and / or neural networks are inherently computerized constructs that cannot be easily realized by human thought in any way without a computer; thus, computerized tools for identifying minimum-requirement training datasets, training machine learning models on such minimum-requirement training datasets, comparing the feature distribution of the deployed dataset with the feature distribution of such minimum-requirement training datasets, and determining the reliability of machine learning models to be executed on the deployed datasets based on such comparisons are also inherently computerized and cannot be realized in any reasonable, practical, or appropriate way without a computer).

[0054] Furthermore, the various innovative implementations of this topic can integrate the teachings described herein related to the intelligent training and / or intelligent deployment of machine learning models into practical applications. As mentioned above, existing techniques for training machine learning models can lead to overtraining / overfitting, which significantly reduces the generalizability of the machine learning models. Also as mentioned above, existing techniques for deploying machine learning models can lead to unreliable execution (e.g., execution on inappropriate data).

[0055] In stark contrast, the various implementations described herein can address and / or improve upon these significant technical problems. Specifically, the various implementations described herein include computerized tools that can facilitate intelligent training of machine learning models. That is, given all sets of available training data, such computerized tools can identify strict subsets of such sets, where strict subsets contain less / fewer training data than the all sets, but where the feature distribution exhibited by the strict subsets is statistically significantly similar to the feature distribution exhibited by the all sets. Since strict subsets can have less / fewer data than the all sets, machine learning models can be trained on strict subsets with a lower risk of overtraining / overfitting. Furthermore, since the feature distribution of strict subsets can match the feature distribution of the all sets, strict subsets can be considered substantially equivalent to the all sets (e.g., compared to the all sets, strict subsets can be considered missing and / or simply omit substantially redundant information). Therefore, it can be advantageous for computerized tools to train machine learning models on strict subsets rather than on the all sets.

[0056] Furthermore, the computational tools described herein can further facilitate the intelligent deployment of machine learning models. Specifically, given a deployment dataset, the computational tool can compare the feature distribution of the deployment dataset with the feature distribution of the training dataset (e.g., all groups and / or strictly subgroups). If the feature distribution of the deployment dataset matches the feature distribution of the training dataset, the computational tool can determine that the machine learning model can be reliably executed on the deployment dataset. In such cases, the computational tool can execute (and / or may recommend) the machine learning model on the deployment dataset. Conversely, if the feature distribution of the deployment dataset does not match the feature distribution of the training dataset, the computational tool can determine that the machine learning model cannot be reliably executed on the deployment dataset. In such cases, the computational tool can prohibit / prevent (and / or discourage) the execution of the machine learning model on the deployment dataset.

[0057] Therefore, the various implementations described herein can train machine learning models in a way that reduces the probability of overtraining / overfitting, and can deploy machine learning models in a way that improves their reliability. Such computerized tools certainly constitute concrete and tangible technical improvements in the field of machine learning models. Therefore, the various implementations described herein clearly qualify as useful and practical applications of computing.

[0058] Furthermore, various implementations of this invention can control real-world physical devices based on the disclosed teachings. For example, various implementations of this invention can electronically train and / or execute real-world machine learning models (e.g., decision trees, neural networks).

[0059] It should be understood that the accompanying figures and descriptions provide non-limiting examples of innovations in this subject matter and are not necessarily drawn to scale.

[0060] Figure 1 A block diagram of an exemplary, non-limiting system 100, which can facilitate intelligent training and / or intelligent deployment of a machine learning model according to one or more embodiments described herein, is shown. As shown, the intelligent artificial intelligence (AI) training / deployment system 102 can be electronically integrated with a machine learning model 104 and / or a set of training data candidates 106 via any suitable wired and / or wireless electronic connection.

[0061] In various embodiments, the set of training data candidates 106 may include any suitable number of training data candidates. In each aspect, each training data candidate can be considered as any suitable electronic piece of information associated with one or more features, attributes, and / or characteristics (e.g., having any suitable data format and / or any suitable data dimension). Relative to Figure 2 This is shown in more detail.

[0062] Figure 2 An exemplary, non-limiting block diagram 200 is shown of a set of training data candidates according to one or more embodiments described herein. That is, Figure 2 A non-limiting, exemplary implementation of the set of training data candidate 106 is described.

[0063] As shown in the figure, in all aspects, for any suitable positive integer n The 106 candidates in this training data set can include n Data candidates: Training data candidate 1 to training data candidate nAs also shown in the figure, in various cases, each training data candidate can be considered as a sample of data associated with any suitable number of features (e.g., a sample of time series data, a sample of waveform data, a sample of image data, and / or any suitable combination thereof). As those skilled in the art will understand, features can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof, which can describe and / or otherwise characterize the respective training data candidates. As some non-limiting examples, features may be: the mean of the corresponding training data candidates; the median of the corresponding training data candidates; the pattern of the corresponding training data candidates; the maximum value of the corresponding training data candidates; the minimum value of the corresponding training data candidates; the skewness of the corresponding training data candidates; the size and / or cardinality of the corresponding training data candidates; the bin entropy of the corresponding training data candidates; the absolute energy of the corresponding training data candidates; one or more C3 statistics of the corresponding training data candidates; the Friedrich coefficients of the corresponding training data candidates; the centroids of the fast Fourier transforms of the corresponding training data candidates; and / or one or more Dickey-Fuller statistics of the corresponding training data candidates. Those skilled in the art will understand that these are merely non-limiting examples of features that can be extracted from and / or otherwise associated with the training data candidates. Any other suitable features may be implemented as needed in various aspects.

[0064] In any case, for any suitable positive integer m Each training data candidate can be compared with m Each feature is associated with and / or characterized by it. For example, training data candidate 1 may include training feature 1 (1) to training feature 1 (2). m ) and / or related to it. In addition, training data candidates n It can include training features n (1) To training features n ( m ) and / or associated with it. In various cases, for any suitable positive integer i ,in The first candidate for each training data i Each feature can be of the same type, format, and / or dimension as the others. For example, if training feature 1(1) is one or more scalars, vectors, matrices, tensors, and / or strings representing the skewness of candidate 1 in the training data, then the training feature... n (1) can also represent candidates for training data. nThe skewness is one or more scalars, vectors, matrices, tensors, and / or strings. For example, if training feature 1 ( m If ) represents one or more scalars, vectors, matrices, tensors, and / or strings representing the bin entropy of candidate 1 in the training data, then the training features n ( m () can also represent candidates for training data. n The binning entropy is one or more scalars, vectors, matrices, tensors, and / or strings.

[0065] Although not in Figure 2 As explicitly stated, in various cases, each training data candidate in the set of 106 training data candidates may be annotated (e.g., may have corresponding ground truth annotations). However, in other cases, each training data candidate in the set of 106 training data candidates may be unannotated.

[0066] Return to Figure 1 The machine learning model 104 can represent any suitable artificial intelligence architecture as needed. For example, in some cases, the machine learning model 104 can represent a decision tree architecture. In such cases, the machine learning model 104 can include any suitable number and / or arrangement of decision nodes, any suitable decision threshold percentage in such decision nodes, any suitable number and / or arrangement of leaf nodes, and / or any suitable classification threshold percentage in such leaf nodes. As another non-limiting example, the machine learning model 104 can represent a neural network architecture. In such cases, the machine learning model 104 can include any suitable number of layers (e.g., an input layer, one or more hidden layers, an output layer), any suitable number of neurons in the various layers (e.g., different layers may have the same and / or different numbers of neurons), any suitable activation function in the various neurons (e.g., sigmoid, softmax, hyperbolic tangent, modified linear unit) (e.g., different neurons may have the same and / or different activation functions), and / or any suitable inter-neuron connection pattern (e.g., forward connection, skip connection, recurrent connection). As will be understood by those skilled in the art, the machine learning model 104 may exhibit any other suitable architecture as needed, such as a support vector machine architecture, a linear and / or logistic regression architecture, and / or a Naive Bayes architecture.

[0067] In any case, machine learning model 104 can be configured to analyze and / or otherwise receive training data candidates from the set of training data candidates 106 as input. Therefore, it may be desirable to train machine learning model 104 via the set of training data candidates 106. As described herein, intelligent AI training / deployment system 102 can facilitate such training.

[0068] In various embodiments, the intelligent AI training / deployment system 102 may include a processor 108 (e.g., a computer processing unit, microprocessor) and a computer-readable storage device 110 operatively and / or communicatively connected to / coupled to the processor 108. The computer-readable storage device 110 may store computer-executable instructions that, when executed by the processor 108, cause the processor 108 and / or other components of the intelligent AI training / deployment system 102 (e.g., receiver component 112, feature component 114, selection component 116, training component 118, deployment component 120, and / or confidence component 122) to perform one or more actions. In various embodiments, the computer-readable storage device 110 may store computer-executable components (e.g., receiver component 112, feature component 114, selection component 116, training component 118, deployment component 120, and / or confidence component 122), and the processor 108 may execute the computer-executable components.

[0069] In various embodiments, the intelligent AI training / deployment system 102 may include a receiver component 112. In various aspects, the receiver component 112 may electronically receive and / or otherwise electronically access the set of training data candidates 106 and / or machine learning models 104. In various cases, the receiver component 112 may electronically retrieve the set of training data candidates 106 and / or machine learning models 104 from any suitable centralized and / or distributed data structure (not shown). In any case, the receiver component 112 may electronically obtain and / or access the set of training data candidates 106 and / or machine learning models 104, enabling other components of the intelligent AI training / deployment system 102 to electronically interact with the set of training data candidates 106 and / or with the machine learning models 104.

[0070] In various implementations, the intelligent AI training / deployment system 102 may also include a feature component 114. In various aspects, as described herein, the feature component 114 may electronically estimate one or more feature distributions representing the performance of the set of training data candidates 106.

[0071] In various implementations, the intelligent AI training / deployment system 102 may also include a selection component 116. In various cases, as described herein, the selection component 116 may electronically identify strict subgroups of training data candidates within the set of training data candidates 106, wherein one or more feature distributions of the strict subgroups match one or more feature distributions of the set of training data candidates 106.

[0072] In various implementations, the intelligent AI training / deployment system 102 may also include a training component 118. In various cases, as described herein, the training component 118 may electronically train the machine learning model 104 on a strict subgroup rather than on that group of training data candidates 106. In various respects, this may be referred to as “intelligent training” of the machine learning model 104.

[0073] In various implementations, receiver component 112 may receive and / or otherwise access a set of deployed data candidates on which the machine learning model 104 is expected to be executed after training. In various aspects, the intelligent AI training / deployment system 102 may further include deployment component 120, which may electronically determine, as described herein, whether the machine learning model 104 can be reliably executed on the set of deployed data candidates. If so, deployment component 120 may enable and / or allow the machine learning model 104 to execute on the set of deployed data candidates. However, if not, deployment component 120 may disable and / or prevent the machine learning model 104 from executing on the set of deployed data candidates. In various cases, this may be referred to as “intelligent deployment” of the machine learning model 104.

[0074] In various implementations, the intelligent AI training / deployment system 102 may also include a confidence component 122. In various cases, as described herein, the confidence component 122 can electronically estimate a confidence score for any given data candidate on which the machine learning model 104 is to be executed.

[0075] Figure 3 A block diagram of an exemplary, non-limiting system 300 comprising a set of training feature distributions according to one or more embodiments described herein is shown. This system can facilitate intelligent training and / or intelligent deployment of machine learning models. As shown, in some cases, system 300 may include the same components as system 100 and may also include a set of training feature distributions 302.

[0076] In various implementations, feature component 114 can electronically generate the set of training feature distributions 302 based on the set of training data candidates 106. Relative to Figure 4 This is described in more detail.

[0077] Figure 4 An exemplary, non-limiting block diagram 400 is shown illustrating how the set of training feature distributions 302 can be computed based on the set of training data candidates 106 according to one or more embodiments described herein.

[0078] As mentioned above, the candidate 106 of this set of training data can include a total of n There are 10 training data candidates, each of which is related to 10 training data candidates.m Each feature is associated with and / or described by it. Therefore, the set of training feature distributions 302 may include m Training feature distributions (e.g., training feature distribution 1 to training feature distribution 2) m ), where each of these training feature distributions has equal to n The cardinality and / or count.

[0079] For example, all n Each training data candidate can have a first feature. Specifically, training data candidate 1 has training feature 1 (1), and training data candidate... n Training features with the same type, format, and / or dimensions as training feature 1(1) n (1). Therefore, feature 114 can be normalized and / or denormalized in this manner. n The frequency distribution and / or probability density distribution of all first features are tabulated. In some cases, if such tabulated distributions are normal (e.g., Gaussian), then such tabulated distributions can be expressed as all candidates 106 across the training data set. n The mean, variance, and / or median of the first feature calculated for each data candidate. In other cases, if such a tabulated distribution is a gamma distribution, then such a tabulated distribution can be expressed as all of the 106 candidates across the training data set. n The first feature calculated for each data candidate is the shape parameter, rate parameter, and / or median. In any case, feature component 114 can describe how the first feature describes all of the training data candidates 106. n The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated, and such tabulated distribution can be referred to as the training feature distribution1.

[0080] For example, all n Each training data candidate can have a final feature. Specifically, training data candidate 1 has training feature 1 ( m ), and training data candidates n Having the same training feature 1 ( m Training features of the same type, format, and / or dimension n ( m Therefore, feature 114 can be normalized and / or denormalized in this manner. n The frequency distribution and / or probability density distribution of all final features are tabulated. As mentioned above, if such tabulated distributions are normal (e.g., Gaussian) distributions, then such tabulated distributions can be expressed as all candidates 106 across the set of training data. nThe mean, variance, and / or median of the final features calculated for each data candidate. Also, as mentioned above, if such a tabulated distribution is a gamma distribution, then such a tabulated distribution can be expressed as all of the 106 candidates across that set of training data. n The shape parameter, rate parameter, and / or median of the final feature calculated for each data candidate. In any case, feature component 114 can describe how the final feature is calculated across all of the set of training data candidates 106. n The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated, and such tabulated distribution can be referred to as the training feature distribution. m .

[0081] More generally, for any suitable positive integer k ,in ,all n The training data candidate can have the first k Each feature (e.g., training data candidate 1 can have training feature 1) k (not shown), and training data candidates n It can have training features n ( k (not shown). Therefore, feature 114 can be normalized and / or denormalized in this manner. n The first of all k The frequency distribution and / or probability density distribution of each feature are tabulated electronically. Similarly, if such a tabulated distribution is a normal (e.g., Gaussian) distribution, it can be expressed as the mean, variance, and / or median, and / or if such a tabulated distribution is a gamma distribution, it can be expressed as a shape parameter, rate parameter, and / or median. In any case, feature component 114 can describe the frequency distribution and / or probability density distribution of the first feature. k How do these features cross all 106 candidates in this set of training data? n The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated electronically, and such tabulated distribution can be referred to as the training feature distribution. k (Not shown).

[0082] In any case, feature component 114 can electronically estimate and calculate training feature distribution 1 to training feature distribution 2. m And / or tabulate them in other ways, and train feature distribution 1 to train feature distribution m It can be considered as a total of the training feature distribution 302.

[0083] Figure 5A block diagram of an exemplary, non-limiting system 500 comprising a set of selected training data candidates and a set of selected training feature distributions according to one or more embodiments described herein is shown. This system can facilitate intelligent training and / or intelligent deployment of machine learning models. As shown, in some cases, system 500 may include the same components as system 300, and may also include a set of selected training data candidates 502 and a set of selected training feature distributions 504.

[0084] In various embodiments, the selection component 116 can electronically identify and / or select selected training data candidates 502 from the set of training data candidates 106, wherein the selected training data candidates 502 have a smaller cardinality than the set of training data candidates 106. That is, the selected training data candidates 502 may contain fewer data candidates than the set of training data candidates 106. Furthermore, in various cases, the selection component 116 can electronically estimate the selected training feature distribution 504 based on the selected training data candidates 502, and the selected training feature distribution 504 may match the set of training feature distributions 302 respectively. In other words, the selection component 116 can electronically identify strict subgroups of the set of training data candidates 106, wherein such strict subgroups exhibit feature distributions that match the set of training feature distributions 302. Relative to Figures 6 to 8 This is described further.

[0085] Figure 6 An exemplary, non-limiting block diagram 600 is shown representing a set of selected training data candidates according to one or more embodiments described herein. That is, Figure 6 A non-limiting, exemplary implementation of the training data candidate 502 selected by the group is described.

[0086] As shown in the figure, in all aspects, for any suitable positive integer x ,in The selected training data candidate 502 can include x Data candidates: selected training data candidate 1 to selected training data candidate 2. x Furthermore, each selected training data candidate in the set of 502 may have already come from, been selected from, and / or been chosen from the set of 106 training data candidates. Therefore, because each training data candidate in the set of 106 can be compared with… m Each of the 502 selected training data candidates in this group is associated with a feature, so each of the selected training data candidates can also be associated with... m These features are associated. For example, the selected training data candidate 1 may have... mOne feature: the selected training feature 1(1) to the selected training feature 1( m Similarly, the selected training data candidates x It can have m One feature: the selected training features x (1) To the selected training features x ( m ).

[0087] In various implementations, the selection component 116 can electronically generate the selected training feature distribution 504 based on the selected training data candidates 502. Relative to Figure 7 This is described in more detail.

[0088] Figure 7 An exemplary, non-limiting block diagram 700 is shown illustrating how a set of selected training feature distributions 504 can be computed based on a set of selected training data candidates 502, according to one or more embodiments described herein.

[0089] As mentioned above, the 502 training data candidates selected by this group can include a total of x (For example, it is smaller than) n ) selected training data candidates, wherein each selected training data candidate is... m Each feature is associated with and / or described by it. Therefore, the selected training feature distribution 504 may include m The selected training feature distributions (e.g., selected training feature distribution 1 to selected training feature distribution 2) are selected from the selected training feature distributions. m ), where each of the selected training feature distributions in this class has an equal to x (For example, not equal to) n The cardinality and / or count of ).

[0090] For example, all x Each selected training data candidate may have a first feature (e.g., selected training data candidate 1 has selected training feature 1(1), and the selected training data candidate...). x Selected training features having the same type, format, and / or dimension as the selected training feature 1(1) x (1) Therefore, much like feature 114, selection component 116 can be normalized and / or denormalized for this type of feature. x The frequency distribution and / or probability density distribution of all first features are tabulated electronically. As mentioned above, if such tabulated distribution is a normal distribution, it can be expressed as all of the 502 selected training data candidates across that set. xThe mean, variance, and / or median of the first feature calculated for each data candidate. In contrast, if such a tabulated distribution were a gamma distribution, it could be expressed as all of the 502 selected training data candidates across that group. x The shape parameter, rate parameter, and / or median of the first feature calculated for each data candidate. In any case, selection component 116 can select all of the selected training data candidates 502 that describe how the first feature is across the set. x The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated, and such tabulated distribution can be referred to as the selected training feature distribution1.

[0091] For example, all x Each selected training data candidate can have a final feature (e.g., selected training data candidate 1 has selected training feature 1). m ), and the selected training data candidates x Having the same training features as the selected 1 ( m Selected training features of the same type, format, and / or dimension x ( m Therefore, component 116 can be selected in a normalized and / or non-normalized manner for this type of component. x The frequency distribution and / or probability density distribution of all final features are tabulated. Similarly, if such a tabulated distribution is a normal distribution, it can be expressed as the mean, variance, and / or median and / or represented by them; and / or if such a tabulated distribution is a gamma distribution, it can be expressed as a shape parameter, rate parameter, and / or median and / or represented by them. In any case, selection component 116 can describe how the final features are across all of the selected training data candidates 502 in the set. x The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated, and such tabulated distribution can be referred to as the distribution of the selected training features. m .

[0092] More generally, for any suitable positive integer k ,in ,all x The selected training data candidates can have the first... k Each feature (e.g., the selected training data candidate 1 may have the selected training feature 1) k (not shown), and the selected training data candidates. x It can have selected training features x ( k(not shown). Therefore, selection component 116 can be configured in a normalized and / or non-normalized manner for this type of... x The first of all k The frequency distribution and / or probability density distribution of each feature are tabulated electronically. Again, if such a tabulated distribution is a normal distribution, then the tabulated distribution can be expressed as the mean, variance, and / or median and / or represented by them, and / or if such a tabulated distribution is a gamma distribution, then the tabulated distribution can be expressed as a shape parameter, rate parameter, and / or median and / or represented by them. In any case, selection component 116 can describe the frequency distribution and / or probability density distribution of the feature. k How do the features vary across all 502 candidates in the selected training data set? x The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated, and such tabulated distribution can be referred to as the distribution of the selected training features. k (Not shown).

[0093] In any case, the selection component 116 can electronically estimate and calculate the selected training feature distribution 1 to the selected training feature distribution 2. m And / or tabulate them in other ways, and the selected training feature distribution 1 to the selected training feature distribution m It can be collectively regarded as the distribution of training features selected by the group, 504.

[0094] In various implementations, the selection component 116 can electronically select / identify the group of selected training data candidates 502, such that the group of selected training feature distributions 504 are respectively matched with the group of training feature distributions 302. Relative to Figure 8 This was explained in more detail.

[0095] Figure 8 An exemplary, non-limiting block diagram 800 is shown illustrating how the set of training feature distributions 302 can be compared with the set of selected training feature distributions 504, according to one or more embodiments described herein.

[0096] As described above, the training feature distribution 302 may include m Distributions: Training feature distribution 1 to training feature distribution 2 m As mentioned above, the training feature distribution 504 selected by this group can also include... m Distributions: The selected training feature distribution 1 to the selected training feature distribution... m Therefore, the selected training feature distribution 504 can be considered as corresponding to the training feature distribution 302 in a one-to-one manner.

[0097] For example, as described above, training feature distribution 1 can convey how the first feature is across all 106 candidates in the set of training data. n There are 502 candidate data distributions, and the selected training feature distribution 1 can convey how the first feature is distributed across all of the 502 selected training data candidates in that group. x There are several candidate data distributions. Therefore, training feature distribution 1 can be considered as corresponding to the selected training feature distribution 1. In various aspects, selection component 116 can pick / identify the group of selected training data candidates 502 such that training feature distribution 1 matches the selected training feature distribution 1. In other words, it may be that a statistical hypothesis test (e.g., any suitable statistical hypothesis test, such as a z-test, t-test, and / or Kolmogorov-Smirnov test) will conclude that training feature distribution 1 is not statistically significantly different from the selected training feature distribution 1 (e.g., when performing a statistical hypothesis test to compare training feature distribution 1 with the selected training feature distribution 1, such a statistical hypothesis test can produce a p-value greater than 0.05, which can indicate that it cannot be concluded that training feature distribution 1 is statistically significantly different from the selected training feature distribution 1).

[0098] Similarly, as mentioned above, the training feature distribution m Can convey the first m How do these features cross all 106 candidates in this set of training data? n There are candidate data distributions, and the selected training feature distributions. m Can convey the first m How do the features vary across all 502 candidates in the selected training data set? x There are several candidate data distributions. Therefore, the training feature distribution... m They can be viewed as corresponding to the selected training feature distributions, respectively. m As described above, the selection component 116 can select / identify the selected training data candidates 502, thereby enabling the training feature distribution to... m With the selected training feature distribution m A match is found. That is, the situation might be that statistical hypothesis tests (e.g., z-test, t-test, Kolmogorov-Smirnov test) will determine the distribution of training features. m Not related to the selected training feature distribution m Statistically significant differences (e.g., when performing statistical hypothesis testing to distribute training features) m With the selected training feature distribution m When making comparisons, such statistical hypothesis tests can produce p-values ​​greater than 0.05, which may indicate that the distribution of training features cannot be definitively determined. m With the selected training feature distribution m(Statistically significant difference).

[0099] In any case, the selection component 116 can electronically select, pick, and / or identify the selected training data candidates 502 such that the selected training feature distributions 504 match the selected training feature distributions 302. More specifically, the selection component 116 can achieve this by initializing the selected training data candidates 502 to empty, iteratively adding data candidates from the selected training data candidates 106 to the selected training data candidates 502, and checking at each iteration whether the selected training feature distributions 504 still match the selected training feature distributions 302.

[0100] In various implementations, once selection component 116 identifies / selects the set of selected training data candidates 502, training component 118 can electronically train machine learning model 104 on the set of selected training data candidates 502 instead of on the set of training data candidates 106 (e.g., in a supervised manner if each training data candidate is annotated; in an unsupervised manner and / or reinforcement learning manner if each training data candidate is unannotated). This is because the set of selected training data candidates 502 can have a smaller cardinality than the set of training data candidates 106 (e.g., Therefore, compared to training the machine learning model 104 on the selected training data candidate 502, there is a lower likelihood of overtraining / overfitting when training the machine learning model 104 on the selected training data candidate 106 instead. Furthermore, because the selected training feature distribution 504 matches the selected training feature distribution 302, although the cardinality of the selected training data candidate 502 is smaller, it can be considered substantially equivalent to the selected training data candidate 106 for training purposes. In other words, the selected training data candidate 502 can be considered as excluding training data candidates that would be essentially redundant for the purpose of training the machine learning model 104. In other words, if machine learning model 104 is trained on the set of training data candidates 106, it can achieve a certain level of performance (e.g., accuracy, precision, retrieval rate), and if machine learning model 104 is instead trained on the set of selected training data candidates 502, it can achieve substantially the same level of performance, since the set of selected training data candidates 502 and the set of training data candidates 106 have matching feature distributions.

[0101] Therefore, machine learning model 104 can be trained on the selected set of training data candidates 502 with a lower likelihood of overtraining / overfitting while achieving the same and / or comparable level of performance compared to if machine learning model 104 were instead trained on the selected set of training data candidates 106. As described above, this can be referred to as “intelligent training” of machine learning model 104. Relative to Figure 9 This clarifies this type of intelligent training.

[0102] Figure 9 A flowchart is shown of an exemplary, non-limiting computer-implemented method 900 that can facilitate intelligent training of machine learning models according to one or more embodiments described herein. In various cases, an intelligent AI training / deployment system 102 can facilitate the computer-implemented method 900.

[0103] In various implementations, action 902 may include access, by a device operatively coupled to the processor (e.g., via 112), to the entire group (e.g., 106) of data candidates that can be used to train a machine learning model (e.g., 104). In various cases, each data candidate may have... m One characteristic.

[0104] In various aspects, action 904 may include actions calculated by the device (e.g., via 114) and exhibited by the entire group. m A first characteristic distribution (e.g., 302). For example, for any suitable positive integer... i ,in , No. i The first feature distribution can represent the first feature distribution of each data candidate. i How the values ​​of a feature are distributed (e.g., vary) across all groups.

[0105] In various cases, action 906 may include initializing a selected subgroup of data candidates by the device (e.g., via 116) (e.g., 502). In various cases, the selected subgroup may initially be empty (e.g., it may start as an empty group).

[0106] In various aspects, action 908 may include inserting data candidates not yet in the selected subgroup from the entire group into the selected subgroup by the device (e.g., via 116). This may result in an increase (e.g., cardinality) in the size of the selected subgroup) (e.g., increment by 1). In some cases, the inserted data candidates may be randomly selected from the entire group. In other cases, the inserted data candidates may be selected according to any suitable priority order (e.g., selecting irrelevant data candidates first). Furthermore, in some cases, action 908 may include inserting more than one data candidate into the selected subgroup at a time (e.g., in such cases, the size and / or cardinality of the selected subgroup may increment by more than 1).

[0107] In various cases, action 910 may include actions calculated by the device (e.g., via 116) as exhibited by the selected subgroup. m A second characteristic distribution (e.g., 504). For example, for any suitable positive integer... j ,in , No. j The second feature distribution can represent the first data candidate. j How the values ​​of the features are distributed (e.g., vary) across the selected subgroups (e.g., compared to the first feature). j The first feature distribution can represent the first feature distribution of each data candidate. j How the values ​​of a feature are distributed across all groups.

[0108] In various aspects, action 912 may include being determined by the device (e.g., via 116). m Are the second characteristic distributions respectively related to...? m This matches the first characteristic distribution. As mentioned above, this determination can be achieved through the application of any suitable statistical hypothesis test. For example, for any suitable positive integer... k ,in , can be in the k The second characteristic distribution and the first k Perform statistical hypothesis testing among the first characteristic distributions to determine the first... k Is the second characteristic distribution related to the first...? k The first characteristic distributions are statistically significantly different for all positive integers. k ,in If the situation is the first k The second characteristic distribution is not related to the first k If the first characteristic distributions are statistically significantly different, then it can be concluded that... m The second characteristic distributions are indeed respectively related to mThe first feature distribution matches. In this case, the computer-implemented method 900 can proceed to action 914. On the other hand, for at least one positive integer... k ,in If the situation is the first k The second characteristic distribution and the first k If the first characteristic distributions are statistically significantly different, then it can be concluded that... m The second characteristic distribution has not yet been respectively compared with... m The first feature distribution matches. In this case, the computer-implemented method 900 can proceed to action 908.

[0109] Although not in Figure 9 As explicitly stated, action 912 may also include visually presenting the results of any performed statistical hypothesis test on any suitable computer display / screen / monitor (not shown) (e.g., the calculated p-score may be visually presented). In some cases, action 912 may even include visually presenting... m Any second characteristic distribution in the second characteristic distribution and / or m Any first characteristic distribution within a first characteristic distribution (e.g., based on such tabulated distributions, presented graphically as violin plots and / or bar charts).

[0110] As shown in the figure, actions 908 to 912 can be repeated and / or iterated until... m The second characteristic distribution and m The first feature distribution matches. At this point, the selected subgroup can be considered complete (e.g., essentially equivalent to the entire group, but also smaller than the entire group).

[0111] In various aspects, action 914 may include training a machine learning model on a selected subgroup by a device (e.g., via 118). Thus, as described above, the machine learning model can be appropriately trained (e.g., since the selected subgroup may be substantially equivalent to the entire group for training purposes), where the risk of overtraining / overfitting is reduced (e.g., since the selected subgroup may be smaller than the entire group).

[0112] So far, the accompanying figures have primarily depicted various implementation schemes of the innovations of this subject matter, which can facilitate the intelligent training of machine learning model 104. Now, consider how these various implementation schemes can facilitate the intelligent deployment of machine learning model 104.

[0113] Figure 10A block diagram is shown of an exemplary, non-limiting system 1000 according to one or more embodiments described herein, comprising a set of deployed data candidates, a set of deployed feature distributions, and suggestions, which can facilitate intelligent training and / or intelligent deployment of machine learning models. As shown, in some cases, system 1000 may include the same components as system 500, and may also include a set of deployed data candidates 1002, a set of deployed feature distributions 1004, and / or suggestions 1006.

[0114] In various embodiments, receiver component 112 may electronically receive, retrieve, and / or otherwise access a set of deployed data candidates 1002 from any suitable source. In various aspects, the set of deployed data candidates 1002 may include any suitable number of deployed data candidates. In various cases, it may be desirable to execute a machine learning model 104 on this set of deployed data candidates 1002 after training. Therefore, each deployed data candidate may have the same data format and / or data dimension as the training data candidates as described above. That is, a deployed data candidate can be considered as any suitable electronic piece of information associated with one or more features, attributes, and / or characteristics. Relative to Figure 11 This is shown in more detail.

[0115] Figure 11 An exemplary, non-limiting block diagram 1100 is shown of a set of deployed data candidates according to one or more embodiments described herein. That is, Figure 11 Non-limiting, exemplary implementations of the group of deployed data candidates 1002 are described.

[0116] As shown in the figure, in all aspects, for any suitable positive integer y The deployed data candidate 1002 may include y Data candidates: Deployed data candidate 1 to deployed data candidate y Since each training data candidate can be compared with... m Each deployed data candidate can be associated with and / or characterized by a feature, thus each deployed data candidate can be similarly associated with... m Each feature is associated with and / or described by it. For example, deployed data candidate 1 may include deployed feature 1(1) to deployed feature 1(2). m ) and / or associated with it. In addition, deployed data candidates y This may include deployed features y (1) Features of deployment y ( m ) and / or associated with it.

[0117] In various implementations, deployment component 120 can electronically generate the set of deployed feature distributions 1004 based on the set of deployed data candidates 1002. Relative to Figure 12 This is described in more detail.

[0118] Figure 12 An exemplary, non-limiting block diagram 1200 is shown illustrating how a set of deployed feature distributions 1004 can be calculated based on a set of deployed data candidates 1002 according to one or more embodiments described herein.

[0119] As described above, the deployed data candidates 1002 may include a total of y Each deployed data candidate is associated with... m Each feature is associated with and / or described by it. Therefore, the set of deployed feature distributions 1004 may include m A deployed feature distribution (e.g., deployed feature distribution 1 to deployed feature distribution 2) m ), where each of these deployed feature distributions has an equal to y The cardinality and / or count.

[0120] For example, all y A deployed data candidate may have a first feature (e.g., deployed data candidate 1 has deployed feature 1(1), and deployed data candidate...). y Deployment features having the same type, format, and / or dimensions as deployed feature 1(1). y (1) Therefore, much like feature component 114 and selection component 116, deployment component 120 can be normalized and / or denormalized for this type of component. y The frequency distribution and / or probability density distribution of all first features are tabulated electronically. As mentioned above, if such tabulated distribution is a normal (e.g., Gaussian) distribution, it can be expressed as mean, variance, and / or median; otherwise, if such tabulated distribution is a gamma distribution, it can be expressed as shape parameter, rate parameter, and / or median. In any case, deployment component 120 can describe how the first features are distributed across all of the set of deployed data candidates 1002. y The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated electronically, and such tabulated distributions can be referred to as deployed feature distributions1.

[0121] For example, all y Each deployed data candidate can have a final feature (e.g., deployed data candidate 1 has deployed feature 1). m), and deployed data candidates y It has the same characteristics as the deployed 1 ( m Deployment characteristics of the same type, format, and / or dimension. y ( m Therefore, the deployment component 120 can be configured in a normalized and / or non-normalized manner for this type of application. y The frequency distribution and / or probability density distribution of all final features are tabulated electronically. Similarly, if such tabulated distributions are normal (e.g., Gaussian), they can be expressed as mean, variance, and / or median; otherwise, if such tabulated distributions are gamma distributions, they can be expressed as shape parameters, rate parameters, and / or median. In any case, deployment component 120 can describe how the final features are distributed across all of the deployed data candidates 1002. y The distribution of candidate data variations (e.g., regardless of the shape and / or type of the distribution) is tabulated electronically, and such tabulated distributions can be referred to as deployed characteristic distributions. m .

[0122] More generally, for any suitable positive integer k ,in ,all y The selected training data candidates can have the first... k A feature (e.g., deployed data candidate 1 may have deployed feature 1 ( k (not shown), and deployed data candidates y It can have deployed features y ( k (not shown). Therefore, the deployment component 120 can be configured in a normalized and / or non-normalized manner for this type of... y The first of all k The frequency distribution and / or probability density distribution of each feature are tabulated electronically. Again, if such a tabulated distribution is a normal (e.g., Gaussian) distribution, it can be expressed as the mean, variance, and / or median; otherwise, if such a tabulated distribution is a gamma distribution, it can be expressed as a shape parameter, rate parameter, and / or median. In any case, deployment component 120 can electronically tabulate the frequency distribution and / or probability density distribution of the feature. k How do these features cross all 1002 deployed data candidates in this set? y The distribution of candidate variations in deployment data (e.g., regardless of the shape and / or type of the distribution) is tabulated electronically, and such tabulated distributions can be referred to as the characteristic distributions of the deployment. k (Not shown).

[0123] In any case, the deployment component 120 can electronically estimate and calculate the deployed feature distribution 1 to the deployment feature distribution. m And / or tabulate them in other ways, and the deployed feature distribution 1 to the deployed feature distribution m It can be collectively regarded as the feature distribution 1004 deployed in this group.

[0124] In various implementations, deployment component 120 can electronically compare the set of deployed feature distributions 1004 with the set of training feature distributions 302. Based on such comparisons, deployment component 120 can electronically generate recommendations 1006. Relative to Figure 13 This was explained in more detail.

[0125] Figure 13 An exemplary, non-limiting block diagram 1300 is shown illustrating how the set of training feature distributions 302 can be compared with the set of deployed feature distributions 1004 according to one or more embodiments described herein.

[0126] As described above, the training feature distribution 302 may include m Distributions: Training feature distribution 1 to training feature distribution 2 m As also mentioned above, the deployed feature distribution 1004 can similarly include... m Distributions: Deployed feature distribution 1 to deployed feature distribution m Therefore, the deployed feature distribution 1004 can be considered as corresponding to the training feature distribution 302 in a one-to-one manner.

[0127] For example, as described above, training feature distribution 1 can convey how the first feature is across all 106 candidates in the set of training data. n There are 1002 data candidate distributions, and the deployed feature distribution 1 can convey how the first feature is distributed across all of the 1002 deployed data candidates. y There are several candidate data distributions. Therefore, the training feature distribution 1 can be considered to correspond to the deployed feature distribution 1, respectively. In each respect, the deployment component 120 can perform any suitable statistical hypothesis test (e.g., z-test, t-test, Kolmogorov-Smirnov test) to check whether the training feature distribution 1 matches the deployed feature distribution 1. In each respect, the statistical hypothesis test can conclude that the training feature distribution 1 is not statistically significantly different from the deployed feature distribution 1 (e.g., the statistical hypothesis test can produce a p-value greater than 0.05), or the statistical hypothesis test can alternatively conclude that the training feature distribution 1 is statistically significantly different from the deployed feature distribution 1 (e.g., the statistical hypothesis test can produce a p-value less than 0.05).

[0128] Similarly, training feature distribution m Can convey the first m How do these features cross all 106 candidates in this set of training data? n There are several candidate data distributions, and the deployed feature distributions are as follows. m Can convey the first m How do these features cross all 1002 deployed data candidates in this set? y There are several candidate data distributions. Therefore, the training feature distribution... m These can be viewed as corresponding to the deployed feature distributions, respectively. m In various aspects, deployment component 120 can perform any suitable statistical hypothesis test (e.g., z-test, t-test, Kolmogorov-Smirnov test) to examine the distribution of training features. m Is it consistent with the deployed feature distribution? m A match. As mentioned above, statistical hypothesis testing can determine the distribution of training features. m Not with the deployed feature distribution m Statistically significant differences (e.g., a statistical hypothesis test yields a p-value greater than 0.05), or statistical hypothesis tests can alternatively conclude that the training feature distribution is significantly different. m With deployed feature distribution m Statistically significant differences (e.g., statistical hypothesis testing can produce p-values ​​less than 0.05).

[0129] If there exists at least one positive integer c ,in And the first one c The feature distribution of the deployed element is not the same as that of the first element. c If a set of training feature distributions matches the training feature distribution (e.g., is statistically significantly different from it), then deployment component 120 can determine that the set of deployed data candidates 1002 is not sufficiently "similar" to the set of training data candidates 106. Therefore, recommendation 1006 can instruct machine learning model 104 not to reliably execute on the set of deployed data candidates 1002. In such cases, deployment component 120 can electronically transmit recommendation 1006 to any suitable computing device (not shown), electronically present recommendation 1006 on any suitable computer display / screen / monitor (not shown), and / or otherwise disable / prevent machine learning model 104 from executing on the set of deployed data candidates 1002.

[0130] On the other hand, for all positive integers c ,in If the first c The feature distribution of the deployed element is not the same as that of the first element. cIf a set of training feature distributions matches the training feature distribution (e.g., is not statistically significantly different from it), then deployment component 120 can determine that the set of deployed data candidates 1002 is sufficiently "similar" to the set of training data candidates 106. Therefore, recommendation 1006 can instruct machine learning model 104 to reliably execute on the set of deployed data candidates 1002. In such cases, deployment component 120 can electronically transmit recommendation 1006 to any suitable computing device (not shown), electronically present recommendation 1006 on any suitable computer display / screen / monitor (not shown), and / or otherwise enable and / or facilitate the execution of machine learning model 104 on the set of deployed data candidates 1002.

[0131] although Figure 13 The diagram shows that deployment component 120 can compare the set of deployed feature distributions 1004 with the set of training feature distributions 302, but this is merely a non-limiting example. In some cases, deployment component 120 may alternatively compare the set of deployed feature distributions 1004 with the set of selected training feature distributions 504 (e.g., this is possible because the set of training feature distributions 302 may match the set of selected training feature distributions 504).

[0132] Therefore, when the deployed feature distribution 1004 matches the training feature distribution 302 (e.g., and / or matches the selected training feature distribution 504), the machine learning model 104 can be executed on the deployed data candidates 1002; and when the deployed feature distribution 1004 fails to match the training feature distribution 302 (e.g., and / or fails to match the selected training feature distribution 504), the machine learning model 104 cannot be executed on the deployed data candidates 1002. As described above, this can be referred to as the "intelligent deployment" of the machine learning model 104. Relative to... Figure 14 This clarifies the nature of such intelligent deployments.

[0133] Figure 14 A flowchart is shown of an exemplary, non-limiting computer-implemented method 1400 that can facilitate the intelligent deployment of machine learning models according to one or more embodiments described herein. In various cases, an intelligent AI training / deployment system 102 can facilitate the computer-implemented method 1400.

[0134] In various implementations, action 1402 may include access by a device operatively coupled to the processor (e.g., via 112) to a first set of data candidates (e.g., 106 or 502) that has been used to train a machine learning model (e.g., 104). In various cases, for any suitable positive integer... mEach data candidate can have m One characteristic.

[0135] In various aspects, action 1404 may include calculations by the device (e.g., via 114 if 106 is "the first group", or via 116 if 502 is "the first group") of the expression exhibited by the first group. m A first characteristic distribution (e.g., if 106 is the "first group", then it is 302, or if 502 is the "first group", then it is 504). For example, for any suitable positive integer... i ,in , No. i The first feature distribution can represent the first feature distribution of each data candidate. i How the values ​​of a feature vary across the first group of distributions (e.g., how they change).

[0136] In various cases, action 1406 may include access by the device (e.g., via 112) to a second set of data candidates on which a machine learning model is to be performed and / or inferred.

[0137] In various aspects, action 1408 may include actions calculated by the device (e.g., via 120) and exhibited by the second group. m A second characteristic distribution (e.g., 1004). For example, for any suitable positive integer... j ,in , No. j The second feature distribution can represent the first data candidate. j How the values ​​of the first feature vary across the second set of distributions (e.g., while the second feature varies across the third set of distributions) j The first feature distribution can represent the first feature distribution of each data candidate. j How the values ​​of each feature are distributed across the first group.

[0138] In various aspects, action 1410 may include being determined by the device (e.g., via 120). m Are the second characteristic distributions respectively related to...? m This matches the first characteristic distribution. As mentioned above, this determination can be achieved through the application of any suitable statistical hypothesis test. For example, for any suitable positive integer... k ,in , can be in the k The second characteristic distribution and the first k Perform statistical hypothesis testing among the first characteristic distributions to determine the first... k Is the second characteristic distribution related to the first...? k The first characteristic distributions are statistically significantly different for all positive integers. k ,in If the situation is the firstk The second characteristic distribution is not related to the first k If the first characteristic distributions are statistically significantly different, then it can be concluded that... m The second characteristic distributions are indeed respectively related to m The first feature distribution matches. In this case, the computer-implemented method 1400 can proceed to action 1412. On the other hand, for at least one positive integer... k ,in If the situation is the first k The second characteristic distribution and the first k If the first characteristic distributions are statistically significantly different, then it can be concluded that... m The second characteristic distribution is not respectively related to m The first feature distribution matches. In this case, the computer-implemented method 1400 can proceed to action 1414.

[0139] Although not in Figure 14 As explicitly stated, action 1410 may also include visually presenting the results of any performed statistical hypothesis test on any suitable computer display / screen / monitor (not shown) (e.g., the calculated p-score may be visually presented). In some cases, action 1410 may even include visually presenting... m Any second characteristic distribution in the second characteristic distribution and / or m Any first characteristic distribution within a first characteristic distribution (e.g., based on such tabulated distributions, presented graphically as violin plots and / or bar charts).

[0140] In various aspects, action 1412 may include instructing the machine learning model (e.g., via 120) to reliably perform on the second set.

[0141] In various cases, action 1414 may include an instruction by the device (e.g., via 120) that the machine learning model cannot be reliably executed on the second set.

[0142] To date, the accompanying figures have primarily illustrated how various implementations of the innovations of this subject matter can facilitate intelligent deployment relative to a set of deployed data candidates (e.g., 1002). Now, consider how the various implementations described herein can facilitate intelligent deployment relative to individual data candidates.

[0143] Figure 15A block diagram of an exemplary, non-limiting system 1500 including a confidence score according to one or more embodiments described herein is shown. This system can facilitate intelligent training and / or intelligent deployment of machine learning models. As shown, in some cases, system 1500 may include the same components as system 1000, and may also include data candidates 1502 and confidence scores 1504.

[0144] In various embodiments, receiver component 112 can electronically receive, retrieve, and / or access data candidate 1502 from any suitable source. In various aspects, data candidate 1502 may have the same format and / or dimensions as the training data candidate and / or deployed data candidate as described above. In any case, it may be desirable to execute machine learning model 104 on data candidate 1502 (e.g., it may be desirable to feed data candidate 1502 as input to machine learning model 104). In various cases, confidence component 122 may electronically generate confidence score 1504, wherein confidence score 1504 can quantify and / or represent how reliably and / or accurately machine learning model 104 can analyze data candidate 1502. More specifically, confidence score 1504 can convey how well data candidate 1502 fits within the set of training feature distributions 302. If data candidate 1502 fits well within the set of training feature distributions 302, it can be inferred that machine learning model 104 is more reliably capable of analyzing data candidate 1502. In contrast, if data candidate 1502 does not fit well within the set of training feature distributions 302, it can be inferred that machine learning model 104 is less reliable in analyzing data candidate 1502. Relative to Figure 16 This further illustrates the point.

[0145] Figure 16 An exemplary, non-limiting block diagram 1600 is shown illustrating how a confidence score 1504 can be calculated based on data candidates 1502 and / or the set of training feature distributions 302, according to one or more embodiments described herein.

[0146] As shown in the figure, data candidate 1502 can have m Features: Feature 1502(1) to Feature 1502( m ) and / or associated with it. In various respects, the confidence component 122 can electronically quantify such m Each of the features fits well within its corresponding training feature distribution in the set of training feature distributions 302, and the confidence component 122 can electronically estimate the confidence score 1504 based on such quantization.

[0147] For example, in various respects, feature 1502(1) may correspond to training feature distribution 1. Thus, confidence component 122 may compute fit metric 1, where fit metric 1 may be a scalar (e.g., and / or vector, matrix, and / or tensor, in some cases, whose value indicates how many features 1502(1) are outliers according to training feature distribution 1). As a non-limiting example, fit metric 1 may be equal to and / or otherwise based on the reciprocal of the distance (e.g., absolute difference) between feature 1502(1) and the median of training feature distribution 1 (e.g., such that fit metric 1 becomes higher as feature 1502(1) gets closer to the median of training feature distribution 1, and such that fit metric 1 becomes smaller as feature 1502(1) gets further away from the median of training feature distribution 1). As another non-limiting example, the fit metric 1 may be equal to and / or otherwise based on the reciprocal of the distance (e.g., the absolute difference) between feature 1502(1) and the mean of training feature distribution 1 (e.g., such that the fit metric 1 becomes higher as feature 1502(1) gets closer to the mean of training feature distribution 1, and such that the fit metric 1 becomes smaller as feature 1502(1) gets further away from the mean of training feature distribution 1).

[0148] For example, in various situations, feature 1502 ( m This can correspond to the training feature distribution. m Therefore, the confidence component 122 can calculate the fit metric. m Among them, the fit metric m Its value can be an indicator based on the distribution of training features. m How many features are there in 1502? m () is a scalar (e.g., and / or vectors, matrices, and / or tensors, in some cases) of outliers. Similarly, as a non-limiting example, a fit metric... m It can be equal to and / or otherwise based on feature 1502 ( m ) and training feature distribution m The inverse of the distance between the medians (e.g., making the fit metric) m With feature 1502 ( m It becomes closer to the training feature distribution m The median becomes higher, and the fit metric becomes higher. m With feature 1502 ( m It becomes further away from the training feature distribution m (and becomes smaller due to the median). As another non-restrictive example, the fit metric m It can be equal to and / or otherwise based on feature 1502 ( m ) and training feature distribution m The reciprocal of the distance between the means (e.g., making the fit metric)m With feature 1502 ( m It becomes closer to the training feature distribution m The mean becomes higher, and the fit metric becomes higher. m With feature 1502 ( m It becomes further away from the training feature distribution m (and become smaller due to the mean).

[0149] In all aspects, the confidence component 122 can fit metric 1 to fit metric 1. m Clustering, and such clustering can be considered as a confidence score of 1504. For example, in some cases, a confidence score of 1504 can be equal to and / or otherwise based on a fit metric 1 to a fit metric 2. m The linear combination of the data (e.g., sums and / or averages). In any case, the confidence score 1504 can be higher if the features of data candidate 1502 fit the training feature distribution 302 better (e.g., if data candidate 1502 is less of an outlier given the training feature distribution 302), and the confidence score 1504 can be lower if the features of data candidate 1502 fit the training feature distribution 302 poorly (e.g., if data candidate 1502 is more of an outlier given the training feature distribution 302).

[0150] In various cases, the confidence score component 122 can electronically present the confidence score 1504 on any suitable computer display / screen / monitor. In various other cases, the confidence score 122 can electronically transmit the confidence score 1504 to any suitable computing device (not shown) as needed.

[0151] In some cases, if the confidence score 1504 fails to meet any given threshold, the confidence component 122 can prevent and / or prohibit the machine learning model 104 from being executed on the data candidate 1502. Conversely, if the confidence score 1504 meets the given threshold, the confidence component 122 can allow and / or enable the machine learning model 104 to be executed on the data candidate 1502. This can also be viewed as the intelligent deployment of the machine learning model 104.

[0152] although Figure 16 The confidence component 122 is shown to compare data candidate 1502 with the set of training feature distributions 302, but this is merely a non-limiting example. In some cases, the confidence component 122 may alternatively compare data candidate 1502 with the set of selected training feature distributions 504 (e.g., this is possible because the set of training feature distributions 302 may match the set of selected training feature distributions 504).

[0153] Figure 17 A flowchart is shown of an exemplary, non-limiting computer-implemented method 1700 that can facilitate intelligent training and / or intelligent deployment of machine learning models according to one or more embodiments described herein. In various cases, the computer-implemented method 1700 can be facilitated by an intelligent AI training / deployment system 102.

[0154] In various implementations, action 1702 may include access by a device operatively coupled to the processor (e.g., via 112) to a first set of data candidates (e.g., 106) that can be used to train a machine learning model (e.g., 104).

[0155] In various aspects, action 1704 may include calculating at least one feature distribution of the first set of data candidates by the device (e.g., via 114) (e.g., 302).

[0156] In various cases, action 1706 may include identification of a strict subgroup of data candidates by the device (e.g., via 116) and in the first set of data candidates (e.g., 502), wherein at least one feature distribution of the strict subgroup of data candidates (e.g., 504) matches at least one feature distribution of the first set of data candidates.

[0157] In various cases, action 1708 may include training a machine learning model on a strict subgroup of data candidates by a device (e.g., via 118).

[0158] although Figure 17 Not explicitly shown, but identifying a strict subgroup of data candidates may include: initializing the strict subgroup to empty by a device (e.g., via 116); iteratively inserting data candidates from a first set of data candidates into the strict subgroup by a device (e.g., via 116); and performing at least one statistical hypothesis test (e.g., relative to 116) by a device (e.g., via 116) at each iteration. Figure 9 (As described) to determine whether at least one feature distribution of the strict subgroup now matches at least one feature distribution of the first set of data candidates. Although in Figure 17 Not explicitly shown, but the computer-implemented method 1700 may also include: visually presenting the results of at least one statistical hypothesis test (e.g., p-value and / or violin plot) by means of a device (e.g., via 116) and on an electronic display.

[0159] Although Figure 17Not explicitly stated, but the computer-implemented method 1700 may further include: accessing a second set of data candidates (e.g., 1002) for inference that can be used for a machine learning model via a device (e.g., via 112); and determining, via 120, whether at least one feature distribution of the second set of data candidates (e.g., 1004) matches at least one feature distribution of the first set of data candidates. In various cases, determining whether at least one feature distribution of the second set of data candidates matches at least one feature distribution of the first set of data candidates may include performing at least one statistical hypothesis test (e.g., relative to...) by the device (e.g., via 120). Figure 14 (As illustrated). In various aspects, the computer-implemented method 1700 may further include: when at least one feature distribution of the second set of data candidates fails to match at least one feature distribution of the first set of data candidates, the device (e.g., via 120) suggests that the machine learning model cannot reliably perform on the second set of data candidates.

[0160] Although Figure 17 Not explicitly stated, but the computer-implemented method 1700 may further include: accessing data candidates on which a machine learning model can be executed by a device (e.g., via 112) (e.g., 1502); and calculating, by the device (e.g., via 122), a confidence level on which the machine learning model can be executed by the data candidates (e.g., 1504), wherein the confidence level is based on how far at least one feature of the data candidate is from at least one mean or median of the distribution of at least one feature of a first set of data candidates (e.g., relative to...). Figure 16 (As explained).

[0161] Therefore, the various implementations described herein include computerized tools that can facilitate intelligent training and / or intelligent deployment of machine learning models. In various aspects, intelligent training can involve identifying a strict subset of training data (e.g., a minimum required set of training data) from all sets of available training data, where the strict subset exhibits a feature distribution that matches the full set. As illustrated herein, training a machine learning model on such a strict subset can reduce the likelihood of overtraining / overfitting without compromising the quality of learning achieved by the machine learning model. Furthermore, in various cases, intelligent deployment can involve performing and / or suggesting the execution of a machine learning model on data encountered during deployment (e.g., in the field) only if the feature distribution of such data matches the feature distribution of the data on which the machine learning model was trained. Such intelligent deployment can help prevent the machine learning model from being executed on insufficient data within the training domain, which can contribute to improving the reliability of the machine learning model. Computerized tools that can facilitate intelligent training and / or intelligent deployment of machine learning models are, of course, useful and practical applications of computing.

[0162] Although this document primarily treats statistical significance as defined by a p-value of 0.05, this is merely a non-limiting example. In all respects, any suitable p-value threshold can be used to characterize statistical significance as needed.

[0163] In various contexts, machine learning algorithms and / or models may be implemented in any suitable manner to facilitate any suitable aspect described herein. To facilitate some of the machine learning aspects described above in various implementations of the innovations of this subject, the following discussion of artificial intelligence (AI) is considered. Various implementations of the innovations of this subject may employ artificial intelligence to facilitate the automation of one or more features of the innovation. These components may employ various AI-based schemes to perform the various implementations / examples disclosed herein. To provide or contribute to the numerous determinations of the innovation (e.g., determination, detection, inference, computation, prediction, prognosis, estimation, derivation, forecasting, detection, estimation), components of the innovation may examine the whole or a subgroup of data to which they have been granted access and may provide inference about or determine the state of the system and / or environment from a set of observations captured via events and / or data. For example, determination may be used to identify a particular context or action, or a probability distribution of states may be generated. These determinations may be probabilistic; that is, the calculation of the probability distribution of states of interest is based on consideration of data and events. Determination may also refer to techniques used to compose higher-level events from a set of events and / or data.

[0164] This determination can lead to the construction of new events or actions from a set of observed events and / or stored event data, regardless of whether the events are closely related in time, and regardless of whether the events and data come from one or more event and data sources. The components disclosed herein can be combined to perform automated and / or determined actions related to the claimed subject matter using various classification schemes (explicitly trained (e.g., via training data) and implicitly trained (e.g., via observed behavior, preferences, historical information, received external information, etc.)) and / or systems (e.g., support vector machines, neural networks, expert systems, Bayesian confidence networks, fuzzy logic, data fusion engines, etc.). Therefore, classification schemes and / or systems can be used to automatically learn and perform multiple functions, actions, and / or determinations.

[0165] The classifier can take the input attribute vector z = (z1, z2, z3, z4, ... z nThis maps the input to a confidence level that it belongs to a certain category, as shown by f(z) = confidence level (category). Such classification can employ probability- and / or statistical analyses (e.g., analyzing utility and cost considerations) to determine the actions to be automated. Support Vector Machines (SVMs) are an example of classifiers that can be used. SVMs operate by finding a hypersurface in the space of possible inputs, where the hypersurface attempts to separate triggering criteria from non-triggering events. Intuitively, this makes the classification correct for test data that is close to but different from the training data. Other directed and undirected model classification methods include, for example, Naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, and can employ any of the probabilistic classification models that provide different independent patterns. Classification, as used in this paper, also includes statistical regression for developing priority models.

[0166] Those skilled in the art will understand that the disclosure herein describes non-limiting examples of various embodiments of the present invention. For ease of description and / or illustration, the term "each" is used in various parts of the disclosure when discussing various embodiments of the present invention. Those skilled in the art will understand that such use of the term "each" is a non-limiting example. In other words, when the disclosure herein provides a description applicable to "each" of a particular object and / or component, it should be understood that this is a non-limiting example of various embodiments of the present invention, and it should also be understood that in various other embodiments of the present invention, such a description may apply to fewer than "each" of the particular object and / or component.

[0167] To provide additional context for the various implementation schemes described herein Figure 18 The following discussion aims to provide a brief general description of suitable computing environments 1800 for various implementations of the embodiments described herein. While the embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that these embodiments may also be implemented in combination with other program modules and / or as a combination of hardware and software.

[0168] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the methods of this invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each operatively coupled to one or more associated devices.

[0169] The embodiments illustrated in this paper can also be practiced in a distributed computing environment where specific tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0170] Computing devices typically include a variety of media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, wherein the two terms are used interchangeably herein, as described below. A computer-readable storage medium or a machine-readable storage medium can be any available storage medium accessible by a computer, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, a computer-readable storage medium or a machine-readable storage medium can be implemented in conjunction with any method or technique for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0171] Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CDROM), digital versatile disc (DVD), Blu-ray disc (BD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” used herein to describe storage devices, memories, or computer-readable media should be understood to exclude only the propagation of transient signals themselves as a modifier, and do not waive the rights of all standard storage devices, memories, or computer-readable media that do not only propagate transient signals themselves.

[0172] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example, through access requests, queries or other data retrieval protocols, to perform various operations on the information stored on the media.

[0173] Communication media typically contain computer-readable instructions, data structures, program modules, or other structured or unstructured data in a data signal. This data signal can be a modulated data signal, such as a carrier wave or other transmission mechanism, and includes any information transmission or delivery medium. The terms "modulated data signal" or "signal" refer to a signal whose one or more characteristics are set or altered to encode information in one or more signals. By way of example and not limitation, communication media include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media.

[0174] Refer again Figure 18 An exemplary environment 1800 for implementing various embodiments of the aspects described herein includes a computer 1802, which includes a processing unit 1804, system memory 1806, and a system bus 1808. The system bus 1808 connects system components, including but not limited to the system memory 1806, to the processing unit 1804. The processing unit 1804 can be any of a variety of commercially available processors. Dual-microprocessor and other multiprocessor architectures may also be used as the processing unit 1804.

[0175] System bus 1808 can be any of several types of bus structures that can be further interconnected to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. System memory 1806 includes ROM 1810 and RAM 1812. The basic input / output system (BIOS) can be stored in non-volatile memory such as ROM, erasable programmable read-only memory (EPROM), or EEPROM, where the BIOS contains basic routines that facilitate, for example, the transfer of information between components within computer 1802 during startup. RAM 1812 may also include high-speed RAM, such as static RAM for caching data.

[0176] Computer 1802 also includes an internal hard disk drive (HDD) 1814 (e.g., EIDE, SATA), one or more external storage devices 1816 (e.g., floppy disk drive (FDD) 1816, memory stick or flash drive reader, memory card reader, etc.), and a drive 1820, such as a solid-state drive or optical disc drive, which can read from or write to a disk 1822 (e.g., CD-ROM, DVD, BD, etc.). Alternatively, in cases involving solid-state drives, disk 1822 is not included unless separate. Although the internal HDD 1814 is shown as residing within computer 1802, the internal HDD 1814 may also be configured for external use in a suitable infrastructure (not shown). Additionally, although not shown in environment 1800, a solid-state drive (SSD) may be used as a supplement to or alternative to the HDD 1814. HDD 1814, external storage device 1816, and drive 1820 can be connected to system bus 1808 via HDD interface 1824, external storage interface 1826, and drive interface 1828, respectively. Interface 1824 for the specific implementation of the external drive may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are contemplated in the embodiments described herein.

[0177] Drives and their associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, etc. For computer 1802, drives and storage media accommodate the storage of any data in a suitable digital format. Although the above description of computer-readable storage media refers to a corresponding type of storage device, those skilled in the art will understand that other types of computer-readable storage media (whether currently existing or developed in the future) can also be used in the exemplary operating environment, and furthermore, any such storage media may contain computer-executable instructions for performing the methods described herein.

[0178] Multiple program modules may be stored in the drive and RAM 1812, including an operating system 1830, one or more application programs 1832, other program modules 1834, and program data 1836. All or part of the operating system, application programs, modules, and / or data may also be cached in RAM 1812. The systems and methods described herein can be implemented using various commercially available operating systems or combinations of operating systems.

[0179] Computer 1802 may optionally include emulation technology. For example, a hypervisor (not shown) or other intermediary may emulate the hardware environment used for operating system 1830, and the emulated hardware may optionally be different from that of the computer. Figure 18 The hardware is shown. In such implementations, the operating system 1830 may include one of a plurality of virtual machines (VMs) hosted at the computer 1802. Furthermore, the operating system 1830 may provide a runtime environment for the application 1832, such as the Java Runtime Environment or the .NET Framework. A runtime environment is a compatible execution environment that allows the application 1832 to run on any operating system that includes a runtime environment. Similarly, the operating system 1830 may support containers, and the application 1832 may take the form of a container, which is a lightweight, standalone, executable software package that includes, for example, code, runtime, system tools, system libraries, and settings for the application.

[0180] Furthermore, the computer 1802 can be enabled using security modules such as Trusted Processing Modules (TPMs). For example, in the case of a TPM, the boot part is hashed in the next boot part and waits for the result to be matched with a security value before loading the next boot part. This process can occur at any layer of the computer 1802's code execution stack, such as at the application execution level or the operating system (OS) kernel level, thereby achieving security at any code execution level.

[0181] Users can input commands and information into computer 1802 through one or more wired / wireless input devices (e.g., keyboard 1838, touchscreen 1840, and pointing devices such as mouse 1842). Other input devices (not shown) may include microphones, infrared (IR) remote controls, radio frequency (RF) remote controls or other remote controls, joysticks, virtual reality controllers and / or virtual reality headsets, gamepads, styluses, image input devices (e.g., cameras), gesture sensor input devices, visual motion sensor input devices, emotion or face detection devices, biometric input devices (e.g., fingerprint or iris scanners), etc. These and other input devices are typically connected to processing unit 1804 via input device interface 1844, which can be connected to system bus 1808, but these and other input devices can be connected via other interfaces, such as parallel ports, IEEE 1394 serial ports, game ports, USB ports, IR interfaces, BLUETOOTH, etc. ® Interfaces, etc.

[0182] The monitor 1846 or other types of display devices can also be connected to the system bus 1808 via an interface such as the video adapter 1848. In addition to the monitor 1846, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

[0183] Computer 1802 can operate in a networked environment using logical connections to one or more remote computers (such as remote computer 1850) via wired and / or wireless communications. Remote computer 1850 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer-to-peer device, or other public network node, and typically includes many or all of the elements described relative to computer 1802, but for simplicity, only memory / storage device 1852 is shown. The depicted logical connections include wired / wireless connections to a local area network (LAN) 1854 and / or a larger network (e.g., a wide area network (WAN) 1856). Such LAN and WAN networking environments are common in offices and companies and are beneficial for enterprise-wide computer networks, such as intranets, all of which can connect to global communication networks, such as the Internet.

[0184] When used in a LAN networking environment, computer 1802 can connect to local network 1854 via a wired and / or wireless communication network interface or adapter 1858. Adapter 1858 facilitates wired or wireless communication with LAN 1854, which may also include a wireless access point (AP) configured thereon for communicating with adapter 1858 in wireless mode.

[0185] When used in a WAN networking environment, computer 1802 may include modem 1860 or may be connected to a communication server on WAN 1856 via other means (such as via the Internet) for establishing communication over WAN 1856. Modem 1860 (which may be internal or external and wired or wireless) may be connected to system bus 1808 via input device interface 1844. In a networking environment, program modules depicted relative to computer 1802 or parts thereof may be stored in remote memory / storage device 1852. It should be understood that the network connection shown is an example, and other means for establishing communication links between computers may be used.

[0186] When used in a LAN or WAN networking environment, in addition to or as an alternative to the external storage device 1816 described above, computer 1802 can access cloud storage systems or other network-based storage systems, such as, but not limited to, network virtual machines that provide one or more aspects of information storage or processing. Generally, the connection between computer 1802 and the cloud storage system can be established, for example, via adapter 1858 or modem 1860 through LAN 1854 or WAN 1856. When computer 1802 is connected to an associated cloud storage system, external storage interface 1826 can manage the storage provided by the cloud storage system by means of adapter 1858 and / or modem 1860, just like other types of external storage devices. For example, external storage interface 1826 can be configured to provide access to cloud storage sources as if these sources were physically connected to computer 1802.

[0187] Computer 1802 is operable to communicate with any wirelessly positioned device or entity, such as a printer, scanner, desktop and / or portable computer, portable data assistant, communications satellite, or any equipment or location associated with a wirelessly detectable tag (e.g., a self-service machine, newsstand, store shelf, etc.). This may include Wi-Fi and Bluetooth. ® Wireless technology. Therefore, communication can be a predefined structure like a regular network, or simply self-organizing communication between at least two devices.

[0188] Figure 19This is a schematic block diagram of a sample computing environment 1900 with which the disclosed subject matter can interact. The sample computing environment 1900 includes one or more clients 1910. Clients 1910 can be hardware and / or software (e.g., threads, processes, computing devices). The sample computing environment 1900 also includes one or more servers 1930. Servers 1930 can also be hardware and / or software (e.g., threads, processes, computing devices). For example, server 1930 can accommodate threads to perform transformations by employing one or more embodiments described herein. One possible communication between clients 1910 and servers 1930 can be in the form of data packets suitable for transmission between two or more computer processes. The sample computing environment 1900 includes a communication framework 1950 that can be used to facilitate communication between clients 1910 and servers 1930. Clients 1910 are operatively connected to one or more client data repositories 1920, which can be used to store information local to client 1910. Similarly, server 1930 is operatively connected to one or more server data repositories 1940, which can be used to store information locally on server 1930.

[0189] This invention can be a system, method, apparatus, and / or computer program product at any possible level of integrated technical detail. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention. The computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A less complete list of more specific examples of computer-readable storage media may also include: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or protrusions in grooves on which instructions are recorded), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0190] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them for storage in a computer-readable storage medium within the corresponding computing / processing device. The computer-readable program instructions used to perform the operations of this invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). Computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can establish a connection with an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can execute computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of the invention by utilizing state information of the computer-readable program instructions.

[0191] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executable via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture comprising instructions implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operations to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions that execute on the computer, other programmable apparatus or other equipment implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible specific implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may not occur in the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes they may be executed in reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or implements a combination of dedicated hardware and computer instructions.

[0193] Although the subject matter has been described above in the general context of computer executable instructions of a computer program product running on one or more computers, those skilled in the art will recognize that the present disclosure may also be implemented, or may be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, etc., that perform specific tasks and / or implement specific abstract data types. Furthermore, those skilled in the art will understand that the computer implementation of the present invention can be practiced using other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, mainframe computers, and computers, handheld computing devices (e.g., PDAs, telephones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices linked via a communication network. However, some (if not all) aspects of the present disclosure can be practiced on a standalone computer. In a distributed computing environment, program modules may reside in both local and remote memory storage devices.

[0194] As used herein, the terms “component,” “system,” “platform,” “interface,” etc., may refer to and / or include computer-related entities or entities associated with an operator having one or more specific functions. Entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, applications running on a server and servers may both be components. One or more components may reside within a program and / or an execution thread, and components may be located on a single computer and / or distributed across two or more computers. Furthermore, a corresponding component may execute on various computer-readable media on which various data structures are stored. Components may communicate via local and / or remote processes, such as based on signals having one or more data packets (e.g., data from one component that interacts with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems). For example, a component can be a device having specific functions provided by mechanical parts operated by an electrical or electronic circuitry system, which is operated by software or firmware applications executed by a processor. In such cases, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component can be a device that provides specific functions through electronic components rather than mechanical parts, wherein the electronic components may include a processor or other means for executing software or firmware that at least partially gives the electronic components functionality. In one aspect, a component may be emulated via a virtual machine, for example, within a cloud computing system.

[0195] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X adopts A or B" is intended to mean any natural inclusive substitution. That is, if X adopts A; X adopts B; or X adopts both A and B, then "X adopts A or B" is satisfied in any of the foregoing cases. Additionally, unless otherwise specified or clear from the context to be directed to the singular form, the articles "a" and "an" used in this specification and accompanying drawings should generally be understood to mean "one or more." As used herein, the terms "example" and / or "exemplary" are used to indicate what is intended as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. Furthermore, any aspect or design described herein as "example" and / or "exemplary" should not be construed as preferred or advantageous over other aspects or designs, nor does it exclude equivalent exemplary structures and techniques known to those skilled in the art.

[0196] As used herein, the term "processor" can refer substantially to any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreading capabilities; a multi-core processor; a multi-core processor with software multithreading capabilities; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. Additionally, a processor can refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Furthermore, a processor can utilize nanoscale architectures (such as, but not limited to, molecular and quantum dot-based transistors, switches, and gates) to optimize space usage or enhance the performance of user equipment. A processor can also be implemented as a combination of computing processing units. In this disclosure, terms such as "repository," "storage device," "data repository," "data storage device," "database," and substantially any other information storage component related to the operation and function of a component are used to refer to a "memory component," an entity specifically embodied in "memory," or a component that includes memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that can act as external cache memory. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). Furthermore, the memory components disclosed in the systems or computer-implemented methods described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0197] The foregoing description includes only examples of systems and computer-implemented methods. Of course, it is impossible to describe every conceivable combination of components or computer-implemented methods for the purposes of describing this disclosure, but those skilled in the art will recognize that many other combinations and substitutions of this disclosure are possible. Furthermore, regarding the extent to which the terms “comprising,” “having,” “possessing,” etc., are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term “comprising,” as interpreted when “comprising” is used as a transitional word in the claims.

[0198] Various embodiments have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best illustrate the principles of the embodiments, the practical application of or improvement of technology found in the market, or to enable others skilled in the art to understand the disclosed embodiments.

Claims

1. A system for facilitating intelligent training and deployment of machine learning models, the system comprising: A processor that executes computer-executable components stored in a computer-readable storage memory, the computer-executable components comprising: A receiver component that accesses a first set of data candidates that can be used to train a machine learning model; A feature component that calculates at least one feature distribution for the first set of data candidates; A selection component identifies a strict subgroup of data candidates from a first set of data candidates, wherein at least one feature distribution of the strict subgroup of data candidates is determined to match at least one feature distribution of the first set of data candidates according to at least one statistical hypothesis test; and A training component that trains the machine learning model on a strict subgroup of the data candidates. The receiver component accesses a second set of data candidates that can be used for inference by the machine learning model, and the computer-executable component further includes: The deployment component determines whether at least one feature distribution of the second set of data candidates matches the at least one feature distribution of the first set of data candidates by performing the at least one statistical hypothesis test.

2. The system of claim 1, wherein the selection component identifies a strict subgroup of the data candidates by: Initialize the strict subgroup to empty; Iteratively insert data candidates from the first set of data candidates into the strict subgroup; and At each iteration, at least one statistical hypothesis test is performed to determine whether the at least one feature distribution of the strict subgroup now matches the at least one feature distribution of the first set of data candidates.

3. The system of claim 2, wherein the selection component visually presents the results of the at least one statistical hypothesis test on an electronic display.

4. The system of claim 3, wherein the result includes at least one calculated p-value associated with the at least one statistical hypothesis test.

5. The system of claim 1, wherein when the at least one feature distribution of the second set of data candidates fails to match the at least one feature distribution of the first set of data candidates, the deployment component suggests that the machine learning model cannot be reliably executed on the second set of data candidates.

6. The system of claim 1, wherein the receiver component accesses data candidates on which the machine learning model can execute, and wherein the computer-executable component further comprises: A confidence component calculates the confidence level at which the machine learning model can be performed on the data candidates, wherein the confidence level is based on how far at least one feature of the data candidate is from at least one mean or median of the distribution of at least one feature of the first set of data candidates.

7. A computer-implemented method for facilitating intelligent training and deployment of machine learning models, the computer-implemented method comprising: Access to the first set of data candidates that can be used to train a machine learning model is achieved by a device operatively connected to the processor. The device calculates at least one feature distribution of the first set of data candidates; The device identifies a strict subgroup of data candidates in the first group of data candidates, wherein at least one feature distribution of the strict subgroup of data candidates is determined to match the at least one feature distribution of the first group of data candidates according to at least one statistical hypothesis test. as well as The machine learning model is trained by the device on a strict subgroup of the data candidates. The computer-implemented method further includes: The device accesses a second set of data candidates that can be used for inference by the machine learning model; as well as The device determines whether at least one feature distribution of the second set of data candidates matches the at least one feature distribution of the first set of data candidates by performing at least one statistical hypothesis test.

8. The computer-implemented method of claim 7, wherein identifying a strict subgroup of the data candidates comprises: The device initializes the strict subgroup to empty; The device iteratively inserts data candidates from the first group of data candidates into the strict subgroup; as well as The device determines, in each iteration, whether the at least one feature distribution of the strict subgroup now matches the at least one feature distribution of the first set of data candidates by performing at least one statistical hypothesis test.

9. The computer-implemented method according to claim 8, further comprising: The results of the at least one statistical hypothesis test are presented visually by the device and on an electronic display.

10. The computer-implemented method of claim 9, wherein the result includes at least one calculated p-value associated with the at least one statistical hypothesis test.

11. The computer-implemented method according to claim 7, further comprising: When the at least one feature distribution of the second set of data candidates fails to match the at least one feature distribution of the first set of data candidates, the device suggests that the machine learning model cannot be reliably executed on the second set of data candidates.

12. The computer-implemented method according to claim 7, further comprising: The device accesses data candidates on which the machine learning model can be executed. as well as The device calculates the confidence level at which the machine learning model can be performed on the data candidates, wherein the confidence level is based on how far at least one feature of the data candidate is from at least one mean or median of the distribution of at least one feature of the first set of data candidates.

13. A computer program product for facilitating intelligent training and intelligent deployment of machine learning models, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to perform a computer-implemented method according to any one of claims 7 to 12.

Images