Method, computer system, and computer program for training an artificial intelligence model (Early stopping of artificial intelligence model training using control limits)

Abstract

To disclose a method including an iterative training loop for training an artificial intelligence model.SOLUTION: The iterative training loop includes: receiving a current set of training data; dividing the current set of training data into a predetermined number of training data subsets; sequentially training the artificial intelligence model with each of the predetermined number of training data subsets using a training portion and calculating a performance metric using a validation portion; and comparing performance metrics from previous iterations of the iterative training loop with the calculated performance metric to determine whether an improving performance metric condition is met. The method further includes stopping the iterative training loop unless the improving performance metric condition is not met at least once within a predetermined number of previous iterations.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This invention relates to artificial intelligence models, and more particularly to the training of artificial intelligence models. [Background technology]

[0002] Automated machine learning can be a process in which a large number of artificial intelligence models or estimators are trained. The artificial intelligence models can be ranked using performance metrics during AI model validation. This has the advantage of making it easier to find a suitable AI model. Traditional methods of training AI models do not utilize control limits to stop training when a desired performance metric is met, and therefore operate with significantly suboptimal efficiency. Summary of the Invention [Problem to be solved by the invention]

[0003] Traditional methods for training artificial intelligence models operate at significantly suboptimal efficiency because they do not leverage control limits to stop training when a desired performance metric is met. [Means for solving the problem]

[0004] One aspect of the present invention provides a method for training an artificial intelligence model. The method includes an iterative training loop. The iterative training loop includes receiving a current set of training data. The iterative training loop further includes dividing the set of training data into a predetermined number of training data subsets. The training data subsets may also be referred to as k-folds. Each of the predetermined number of training data subsets is divided into a training portion and a validation portion. The training portion of the training data subset is used to further train the artificial intelligence model. The validation portion is used to confirm performance of the artificial intelligence model after being trained by its respective training portion.

[0005] The iterative training loop further comprises using the training portion to successively train the artificial intelligence model with each of the predetermined number of training data subsets. The successive training of the artificial intelligence model comprises using the validation portion to calculate a performance metric for the predetermined number of training data subsets. The iterative training loop further comprises calculating training control limits from an average of the performance metric for the predetermined number of training data subsets and a range of the performance metric for the predetermined number of training data subsets. The average of the performance metric may be calculated, as well as a range, such as between a maximum and minimum value of the performance metric. The average and range of the performance metric may be used to calculate training control limits. The training control limits may be used to evaluate results of subsequent periods of the iterative training loop.

[0006] The iterative training loop further includes comparing the training control limit from the previous iteration of the iterative training loop to the mean of the performance metric across a given number of training datasets, in order to determine whether the improvement performance metric condition is met.

[0007] The method further comprises stopping the iterative training loop unless the improvement performance metric condition has not been met at least once in a predetermined number of previous iterations.

[0008] According to a further aspect of the present invention, the present invention provides a computer-readable storage medium having computer-readable program code embodied therein. The computer-readable program code is configured to implement one embodiment of a method for training an artificial intelligence model.

[0009] According to a further aspect of the present invention, the present invention provides a computer system comprising a processor and a memory for storing machine-executable instructions. The execution of the machine-executable instructions is performed by the processor implementing a method according to one embodiment.

[0010] According to a further aspect of the present invention, there is provided a computer program product comprising a computer readable storage medium having computer readable program code embodied therein, the computer readable program code being configured to implement a method according to one embodiment.

[0011] According to a further aspect of the present invention, there is provided a computer program product, the computer program product comprising a computer-readable storage medium having stored thereon an artificial intelligence model trained according to an embodiment of the method.

[0012] According to a further aspect of the present invention, there is provided memory stored data for access by an application program running on a data processing system, the application program comprising an artificial intelligence model trained according to one embodiment of the method. [Brief explanation of the drawings]

[0013] Hereinafter, embodiments of the present invention will be described in more detail, by reference only to the following drawings.

[0014] [Figure 1] An example of a computer system according to at least one embodiment of the present invention is shown. [Figure 2] Figure 1 shows an exemplary computing environment in which the computer systems are connected according to at least one embodiment of the present invention.

[0015] [Figure 3] Further examples of computer systems according to at least one embodiment of the present invention are shown.

[0016] [Figure 4] A method for operating the computer system shown in Figure 3, according to at least one embodiment of the present invention, is shown.

[0017] [Figure 5]An example of a method according to at least one embodiment of the present invention is shown. DETAILED DESCRIPTION OF THE INVENTION

[0018] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are chosen to best explain the principles of the embodiments, practical applications of or technical improvements to the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0019] The embodiment may be beneficial because it may provide a means to reduce the computational load associated with training an artificial intelligence model. If the model is not fully running, training can be terminated early, saving both time and computational resources.

[0020] Another benefit may be that adding a condition that stopping occurs only if the improved performance metric condition is not met at least once within a predetermined number of previous iterations may provide a higher quality or more accurate artificial intelligence model. This may occur because the training of an artificial intelligence model may temporarily plateau. If the artificial intelligence model has not improved within a predetermined number of previous iterations, then the quality of the artificial intelligence model after training may not be sufficient, and it may be safely stopped or discarded.

[0021] The set of training data may be different or distinct from the current set of training data from a previous iteration of the iterative training loop, i.e., a new or different current set of training data is used each time the training loop is iterated, meaning that the artificial intelligence model is further trained when the iterative training loop is executed multiple times.

[0022] Each time the artificial intelligence model is validated by the validation section, it provides data points regarding performance metrics. Since the training data is divided into a predetermined number of training data subsets, the number of times the metrics are calculated is also that predetermined number.

[0023] Another potential advantage is that training control limits can be used to define whether the quality of subsequent iterations is improving or not.

[0024] In some examples, the first time an iterative training loop is run, there may be no training control limits. In this case, the training control limits may be ignored. In subsequent iterations, the training control limits provide an objective measure of whether the training of the artificial intelligence model is improving with each iterative training loop.

[0025] In another embodiment, the method further comprises simultaneously training multiple artificial intelligence models using the artificial intelligence model training method. For example, there may be a system in which a large number of different types of models or parameters within the artificial intelligence models are changed to identify or search for the best artificial intelligence model. The method may allow selected models from the multiple artificial intelligence models that are not performing well to be stopped before being fully trained. This may have the significant advantage of reducing the amount of time to train the remaining multiple artificial intelligence models, as well as freeing up additional computational resources to accelerate the process.

[0026] In other embodiments, the method further comprises selecting a global control limit for each iteration of the iterative training loop. The global control limit is selected by selecting a control limit from the iterative training loop that shows the best performance metric of multiple artificial intelligence models. When multiple artificial intelligence models are trained and a training control limit is selected for each of these models, the training control limit that shows the best performance metric is selected. In the next iteration, all of the multiple artificial intelligence models are compared against this global control limit. The iterative training loop further comprises replacing the training control limit from the previous iteration of the iterative training loop for each of the multiple artificial intelligence models with the global control limit from the previous iteration of the iterative training loop. This embodiment may have the benefit of having competition among the multiple artificial intelligence models. If a particular artificial intelligence model does not show performance improvement compared to the global control limit for a predetermined number of iterations, then training of this model is stopped. This may be beneficial in that it reduces the amount of artificial intelligence models that need to be trained.

[0027] In another embodiment, the training control limits for each of the multiple artificial intelligence models during each iteration of the iterative training loop are calculated separately. In this embodiment, a global control limit is not used. In this case, the multiple artificial intelligence models are evaluated against their own individual performance.

[0028] In other embodiments, an increasing value of the performance metric indicates better artificial intelligence model performance. The control limit is calculated by adding the range of the performance metric multiplied by a constant to the mean of the performance metric. The constant may be identified or selected, for example, using a constant table from a statistical process control chart.

[0029] In another embodiment, a decreasing value of the performance metric indicates better artificial intelligence model performance. The control limit is calculated by subtracting the range of the performance metric multiplied by a constant from the mean of the performance metric. The constant may be selected from a table of constants from a statistical process control chart, as above. In this embodiment, the control limit may be a lower control limit.

[0030] In other embodiments, the control limits are statistical process control limits. These may be, for example, upper or lower control limits with respect to an x-bar graph from statistical process control.

[0031] In another embodiment, the performance metric is accuracy rate.

[0032] In another embodiment, the performance metric is precision.

[0033] In another embodiment, the performance metric is recall.

[0034] In another embodiment, the performance metric is an area under the curve metric.

[0035] In another embodiment, the performance metric is the true positive rate.

[0036] In another embodiment, the performance metric is the true negative rate.

[0037] In another embodiment, the performance metric is the false positive rate.

[0038] In another embodiment, the performance metric is the sum of the true positive rate and the true negative rate, both divided by the overall sample size.

[0039] In other embodiments, the performance metric is the mean squared error.

[0040] In another embodiment, the performance metric is the F1 score.

[0041] In another embodiment, the artificial intelligence model is a convolutional neural network. Training of the artificial intelligence model using the training data subset is performed using a deep learning algorithm. This embodiment may be beneficial because it may provide an efficient means of evaluating different topologies and training techniques for convolutional neural networks. For example, hyperparameters or other values ​​may be varied for a particular neural network, which may be trained and then evaluated using an iterative training loop. Models that perform poorly may be discontinued early.

[0042] In another embodiment, the artificial intelligence model is a neural network.

[0043] In another embodiment, the artificial intelligence model is a classifier neural network.

[0044] In another embodiment, the artificial intelligence model is a convolutional neural network.

[0045] In another embodiment, the artificial intelligence model is a Bayesian neural network.

[0046] In another embodiment, the artificial intelligence model is a Bayesian network.

[0047] In another embodiment, the artificial intelligence model is a Bayesian network.

[0048] In another embodiment, the artificial intelligence model is a naive Bayes classifier.

[0049] In another embodiment, the artificial intelligence model is a belief network.

[0050] In another embodiment, the artificial intelligence model is a decision-making network.

[0051] In another embodiment, the artificial intelligence model is a decision tree.

[0052] In another embodiment, the artificial intelligence model is a support vector machine.

[0053] In another embodiment, the artificial intelligence model is a regression analysis.

[0054] In another embodiment, the artificial intelligence model is a genetic algorithm.

[0055] In another embodiment, the training of multiple artificial intelligence models is performed such that each of the models can be freely selected from those mentioned above.

[0056] In other embodiments, execution of the instructions causes the processor to train multiple artificial intelligence models simultaneously using the method for training an artificial intelligence model.

[0057] In another embodiment, execution of the instructions causes the processor to select a global control limit for each iteration of the iterative training loop. Execution of the instructions causes the processor to select the control limit from the iterative training loop that exhibits the best performance metric. The iterative training loop further comprises replacing the training control limit from a previous iteration of the iterative training loop for each of the plurality of artificial intelligence models with the previous global control limit.

[0058] In another embodiment, training control limits for each of the multiple artificial intelligence models during the iterations of the iterative training loop are calculated separately.

[0059] Embodiments of the present invention may be implemented using a computing device, which may also be referred to as a computer system, client, or server. Referring now to FIG. 1, a schematic of an example computer system is shown. Computer system 10 is only one example of a suitable computer system and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the present invention described herein. Regardless, computer system 10 is capable of implementing or performing any of the functions described above, or a combination thereof.

[0060] There are numerous other general-purpose or special-purpose computing system environments or configurations that the computer system / server 12 operates in within computer system 10. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed computing environments that include any of the above systems or devices.

[0061] The computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0062] 1, computer system / server 12 in computer system 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components, including system memory 28, to processor 16. Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0063] Computer system / server 12 typically includes a variety of computer system-readable media, which may be any available media that can be accessed by computer system / server 12, including both volatile and nonvolatile media, removable and non-removable media.

[0064] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 or cache memory 32, or a combination thereof. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such an example, each may be connected to the bus 18 by one or more data media interfaces. As further shown and described below, the memory 28 may include at least one program product including a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0065] A program / utility 40 including a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28, by way of example and not limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions or methodologies, or combinations thereof, of embodiments of the present invention as described herein.

[0066] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc., one or more devices that allow a user to interact with the computer system / server 12, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 22. Furthermore, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. While not shown, it should be understood that other hardware or software components, or combinations thereof, may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0067] A computer system such as computer system 10 shown in Figure 1 can be used to perform the operations disclosed herein, such as training an artificial intelligence module. Such a computer system may be a stand-alone computer without a network connection that can receive data to be processed, such as a training data set for training the artificial intelligence module, via a local interface. However, such operations can equally be performed using a computer system connected to a network, such as a communications network or a computing network, or a combination thereof.

[0068] 2 illustrates a computing environment in which a computer system, such as computer system 10, is connected to a network 200, for example, using a network adapter 20. Network 200 may be a communications network, such as, but not limited to, the Internet, a local area network (LAN), or a wireless network, such as a mobile communications network. Network 200 may also comprise a computing network, such as a cloud computing network. Computer system 10 may receive data to be processed from network 200, such as a training dataset for training an artificial intelligence model, or may provide computing results, such as a trained artificial intelligence module after being trained using the training dataset, to another computing device connected to computer system 10 via network 200, or a combination thereof.

[0069] Computer system 10 may, in whole or in part, perform the operations described herein in response to requests received via network 200. In particular, computer system 10 may perform such operations in distributed computing together with one or more further computer systems that may be connected to computer system 10 via network 200. For that purpose, computing system 10 or any further involved computer systems or combination thereof may use network 200 to access further computing resources, such as dedicated or shared memory.

[0070] Figure 3 shows an example of computer system 10. The computer system 10 shown in Figure 3 is one representation of the computer system shown in Figure 1. The processing unit 16 is shown connected to the network adapter 20, the I / O interface 22, and the memory 28. The memory 28 is intended to represent various types of memory that may be accessible to the processing unit 16.

[0071] Memory 28 is shown as containing machine-executable instructions 300. These machine-executable instructions 300 enable the processing unit 16 to perform various computational and data processing tasks. For example, this could include calculating various means and metrics, as well as calculating control limits for training an artificial intelligence model and implementing various algorithms. Memory 28 is further shown as containing an artificial intelligence model 302 to be trained. Memory 28 is further shown as containing the current set of training data 304. During each iteration loop, the current set of training data 304 is replaced with new training data. This could, for example, be provided to memory 28 in each loop, or retrieved from an archive or database containing the training data.

[0072] Memory 28 is further shown to contain sets of training data formed by dividing the current set of training data 304 into a predetermined number of groups or sets of training data. These may also be referred to as k-folds. Memory 28 is further shown to contain performance metrics calculated for each set of training data 306 during the training process. Memory 28 is further shown to contain the average of all performance metrics 308 calculated for a particular loop and a particular artificial intelligence model 302. Memory 28 is further shown to contain a range 312 of all performance metrics for a particular loop calculated for a particular artificial intelligence model 302. Memory contains a training control limit 314 calculated from the average of the performance metrics 310 for the loop and the range 312 of performance metrics for the loop. The current performance metric 308 is compared to the training control element from the previous iteration 316, which is also shown to be stored in Memory 28.

[0073] Figure 4 shows a flowchart illustrating how to operate the computer system 10 shown in Figure 3. The method begins with step 400, labeled as Start. The method then proceeds to step 402, where the current set of training data 304 is received. During each iteration of the loop, the current set of training data 304 is replaced. Next, the method proceeds to step 404, where the current set of training data 304 is divided into a predetermined number of training data subsets 306. A predetermined set of training data 306 may also be referred to as k-folds. Next, in step 406, the artificial intelligence model 302 is trained by successively training the artificial intelligence model 302 using the training portions of the predetermined number of training data subsets.

[0074] After each training portion is used, the corresponding validation portion is used to calculate a performance metric 308. Next, in step 408, the method proceeds by determining whether the model is fully trained. If the model is, in fact, fully trained, the method proceeds to step 410, where the method ends. If the model is not fully trained, the method proceeds to step 412. In step 412, training control limits 314 are calculated from the average of the performance metric 310 and the range 312 of the performance metric. Then, in step 414, training control limits 316 from the previous iteration are compared to the average of the performance metric 310 for the particular loop and particular model. This is used to determine whether an improvement performance metric condition is met. The improvement performance metric is a measure of whether the model's performance continues to improve with further training. Next, in step 416, the method then proceeds to determine whether the improvement performance metric condition was met at least once in a predetermined number of previous iterations. If the improvement performance metric has been met at least once in a predetermined number of previous iterations, then further training will improve the artificial intelligence model, and the method returns to step 402 for further training, where the current set of training data is replaced with a new current set of training data.

[0075] If the improvement performance metric has not been met at least once in the predetermined number of previous iterations, the artificial intelligence model will no longer be improved by further training, and model training is discontinued. When model training is discontinued, the method returns to step 410, and training of the artificial intelligence model 302 is stopped.

[0076] The method of FIG. 4 may be modified in various ways. In one example, the method is applied to simultaneously train multiple artificial intelligence models. This can also take a variety of different forms. In one example, the artificial intelligence models are all trained simultaneously but evaluated independently. Training control limits are calculated separately for each artificial intelligence model, which are used to determine whether training of that artificial intelligence model should be stopped. In another example, training of artificial intelligence models may be conducted competitively. For example, training control limits are calculated separately for each artificial intelligence model in each loop, and the best training control limit is then selected as the global control limit. The data is then compared to the global training control limit from the previous iteration instead of the training control limit to measure the artificial intelligence model's substantial improvement over the best model of the previous iteration of the loop. If the model is not able to improve over the best model of the previous iteration for a predetermined number of times, then training of that particular artificial intelligence model is stopped. This allows for very quick elimination of artificial intelligence models that are subpar relative to the others. This can lead to significant computational savings.

[0077] FIG. 5 shows a flowchart illustrating an alternative method for automated training of an artificial intelligence model. In some cases, various portions of the method shown in FIG. 5 may be interchanged with portions of the method shown in FIG. 4. Step 500 illustrates a model selection step. This includes providing training data 502. The training data is also collectively referred to as a training data allocation. Step 500 also includes providing various estimators 504. The estimators referred to herein are individual artificial intelligence models. One of the allocations is the equivalent of the current set of training data 304. During each loop, the model then proceeds to a cross-validation process 508. In cross-validation, training and validation are performed, and the allocation or current set of training data 304 is divided into k-folds of data. In some cases, the k-folds of data may be the equivalent of a set of training data 306. During training, performance metrics 308 are calculated for each of the k-folds 306. In step 510, k-fold statistics are calculated. The k-fold statistics may include all of the averages of the performance metrics 308, as well as the ranges 312 for the performance metrics 308. The k-fold statistics may then be used to calculate control limits or training control limits 314. The control limits 314 are maintained from loop to loop and are used in step 514 to compare the training control limits from the previous iteration 316 with the averages of the performance metrics 310. Box 512 represents a function used to test whether the averages of the performance metrics 310 have shown improved performance within the most recent predetermined number of loops. This is then used to provide a stop or continue decision, per step 416.

[0078] At each iteration, an estimator (artificial intelligence model) from the set of estimators 504 (multiple artificial intelligence models) is evaluated on a subsample of data points, also referred to as an assignment. The assignment is the current set of training data. The evaluation is performed using a cross-validation technique. The cross-validation technique is based on dividing the data into k-folds (training data subsets), and then training and evaluating a model performed on each of the k-folds.

[0079] For each assignment to each estimator, k metrics are calculated (k-fold). Each of the k metrics may correspond to a machine learning metric that describes the performance of the trained model (using performance metrics), such as accuracy, precision, recall, etc.

[0080] The process ends when the entire dataset (all assignments) has been examined to provide a recommendation (the most promising estimator) for the next stage (in simplification).

[0081] The system described below enables the provision of recommendations much faster, based on an analysis of ongoing assignment and cross-validation results.

[0082] The system may comprise one or more of the following stages:

[0083] 1. Calculate the control limits: the upper control limit (UCL) and lower control limit (LCL) for the folds_score (metric) per allocation (i) and estimator (e).

[0084] UCL = mean(folds_score) + R*A2

[0085] LCL = mean(folds_score) - R*A2

[0086] Here, R = max(folds_score) - min(folds_score), and A2 is obtained from the statistical process control chart.

[0087] 2. Next, calculate the average value for the following assignment fold scores.

[0088] 3. Analyze the relationship between the average value of the assignment (i + j), the estimator (e), and the assignment (i). Here, j = 1,.., len(allocations) IF UCL > average value > LCL. This means that there is no increase or decrease in the metric value calculated for the (i + j) assignment and the estimator (e).

[0089] IF average value > UCL. This means that there is an improvement in the metric value (model performance / quality) for the (i + j) assignment and the estimator (e).

[0090] IF average value < LCL. This means that there is a deterioration in the model performance for the (i + j) assignment and the estimator (e).

[0091] 4. Repeat the analysis until j = iters_no_improvement. For example, iters_no_improvement = 3 means that the next 3 assignments will be analyzed. If there is no improvement in the next j (in this example, 3) assignments regarding the estimator (e), further investigation of the next assignment for this specific estimator (e) will be stopped. Another variation is to skip the next assignment if the condition that the average metric value has not improved is met.

[0092] Regarding such data, the behavior of the system can be simulated. As an example, the LGBM estimator example was investigated. LightGBM (LGBM) is a gradient boosting framework based on the decision tree algorithm.

[0093] The steps are as follows.

[0094] 1. For a specific allocation (500), cross-validate the estimator performance and calculate the UCL. In this example, the upper control limit was used as the training control limit.

[0095] 2. For the next assignment (1000), the cross-validation results calculate the UCL, average the current mean value with the UCL from the previous assignment (500), and perform a comparative analysis. In this example, the improvement detected was 0.71%.

[0096] 3. For the next assignment (1500), the cross-validation results calculate the UCL, average the current mean value with the UCL from the previous assignment (1000), and perform a comparative analysis. In this example, no improvement was detected. This is the first iteration with no improvement.

[0097] 4. For the next allocation (2000), the cross-validation results calculate the UCL, average the current mean value with the UCL from the previous allocation (1500), and perform a comparative analysis. In this example, no improvement was detected. This is the second iteration with no improvement.

[0098] 5. For the next assignment (2500), the cross-validation results calculate the UCL, average the current mean value with the UCL from the previous assignment (2000), and perform a comparative analysis. In this example, no improvement was detected. This is the third iteration with no improvement.

[0099] 6. Check the number of iterations without improvement. If it is equal to the no_iter_improvement parameter (aka window size, or a given number of previous iterations), stop the search process for the estimator: LGBM.

[0100] 7. Repeat for other estimators (artificial intelligence models).

[0101] The present invention may be a system, a method, or a computer program product, or a combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0102] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction-execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0103] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or they may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the individual computing / processing device.

[0104] Computer-readable program instructions for carrying out operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuitry by utilizing state information of the computer-readable program instructions to implement aspects of the present invention.

[0105] Aspects of the present invention will be described herein with reference to flowcharts or block diagrams or combinations thereof of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram or combination thereof, and any combination of blocks in a flowchart or block diagram or combination thereof, can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device that produces a machine, so that instructions executed via the computer's processor or other programmable data processing device generate means for implementing functions / operations specified in blocks or multiple blocks of a flowchart or block diagram or a combination thereof. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, a programmable data processing device, or other device or combination thereof to function in a specific manner, and as a result, the computer-readable storage medium containing the instructions comprises a product containing instructions that implement modes of functions / operations specified in blocks or multiple blocks of a flowchart or block diagram or a combination thereof.

[0107] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to generate a computer implementation process by causing a series of operational steps to be performed on the computer, other programmable device, or other device, resulting in instructions being executed on the computer, other programmable device, or other device that implement functions / operations specified by blocks or multiple blocks in a flowchart or block diagram or a combination thereof.

[0108] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the drawing. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functionality they involve. It should also be noted that each block in a block diagram or flowchart, or a combination thereof, and the combination of blocks in a block diagram or flowchart, or a combination thereof, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or executes a combination of dedicated hardware and computer instructions.

[0109] Various examples may be described by one or more of the following features, in some cases in the numbered sections below.

[0110] Clause 1: A method for training an artificial intelligence model, the method comprising an iterative training loop, the iterative training loop receiving a current set of training data, the current set of training data being different from the current set of training data from a previous iteration of the iterative training loop; and dividing the current set of training data into a predetermined number of training data subsets, each of the predetermined number of training data subsets being divided into a training portion and a validation portion; and successively training the artificial intelligence model using each of the predetermined number of training data subsets using the training portion, wherein successively training the artificial intelligence model includes: calculating a performance metric for a predetermined number of training data subsets; calculating training control limits from an average of the performance metric for the predetermined number of training data subsets and a range of the performance metric for the predetermined number of training data subsets; and comparing the training control limits from a previous iteration of the iterative training loop to the average of the performance metric for the predetermined number of training data subsets to determine whether an improved performance metric condition is met, the method further comprising stopping the iterative training loop unless the improved performance metric condition has been met at least once in a predetermined number of previous iterations.

[0111] Clause 2: The method of clause 1, further comprising simultaneously training multiple artificial intelligence models using the method for training artificial intelligence models.

[0112] Clause 3: The method of clause 2, wherein the training of the plurality of artificial intelligence models is each selected from any one of the following: a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayesian network, a naive Bayes classifier, a belief network or decision network, a decision tree, a support vector machine, a regression analysis, and a genetic algorithm.

[0113] Clause 4: The method of clause 1 or 2, wherein the method further comprises selecting a global control limit for each iteration of the iterative training loop, the global control limit being selected by selecting the control limit from the iterative training loop that exhibits a best performance metric for the plurality of artificial intelligence models, and the iterative training loop further comprises replacing the training control limit from a previous iteration of the iterative training loop for each of the plurality of artificial intelligence models with a previous global control limit from a previous iteration of the iterative training loop.

[0114] Clause 5: The method of clause 2 or 3, wherein the training control limits for each of the plurality of artificial intelligence models during each iteration of the iterative training loop are calculated separately.

[0115] Clause 6: The method of any one of clauses 1 to 5, wherein an increasing value of the performance metric indicates better artificial intelligence model performance, and wherein the control limit is calculated as the mean of the performance metric plus a constant multiple of the range of the performance metric.

[0116] Clause 7: The method of any one of clauses 1 to 6, wherein a decreasing value of the performance metric indicates better artificial intelligence model performance, and wherein the control limit is calculated as the mean of the performance metric minus a constant multiple of the range of the performance metric.

[0117] Clause 8: The method of any one of clauses 1 to 7, wherein the control limits are statistical process control limits.

[0118] Section 9: The performance metrics are selected from any one of the following: accuracy, precision, recall, area under the curve metric, true positive rate, true negative rate, false positive rate, the sum of the true positive rate and the true negative rate, both divided by the total sample size, mean squared error, and F1 score, as described in any one of Sections 1 to 8.

[0119] Clause 10: The method of any one of clauses 1 to 9, wherein the artificial intelligence model is a convolutional neural network and the training of the artificial intelligence model is performed using a deep learning algorithm.

[0120] Section 11: The artificial intelligence model is one of the following: neural networks, classifier neural networks, convolutional neural networks, Bayesian neural networks, Bayesian networks, Bayesian networks, simple Bayesian classifiers, belief networks, or decision networks, decision trees, support vector machines, regression analysis, and genetic algorithms, according to any one of Sections 1 to 9.

[0121] Clause 12: A computer program product comprising a computer-readable storage medium having computer-readable program code embodied thereon, said computer-readable program code being configured to implement the method of any one of clauses 1 to 11.

[0122] Clause 13: A computer system, comprising: a processor configured to control the computer system; and a memory storing machine-executable instructions, execution of which causes the processor to perform an iterative training loop for training an artificial intelligence model, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from the current set of training data from a previous iteration of the iterative training loop; and dividing the set of training data into a predetermined number of training data subsets, each of the predetermined number of training data subsets being divided into a training portion and a validation portion; and successively training the artificial intelligence model using the training portion of each of the predetermined number of training data subsets, and calculating training control limits from an average of the performance metric for each of the predetermined number of training data subsets and a range of the performance metric for each of the predetermined number of training data subsets, and comparing the training control limits from a previous iteration of the iterative training loop to the average of the performance metric for each of the predetermined number of training data subsets to determine whether an improved performance metric condition exists, the method further comprising: stopping the iterative training loop unless the improved performance metric condition has not existed at least once in a predetermined number of previous iterations.

[0123] Section 14: The computer system according to Section 13, wherein the execution of the instruction causes the processor to train multiple artificial intelligence models simultaneously using the method for training the artificial intelligence models.

[0124] Section 15: The computer system according to Section 14, wherein the execution of the instruction causes the processor to select a global control limit for each iteration of the iterative training loop, the execution of the instruction causes the processor to select the control limit from the iterative training loop that exhibits the best performance metric, and the iterative training loop further replaces the training control limit from the immediately preceding iteration of the iterative training loop for each of the plurality of artificial intelligence models with the immediately preceding global control limit.

[0125] Section 16: The training control limits for each of the multiple artificial intelligence models during each iteration of the iterative training loop are calculated separately in the computer system described in Section 14.

[0126] Section 17: The computer system described in any one of Sections 13 to 16, wherein the performance metrics are selected from one of the following: accuracy, precision, recall, area under the curve metric, true positive rate, true negative rate, false positive rate, the sum of the true positive rate and the true negative rate, both divided by the total sample size, mean squared error, and F1 score.

[0127] Section 18: The artificial intelligence model is one of the following computer systems described in any one of Sections 13 to 17: neural networks, classifier neural networks, convolutional neural networks, Bayesian neural networks, Bayesian networks, Bayesian networks, simple Bayesian classifiers, belief networks, or decision networks, decision trees, support vector machines, regression analysis, and genetic algorithms.

[0128] Section 19: A computer program product comprising a computer-readable storage medium storing an artificial intelligence model trained by the method described in any one of Sections 1 to 11.

[0129] Clause 20: Memory for storing data for access by an application program running on a data processing system, the memory comprising an artificial intelligence model trained by the method of any one of clauses 1 to 11.

[0130] While various embodiments of the present invention have been described for illustrative purposes, they are not intended to be exhaustive or limit the scope to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described. The terminology used herein has been chosen to best describe the principles, practical applications, or technical improvements to the technologies available on the market of the embodiments, or to enable other those skilled in the art to understand the embodiments disclosed herein. According to this specification, the following items are also disclosed. [Item 1] 1. A method for training an artificial intelligence model, the method comprising an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training the artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; comparing the performance metric from a previous iteration of the iterative training loop to the calculated performance metric for the training data subset to determine whether an improved performance condition is met; stopping the iterative training loop in response to determining that an improved performance metric condition has been met at least once in a predetermined number of previous iterations. How to train an artificial intelligence model. [Item 2] Item 10. The method of claim 1, further comprising training multiple artificial intelligence models simultaneously using the iterative training loop. [Item 3] Item 1. The method according to item 1, wherein the control limits are statistical process control limits. [Item 4] Item 1. The method of item 1, wherein the performance metric is selected from the group consisting of accuracy rate, precision rate, recall rate, area under the curve metric, true positive rate, true negative rate, false positive rate, the sum of the true positive rate and the true negative rate both divided by the total sample size, mean squared error, and F1 score. [Item 5] Item 10. The method of item 1, wherein the artificial intelligence model is a convolutional neural network, and the training of the artificial intelligence model is performed using a deep learning algorithm. [Item 6] 6. The method of any one of items 1 to 5, wherein the artificial intelligence model is selected from the group consisting of a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayesian network, a naive Bayes classifier, a belief network or decision-making network, a decision tree, a support vector machine, a regression analysis, and a genetic algorithm. [Item 7] Item 3. The method of item 2, wherein each of the plurality of artificial intelligence models is selected from the group consisting of a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayesian network, a naive Bayes classifier, a belief network or a decision network, a decision tree, a support vector machine, a regression analysis, and a genetic algorithm. [Item 8] 1. A computer system comprising: a processor configured to control the computer system; The system comprises a memory storing machine-executable instructions, wherein the execution of the instructions causes the processor to perform an iterative training loop, and the iterative training loop is Receiving the current set of training data, wherein the current set of training data is different from the training data from the previous iteration of the iterative training loop. dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training an artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; To determine whether the improvement performance conditions are met, the performance metric from the previous iteration of the iterative training loop is compared with the calculated performance metric for the training data subset. The system includes stopping the iterative training loop in response to the determination that the improvement performance metric condition has been met at least once in a predetermined number of previous iterations, Computer system. [Item 9] 9. The computer system of claim 8, wherein the iterative training loop further comprises simultaneously training multiple artificial intelligence models using the iterative training loop. [Item 10] 10. The computer system according to item 8 or 9, wherein the control limits are statistical process control limits. [Item 11] 1. A method for training an artificial intelligence model, the method comprising an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training the artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; calculating current training control limits based at least in part on the calculated performance metrics for the training data subset; comparing training control limits from a previous iteration of the iterative training loop with the calculated current training control limits to determine if an improvement performance metric condition is met; stopping the iterative training loop in response to determining that the improved performance metric condition has been met at least once in a predetermined number of previous iterations. How to train an artificial intelligence model. [Item 12] Item 12. The method of item 11, further comprising training multiple artificial intelligence models simultaneously using the iterative training loop. [Item 13] Item 13. The method of item 12, wherein the method further comprises selecting a global control limit for each iteration of the iterative training loop, the global control limit being selected by selecting a control limit from the iterative training loop that exhibits the best performance metric of the plurality of artificial intelligence models, and the iterative training loop further comprises replacing the training control limit from a previous iteration of the iterative training loop for each of the plurality of artificial intelligence models with a previous global control limit from a previous iteration of the iterative training loop. [Item 14] The training control limits for each of the multiple artificial intelligence models during each iteration of the iterative training loop are calculated separately, as described in item 12. [Item 15] The method according to item 11, wherein an increasing value of the performance metric indicates better artificial intelligence model performance, and the control limit is calculated by adding a constant multiplier of the range of the performance metric to the average of the performance metric. [Item 16] The method according to item 11, wherein a decreasing value of the performance metric indicates better artificial intelligence model performance, and the control limit is calculated by subtracting a constant multiple of the range of the performance metric from the average of the performance metric. [Item 17] The control limits are the statistical process control limits, as described in item 11. [Item 18] 18. The method of any one of items 11 to 17, wherein comparing training control limits from a previous iteration of the iterative training loop to an average of the performance metric for the training data subset to determine whether an improved performance metric condition is met comprises comparing training control limits from a immediately preceding iteration of the iterative training loop to the average of the performance metric for the training data subset. [Item 19] A computer program comprising program instructions, wherein the program instructions are: Receiving the current set of training data, wherein the current set of training data is different from the training data from the previous iteration of the iterative training loop. dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training an artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; Calculating the current training control limit based at least in part on the calculated performance metrics for the training data subset, To determine whether the improvement performance metric conditions are met, the training control limit from the previous iteration of the iterative training loop is compared with the calculated current training control limit, The system includes an instruction to stop the iterative training loop in response to a determination that the improvement performance metric condition has been met at least once in a predetermined number of previous iterations, Computer program. [Item 20] 1. A computer system comprising: a processor configured to control the computer system; The system comprises a memory storing machine-executable instructions, wherein the execution of the instructions causes the processor to perform an iterative training loop, and the iterative training loop is Receiving the current set of training data, wherein the current set of training data is different from the training data from the previous iteration of the iterative training loop. dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training an artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; Calculating the current training control limit based at least in part on the calculated performance metrics for the training data subset, To determine whether the improvement performance metric conditions are met, the training control limit from the previous iteration of the iterative training loop is compared with the calculated current training control limit, The method includes stopping the iterative training loop in response to the determination that the improvement performance metric condition has been met at least once in a predetermined number of previous iterations, Computer system. [Item 21] The execution of the aforementioned instructions is performed by the computer system described in item 20, which causes the processor to train multiple artificial intelligence models simultaneously using a training management loop. [Item 22] The computer system according to item 21, wherein the execution of the instruction causes the processor to select a global control limit for each iteration of the iterative training loop, the execution of the instruction causes the processor to select a control limit from the iterative training loop that exhibits the best performance metric, and the iterative training loop further replaces the training control limit from the immediately preceding iteration of the iterative training loop for each of the plurality of artificial intelligence models with the immediately preceding global control limit. [Item 23] 22. The computer system of claim 21, wherein the training control limits for each of the plurality of artificial intelligence models during each iteration of the iterative training loop are calculated separately. [Item 24] The computer system described in item 21, wherein the performance metrics are selected from the group consisting of accuracy, precision, recall, area under the curve metric, true positive rate, true negative rate, false positive rate, the sum of the true positive rate and the true negative rate obtained by dividing both by the total sample size, mean squared error, and F1 score. [Item 25] The artificial intelligence model is a computer system as described in any one of items 21 to 24, selected from the group consisting of neural networks, classifier neural networks, convolutional neural networks, Bayesian neural networks, Bayesian networks, Bayesian networks, simple Bayesian classifiers, belief networks, or decision networks, decision trees, support vector machines, regression analysis, and genetic algorithms.

Claims

1. 1. A method for training an artificial intelligence model, the method comprising an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into a plurality of training data subsets, each training data subset having a respective training portion and a respective validation portion; sequentially training the artificial intelligence model with each training data subset of the plurality of training data subsets using the respective training portions, wherein sequentially training the artificial intelligence model includes calculating a performance metric for each training data subset of the plurality of training data subsets using the respective validation portions after training with the respective training portions; comparing average-based control limits from previous iterations of the iterative training loop to the averages of the calculated performance metrics for the plurality of training data subsets to determine whether an improved performance metric condition is met; stopping the iterative training loop in response to determining that the improved performance metric condition has not been met at least once in a predetermined number of previous iterations. How computers train artificial intelligence models.

2. The method of claim 1 , further comprising training multiple artificial intelligence models simultaneously using the iterative training loop.

3. The method described in claim 1, wherein the control limits of the performance metric are statistical process control limits.

4. 2. The method of claim 1, wherein the performance metric is selected from the group consisting of accuracy, precision, recall, area under the curve metric, true positive rate, true negative rate, false positive rate, the sum of the true positive rate and the true negative rate both divided by the overall sample size, mean squared error, and F1 score.

5. 10. The method of claim 1, wherein the artificial intelligence model is a convolutional neural network, and the training of the artificial intelligence model is performed using a deep learning algorithm.

6. 6. The method of any one of claims 1 to 5, wherein the artificial intelligence model is selected from the group consisting of a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayesian network, a naive Bayes classifier, a belief network or decision-making network, a decision tree, a support vector machine, a regression analysis, and a genetic algorithm.

7. 3. The method of claim 2, wherein each of the plurality of artificial intelligence models is selected from the group consisting of a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayesian network, a naive Bayes classifier, a belief network or decision making network, a decision tree, a support vector machine, a regression analysis, and a genetic algorithm.

8. 1. A computer system comprising: a processor configured to control the computer system; and a memory storing machine-executable instructions, execution of which causes the processor to perform an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into a plurality of training data subsets, each training data subset having a respective training portion and a respective validation portion; Successively training an artificial intelligence model with each training data subset of the plurality of training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for each training data subset of the plurality of training data subsets using the respective validation portions after training with the respective training portions; comparing average-based control limits from a previous iteration of the iterative training loop to the averages of the calculated performance metrics for the plurality of training data subsets to determine whether an improved performance metric condition is met; stopping the iterative training loop in response to determining that the improved performance metric condition has not been met at least once in a predetermined number of previous iterations. Computer system.

9. 10. The computer system of claim 8, wherein the iterative training loop further comprises using the iterative training loop to simultaneously train multiple artificial intelligence models.

10. The computer system of claim 8 or 9, wherein the control limits of the performance metric are statistical process control limits.

11. 1. A method for training an artificial intelligence model, the method comprising an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training the artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; calculating current training control limits based at least in part on the calculated performance metrics for the training data subset; comparing training control limits from a previous iteration of the iterative training loop with the calculated current training control limits to determine if an improvement performance metric condition is met; stopping the iterative training loop in response to determining that the improved performance metric condition has not been met at least once in a predetermined number of previous iterations. How computers train artificial intelligence models.

12. The method of claim 11 , further comprising training multiple artificial intelligence models simultaneously using the iterative training loop.

13. 13. The method of claim 12, further comprising selecting a global control limit for each iteration of the iterative training loop, wherein the global control limit is selected by selecting a control limit from the iterative training loop that exhibits the best performance metric of the plurality of artificial intelligence models, and wherein the iterative training loop further comprises replacing the training control limit from a previous iteration of the iterative training loop for each of the plurality of artificial intelligence models with a previous global control limit from a previous iteration of the iterative training loop.

14. The method of claim 12 , wherein the training control limits for each of the plurality of artificial intelligence models during each iteration of the iterative training loop are calculated separately.

15. 12. The method of claim 11, wherein an increasing value of the performance metric indicates better artificial intelligence model performance, and control limits are calculated as the mean of the performance metric plus a constant multiple of the range of the performance metric.

16. 12. The method of claim 11, wherein a decreasing value of the performance metric indicates better artificial intelligence model performance, and a control limit is calculated as the mean of the performance metric minus a constant multiple of the range of the performance metric.

17. The method of claim 11, wherein the training control limits of the performance metric are statistical process control limits.

18. 18. The method of claim 11, wherein comparing training control limits from a previous iteration of the iterative training loop to an average of the performance metric for the training data subset to determine whether an improved performance metric condition is met comprises comparing training control limits from a immediately preceding iteration of the iterative training loop to the average of the performance metric for the training data subset.

19. 1. A computer program comprising program instructions, the program instructions comprising instructions for performing an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training an artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; calculating current training control limits based at least in part on the calculated performance metrics for the training data subset; comparing training control limits from a previous iteration of the iterative training loop with the calculated current training control limits to determine if an improvement performance metric condition is met; and stopping the iterative training loop in response to determining that the improved performance metric condition has not been met at least once in a predetermined number of previous iterations. Computer program.

20. 1. A computer system comprising: a processor configured to control the computer system; and a memory storing machine-executable instructions, execution of which causes the processor to perform an iterative training loop, the iterative training loop comprising: receiving a current set of training data, the current set of training data being different from training data from a previous iteration of the iterative training loop; dividing the current set of training data into training data subsets having respective training portions and respective validation portions; successively training an artificial intelligence model on the training data subsets using the respective training portions, wherein successively training the artificial intelligence model includes calculating a performance metric for the training data subsets using the respective validation portions; calculating current training control limits based at least in part on the calculated performance metrics for the training data subset; comparing training control limits from a previous iteration of the iterative training loop with the calculated current training control limits to determine if an improvement performance metric condition is met; stopping the iterative training loop in response to determining that the improved performance metric condition has not been met at least once in a predetermined number of previous iterations. Computer system.

21. 21. The computer system of claim 20, wherein execution of the instructions causes the processor to train multiple artificial intelligence models simultaneously using a training management loop.

22. 22. The computer system of claim 21, wherein execution of the instructions causes the processor to select a global control limit for each iteration of the iterative training loop, wherein execution of the instructions causes the processor to select a control limit from the iterative training loop that exhibits a best performance metric, wherein the iterative training loop further comprises replacing the training control limit from a previous iteration of the iterative training loop for each of the plurality of artificial intelligence models with the previous global control limit.

23. 22. The computer system of claim 21, wherein the training control limits for each of the plurality of artificial intelligence models during each iteration of the iterative training loop are calculated separately.

24. 22. The computer system of claim 21, wherein the performance metric is selected from the group consisting of accuracy, precision, recall, area under the curve metric, true positive rate, true negative rate, false positive rate, the sum of the true positive rate and the true negative rate both divided by the overall sample size, mean squared error, and F1 score.

25. 25. The computer system of any one of claims 21 to 24, wherein the artificial intelligence model is selected from the group consisting of a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayesian network, a naive Bayes classifier, a belief network or decision making network, a decision tree, a support vector machine, a regression analysis, and a genetic algorithm.

Images