Methods and media for gradient-based automatic tuning of machine learning and deep learning models

Abstract

The present disclosure relates to gradient-based automatic tuning of machine learning and deep learning models. In this, a horizontal scalable technique efficiently configures a machine learning algorithm to achieve optimal accuracy without knowing the input. In an embodiment, for each specific hyperparameter, and for each epoch, a computer processes the specific hyperparameter. An epoch explores one hyperparameter based on a hyperparameter tuple. From each tuple, a respective score is computed. A tuple contains different combinations of values, each value being contained in a value range of a different hyperparameter. All values of a tuple that belong to a specific hyperparameter are different. All values of a tuple that belong to other hyperparameters remain constant. A value range of a specific hyperparameter is narrowed down according to an intersection of a first line based on scores and a second line based on scores. The machine learning algorithm is optimally configured according to the repeatedly narrowed down hyperparameter value ranges. The configured algorithm is invoked to obtain a result.

Description

[0001] This application is a divisional application of invention patent application 201880062156.9 entitled "Gradient-based automatic adjustment of machine learning and deep learning models" filed on September 28, 2018. Technical Field

[0002] This disclosure relates to machine learning. This paper presents a horizontally scalable technique for the efficient configuration of machine learning algorithms to achieve optimal accuracy without requiring known input. Background Technology

[0003] While machine learning (ML) applications are becoming ubiquitous, data science expertise may remain scarce. For example, relying on data scientists may not be sustainable or scalable for large enterprise vendors and their enterprise customers.

[0004] ML modeling involves:

[0005] Select an appropriate model.

[0006] Tuning a model to a given dataset is the most time-consuming and ad-hoc task, which relies heavily on the expertise of data scientists.

[0007] Model tuning is laborious for the following reasons. It may involve a large number of hyperparameters, especially for deep neural network models with many parameters, such as multilayer perceptrons (MLPs) or convolutional neural networks (CNNs).

[0008] Perhaps more importantly, hyperparameters can have a wide range of possible values. The choice of actual values ​​can significantly impact model performance, such as measured prediction accuracy. Unfortunately, the selection of hyperparameter values ​​is often specific and largely depends on the data scientist's experience.

[0009] Training models is typically time-consuming, and tuning requires retraining the model and evaluating it several times in each tuning session. To address these challenges, industry seeks to improve efficiency by automatically tuning ML models for a given dataset and a given ML algorithm, such as a neural network or support vector machine (SVM). A widely used approach is exhaustive grid search, which tries all possible combinations of values ​​for each hyperparameter and uses the optimal combination.

[0010] Another popular approach is Bayesian optimization. Typically, Bayesian optimization assigns a prior distribution to each hyperparameter. The optimization process initially selects hyperparameter values ​​based on this distribution and measures the model's performance. Subsequently, based on the observed performance of the tried hyperparameter values, the next optimal value to explore is selected based on Bayesian probability, hoping to obtain the optimal model performance, which may be represented by an objective function. Unfortunately, many of these hyperparameter tuning (or automatic tuning) methods suffer from one or more of the following drawbacks:

[0011] Bayesian optimization requires known input. Typically, detailed parameter distribution information is needed to achieve better model performance.

[0012] Bayesian optimization is inherently sequential and difficult to parallelize in a meaningful way.

[0013] Bayesian optimization applications are domain-specific. The objective function is often not generalizable and does not easily extend to large datasets, machine learning, and deep learning models.

[0014] Other alternative methods for optimizing hyperparameters include random search, where a given statistical distribution for each hyperparameter is used to search over a selected number of maximum trials. Both Bayesian optimization and random search require detailed input and generally do not guarantee finding the optimal combination of values. Despite state-of-the-art techniques, even with state-of-the-art solutions such as the GpyOpt Python library, Bayesian optimization remains slow and potentially inaccurate. Attached Figure Description

[0015] In the attached diagram:

[0016] Figure 1 This is a block diagram depicting an example computer in an embodiment, which repeatedly narrows the range of hyperparameter values ​​for a machine learning algorithm to find the optimal hyperparameter values ​​for configuring the algorithm for training.

[0017] Figure 2 This is a block diagram depicting an example computer in the embodiment, which uses gradient search space reduction to estimate the optimal point on a curve of hyperparameter score values;

[0018] Figure 3 This is a flowchart depicting an example process in an embodiment, which is used to repeatedly narrow down the range of hyperparameter values ​​of a machine learning algorithm to find the optimal hyperparameter values ​​for configuring the algorithm for training.

[0019] Figure 4 This is a block diagram depicting an example computer in the embodiment, which combines epoch management with dynamic tracking of the best scores and values ​​for numerical and / or categorical hyperparameters;

[0020] Figure 5 This is a flowchart describing an example process for optimizing categorical hyperparameters in an embodiment;

[0021] Figure 6 This is a block diagram depicting an example computer with horizontal scaling in the embodiments;

[0022] Figure 7 This is a block diagram illustrating a computer system on which embodiments of the present invention can be implemented;

[0023] Figure 8 It is a block diagram of a basic software system that can be used to control the operation of a computing system. Detailed Implementation

[0024] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be clear that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0025] This document describes the embodiments based on the following summary:

[0026] 1.0 General Overview

[0027] 2.0 Example Computer

[0028] 2.1 Machine Learning Algorithms

[0029] 2.2 Meta-model

[0030] 2.3 Hyperparameters

[0031] 2.4 Value Range

[0032] 2.5 Hyperparameter tuple

[0033] 2.6 Configuration and Training

[0034] 2.7 Exploration

[0035] 2.8 Reduction

[0036] 2.9 Era

[0037] 2.10 Tuple Generation

[0038] 2.11 Score

[0039] 3.0 Gradient search space reduction

[0040] 3.1 Curve Sampling

[0041] 3.2 Gradient Detection

[0042] 3.3 Peak Approximation

[0043] 3.4 Narrowing the scope

[0044] 3.5 Next Epoch

[0045] 3.6 Results

[0046] 4.0 Example Optimization Processing

[0047] 4.1 Initialization

[0048] 4.2 Exploration

[0049] 4.3 Era Ordering

[0050] 4.4 The Climb After the Era Sequence

[0051] 4.5 Final Configuration

[0052] 5.0 Tracking Best Scores

[0053] 5.1 Optimal Combination

[0054] 5.2 Categorical Hyperparameters

[0055] 5.3 Revisit

[0056] 6.0 Category-based optimization

[0057] 6.1 Category-based combinations

[0058] 7.0 Horizontal scaling

[0059] 7.1 Multiprocessing

[0060] 7.2 Concurrency

[0061] 7.3 Load Balancing

[0062] 7.4 Synchronization

[0063] 7.5 Cross-validation

[0064] 8.0 Hardware Overview

[0065] 9.0 Software Overview

[0066] 10.0 Cloud Computing

[0067] General Overview

[0068] This paper presents a horizontally scalable technique for efficiently configuring machine learning algorithms to achieve optimal accuracy without requiring knowledge of the input. In an embodiment, for each specific hyperparameter that is not a categorical hyperparameter, and for each epoch in the epoch sequence, the computer processes that specific hyperparameter as follows: An epoch explores a hyperparameter based on a computer-generated tuple of hyperparameters. For each tuple, a score is calculated based on that tuple.

[0069] A hyperparameter tuple contains distinct combinations of values, each falling within the current value range of a different hyperparameter. All values ​​belonging to a specific hyperparameter tuple are distinct. All values ​​belonging to any other hyperparameter tuple remain constant throughout the epoch, such as being the best value for that other hyperparameter so far. The computer narrows down the current value range of a specific hyperparameter based on the intersection of a first fractional-based line and a second fractional-based line.

[0070] Machine learning algorithms are optimally configured based on a repeatedly narrowed range of hyperparameter values. The configured algorithm is then invoked to obtain results, such as identifying patterns or classifications among multiple possible patterns.

[0071] 2.0 Sample Computer

[0072] Figure 1 This is a block diagram depicting an example computer 100 in an embodiment. Computer 100 repeatedly narrows the range of hyperparameter values ​​for a machine learning algorithm to discover the optimal hyperparameter values ​​for configuring the algorithm for training. Computer 100 can be one or more computers, such as embedded computers, personal computers, rack servers such as blade servers, mainframes, virtual machines, or any computing device that uses temporary memory during numerical and symbolic processing.

[0073] 2.1 Machine Learning Algorithms

[0074] Computer 100 includes or accesses specifications (not shown) for configurable and trainable machine learning algorithms that can perform analyses such as classification, regression, clustering, or anomaly detection. For example, the machine learning algorithm could be a support vector machine (SVM), an artificial neural network (ANN), a decision tree, or a random forest.

[0075] 2.3 Hyperparameters

[0076] The configurable features of a machine learning algorithm are called hyperparameters. For example, a machine learning algorithm may have hyperparameters 121-123, which may be specific to a particular machine learning algorithm.

[0077] If the machine learning algorithm is a support vector machine, the hyperparameters typically include C and γ (gamma). If the machine learning algorithm is a neural network, the hyperparameters may include features such as the number of layers and / or the number of neurons per layer.

[0078] 2.4 Value Range

[0079] Each hyperparameter has its own range of natural values. For example, the count of neural layers can be an integer ranging from three to ten.

[0080] Depending on the hyperparameter, the possible range of values ​​can be numeric, such as integers or real numbers. Some ranges may be categorical and limited to discrete symbolic (i.e., non-numeric) values, such as labels or text without a relative order. Categorical hyperparameters may require special handling, as discussed later in this paper.

[0081] 2.5 Hyperparameter Tuples

[0082] Based on combinations of hyperparameter values, machine learning algorithms can have many configuration alternatives. Each combination of values ​​can be represented as a configuration tuple. For example, each different configuration of a machine learning algorithm is based on tuples of different values ​​of hyperparameters 121-123, such as tuples 151-152.

[0083] Each of the hyperparameters 121-123 can logically be a separate axis / dimension in a multidimensional hyperspace (not shown). Each different configuration of the machine learning algorithm is represented by a different tuple (such as 151) that corresponds to a different point in that hyperspace.

[0084] 2.6 Configuration and Training

[0085] Computer 100 can use a tuple to configure and train a machine learning algorithm, and then repeat the process with slightly different tuples to test for possible improvements in algorithm performance. By repeatedly refining the tuple and retraining, the optimal tuple configuration can be discovered.

[0086] Training machine learning algorithms is computationally very expensive, which can be exacerbated by the amount of raw data in the training dataset (not shown), the amount of hyperparameters (such as 121-123), or the amount of values ​​in the range of hyperparameters (such as 121).

[0087] 2.7 Exploration

[0088] Some of the hyperparameters 121-123 may be continuous variables, meaning that even a tiny subrange of such hyperparameters could contain an infinite number of points. Due to this difficult combination, computer 100 should not use many or most points in the hyperspace to configure machine learning algorithms.

[0089] Computer 100 can efficiently explore and narrow down the hyperspace to find which configurational alternatives for machine learning algorithms will produce the best training results. During operation, exploring the hyperspace requires intelligently refining existing tuples to generate new ones.

[0090] 2.8 reduced

[0091] Shrinking the hyperspace requires using new tuples to configure the algorithm, train the configured algorithm, and record the training performance metrics(s), such as the accuracy achieved by the trained algorithm or the time spent during training. Shrinking requires reducing the range of possible values ​​for each hyperparameter.

[0092] For example, hyperparameters 121-123 can have current value ranges of 131A, 132A, and 133A, respectively, which are narrowed to achieve corresponding current value ranges of 131B, 132B, and 133B. Within a current value range (such as 131A), depending on the type of hyperparameter 121, there can be many or an unlimited number of values, such as 141-143.

[0093] Tuples (such as 151-152) have one value for each hyperparameter 121-123. For example, tuple 152 has values ​​143, 146, and 148 that fit within their respective current value ranges 131A, 132A, and 133A.

[0094] 2.9 Era

[0095] Most possible tuples are never generated, and those that are generated are not all generated simultaneously. For example, tuples such as 151-152 were generated and tested during epoch 111, while other tuples (not shown) were generated during other epochs such as 112.

[0096] An epoch is a logically sequential stage. In this embodiment, epochs also occur in a strict chronological order, such that epoch 112 cannot begin before epoch 111 ends.

[0097] In one embodiment, each hyperparameter has its own more or less independent epoch sequence, such that hyperparameter 121 may have gone through many epochs while hyperparameter 122 is still in its first epoch. In another embodiment, all hyperparameters share the same epoch sequence.

[0098] 2.10 Tuple Generation

[0099] When a given hyperparameter has its own epoch sequence, all tuples generated during that epoch have the same value for all hyperparameters (except for the value of the given hyperparameter). For example, as shown in the figure, the given hyperparameter is 123, which is the only hyperparameter explored during that epoch.

[0100] For example, tuples 151-152 share the same values ​​143 and 146 for the corresponding hyperparameters 121-122. While values ​​143 and 146 remain constant during the exploration of the given hyperparameter 123 in epoch 111, value 143 can be a different constant than value 146, or even a different (number) type. Heuristics for choosing constants for these other (i.e., unexplored) hyperparameters will be discussed later in this paper.

[0101] However, tuple 152 has a value of 148 for hyperparameter 123, and tuple 151 has a different value of 147 for hyperparameter 123. Heuristics for changing the values ​​of hyperparameters being explored will be discussed later in this paper. Exploring other hyperparameters requires generating tuples in a similar manner.

[0102] 2.11 Score

[0103] Each generated tuple is tested by configuring a machine learning algorithm using each generated tuple, training the configured algorithm, and performing performance measurements on the algorithm during or after training to score the tuples. For example, tuples 151-152 have corresponding scores 171-172.

[0104] Depending on the implementation, scores 171-172 may be applicability scores (such as classification scores), F1 scores, scores based on loss metrics, or functions of some or all of these. Classification scores measure the accuracy of a trained algorithm in recognizing one or more patterns (such as face recognition).

[0105] The F1 score measures the accuracy of binary identification (such as whether an email is spam). The loss metric measures inaccuracy, so a lower score is better. For scores based on an inverse scale that measures accuracy, a higher score is better. In this embodiment, scores 171-172 alternatively measure the time spent training, so a lower score is better because it indicates faster training.

[0106] The current value range 133A can be, for example, a natural sequence of numeric values, such as 147-149. Although not shown, the current value range 133A can be defined by a minimum and a maximum value.

[0107] During or at the end of epoch 111, the current value range 133A of the explored hyperparameter 123 can be narrowed by adjusting the minimum and / or maximum values ​​of the range to exclude values ​​of tuples that produce poor scores. The narrowed range can be propagated to the next epoch.

[0108] For example, the current value range 133B of epoch 112 could be the result of shrinking the current value range 133A during epoch 111. Heuristics for optimal shrinking will be discussed later.

[0109] 3.0 Gradient search space reduction

[0110] Figure 2 It is a description Figure 1 The block diagram of the example computer 100 is shown. Computer 100 uses gradient search space reduction to estimate the optimal point on the curve of the hyperparameter fractional values.

[0111] During epoch 111, values ​​for hyperparameter 123 are explored only within the current value range 133A. The horizontal (independent) axis of the graph shown spans the current value range 133A.

[0112] 3.1 Curve Sampling

[0113] For example, the current value range 133A could be a continuous sequence of real numbers. Setting the hyperparameter 123 to explore tuples of those real numbers can produce training scores that are calibrated by the vertical (subordinate) axis of the graph and appear along the curve shown on the right with a thick tail.

[0114] Since computer 100 does not know the shape of the curve shown and does not know where the peak (optimal) score of the curve corresponds to the optimal value of hyperparameter 123, exploration is required. Exploration requires sampling values ​​within the current value range 133A.

[0115] In one embodiment, computer 100 selects a fixed number of values ​​within the current value range 133A. In another embodiment, the values ​​are selected so that they are equidistant within the current value range 133A, and therefore equidistant along the horizontal axis.

[0116] Tuples can be generated, each with one of the selected values ​​for hyperparameter 123. Within these tuples, hyperparameters 121-122 remain constant.

[0117] Machine learning algorithms can be configured, trained, and scored based on these tuples. Each point on the curve represents a score for a different (actual or possible) tuple with different values ​​for hyperparameters 1, 2, and 3.

[0118] 3.2 Gradient Detection

[0119] For example, if three values ​​are sampled within the current value range 133A, their fractions can be plotted as X on a curve, such as B, C, and E. The utility of points B, C, and E is reduced because they are too far apart to accurately assess the slope (i.e., gradient) of the curve at these points.

[0120] A small, fixed offset can be added to each sampled value to derive more or fewer neighboring values, which can also be used to generate configuration tuples for training and scoring. For example, points B, C, and E have neighboring points A, D, and F. Since the added offset is positive, these neighboring points are slightly to the right of the original points.

[0121] Each original point and its adjacent points form a pair of points, such as AB, CD, and EF. Each pair of points forms a line segment with an accurate gradient.

[0122] Important information can be derived from the pair gradient and the scores. For example, the scores can be sorted to determine that B is better than C, and that A is the best.

[0123] 3.3 Peak Approximation

[0124] The pairs can be sorted by their scores to determine that pair AB is the best, and pair CD is the second best. The line segments of these two best pairs can be extended to form lines that intersect at a specific point.

[0125] As shown in the figure, the intersection point is more or less an accurate approximation of the actual peak value (it has been shown but not yet discovered). Since the intersection point is an imperfect approximation, it cannot be immediately used as the optimal value for hyperparameter 123.

[0126] 3.4 Narrowing the scope

[0127] Instead, the current value range 133A is narrowed to somewhat confine the intersection. The precise minimum and maximum values ​​of the narrowed range are calculated as follows.

[0128] As shown in the figure, if the best pair (AB) is to the left of the intersection point, then the new minimum value is based on AB. Otherwise, the new maximum value is based on AB.

[0129] Within the optimal pair AB, one point (A or B) scores higher than the other. In this embodiment, the horizontal position of the higher-scoring point is used to set a new minimum or maximum value for the new range of values ​​for hyperparameter 123. Alternatively, the lower-scoring point of the optimal pair is used in this embodiment.

[0130] The new minimum or maximum value that is not based on AB is instead based on the intersection point. Since the intersection point is only an approximation, it should not be used directly as the new minimum or maximum value, otherwise the actual peak is too likely to be excluded from the narrowed range.

[0131] Instead, the horizontal position of the intersection point is shifted away from the optimal pair by a small, fixed offset (e.g., to the right as shown in the figure), thereby increasing the likelihood that the actual peak value is included in the reduced range shown in the figure. Although the current value range 133A initially spanned the entire width of the graph, after the reduction shown in the figure, the new range is reduced to only the width of the shaded vertical band.

[0132] 3.5 Next Epoch

[0133] This narrowing ends at epoch 111, followed by another epoch for the same specific hyperparameter. The best hyperparameter value so far, the fraction, and the latest current value range can be propagated to the next epoch 112. (See reference...) Figure 1 When the next epoch 112 begins, the repeated shrinking of the current value range 133A can be reused as the current value range 133B.

[0134] Other hyperparameters 121-122 can also each complete their own epochs at different times. Depending on the implementation, an epoch that completes earlier for one hyperparameter may or may not wait for an epoch that completes later for different hyperparameters, allowing parallel epochs for different hyperparameters to be synchronized or asynchronous.

[0135] In any case, the best hyperparameter value, score, and / or tuple so far can be shared (e.g., published in a global variable) for use in epochs of other hyperparameters. Exploring an epoch of a hyperparameter allows the current best value of each other hyperparameter to be used as a constant for that other hyperparameter in all tuples generated by that epoch.

[0136] For example, starting at epoch 111, it can set the same values ​​143 and 146 as constants that appeared as optimal values ​​in previous epochs exploring the corresponding hyperparameters 121-122. Thus, while all but one value of each tuple remains constant throughout the epoch, the constants can be improved across epochs, ensuring that the final result is optimal for the combination of hyperparameters rather than just for each isolated hyperparameter.

[0137] Because epochs occur sequentially for a given hyperparameter, further reduction can be achieved by repeatedly performing intersection-based reduction by sampling points within the reduced range and synthesizing neighboring points. Depending on the conditions discussed later in this paper, the reduction may be repeated several or even multiple times, but only once per epoch.

[0138] Finally, the epoch sequence ends, and the repeated shrinking more or less converges to the actual peak. Therefore, a near-optimal (so far best) value for hyperparameter 123 can be found at the end of the final epoch.

[0139] 3.6 Results

[0140] Ultimately, from the epochs of exploring hyperparameters, one or more optimal tuples are discovered, representing one or more more optimal configurations of the machine learning algorithm. Computer 100 (or a downstream computer) can then use this optimal tuple(s) to achieve useful results. For example, computer 100 can use the optimal tuple(s) to ultimately train one or more alternative configurations of the algorithm extensively. For instance, a useful result could be a well-configured and well-trained instance of an algorithm ready for production use.

[0141] The techniques presented in this paper improve the performance of the computer 100 itself in various ways. By intelligently pruning the hyperparameter hyperspace, excessive exploration of hyperparameter configurations is avoided. By ultimately selecting the optimal tuple / configuration, excessive training of too many different algorithm configurations is avoided.

[0142] Therefore, subsequent training (e.g., by computer 100) is performed more quickly. Similarly, the trained selected algorithm(s) configuration achieves higher accuracy in production use (e.g., by computer 100). Thus, computer 100 is accelerated as an algorithm training computer and becomes more reliable (accurate) as a production inference computer. By reducing the computational burden of these activities, the technique described in this paper is accelerated (i.e., time-saving) and energy-saving.

[0143] 4.0 Example Optimization Processing

[0144] Figure 3 This is a flowchart depicting the computer 100 in the embodiment repeatedly narrowing the range of hyperparameter values ​​for a machine learning algorithm to find the optimal hyperparameter values ​​for configuring the algorithm for training. (Reference) Figure 1-2 discuss Figure 3 .

[0145] Steps 302 and 304 discover the optimal configuration tuples for the hyperparameter values ​​of the algorithm. Step 306 uses the optimal tuples to configure and extensively train the algorithm. Steps 302, 304, and 306 can be performed in a laboratory setting. Step 308 uses the trained algorithm to obtain useful results, such as pattern recognition, for use in production environments.

[0146] Steps 302 and 304 involve epochs. Each (numerical) hyperparameter can have its own sequence of exploration epochs. For example, hyperparameter 123 is explored during epochs 111-112.

[0147] Each epoch generates multiple tuples for a specific hyperparameter. For example, computer 100 generates tuples 151-152 during epoch 111.

[0148] Half of the tuples are generated by sampling at equal intervals along the current value range of a particular hyperparameter, such as... Figure 2 The tuples B, C, and E are derived from this. The other half of the tuples is generated by adding a small, fixed offset to the value of a specific hyperparameter for each sampled tuple. For example, tuples A, D, and F are based on offsets relative to their corresponding tuples B, C, and E.

[0149] 4.1 Initialization

[0150] As mentioned above, most values ​​of the tuples in an epoch remain constant, and can be based on the best values ​​of the hyperparameters up to the beginning of that epoch. However, each hyperparameter has its own first epoch, which does not have previous epochs to provide the best value.

[0151] In one embodiment, the midpoint (or random value) of each initial value range for each numerical hyperparameter is used to form the initial optimal tuple and the initial optimal value shared by all hyperparameters. In an embodiment not shown, Figure 3 The process can begin with an initial exploration of the initial optimal value.

[0152] Initial exploration can generate tuples (and score them) by exhaustively combining the minimum or maximum possible values ​​based on each numerical hyperparameter. In this embodiment, a midpoint value is also used. In this embodiment, each hyperparameter is initially explored independently while the values ​​of other hyperparameters are kept at some initial constant, such as a midpoint value.

[0153] 4.2 Exploration

[0154] In step 302, each tuple is generated and processed individually. Therefore, step 302 is repeated for each of the many tuples.

[0155] Finally, in step 302, a score is calculated for the tuple. For example, computer 100 generates tuple 152, which has an exploration value 148 for hyperparameter 123 and hyperparameter-specific constants 143 and 146 for other hyperparameters 121-122.

[0156] Step 302 configures a machine learning algorithm based on tuple 152, and then trains and scores the configured algorithm to obtain a score 172 for tuple 152.

[0157] After repeating step 302, all tuples in epoch 111 have fractions. These fractions are used in step 304 at the end of epoch 111. Therefore, step 304 occurs once per epoch.

[0158] In step 304, the current value range of a specific hyperparameter is narrowed down based on the intersection of a first line based on tuple scores and a second line also based on tuple scores. For example, the scores (and their tuples) are sorted to select the four best (highest-scoring) tuples. For example, the four best tuples sorted in descending order are A, B, C, and D.

[0159] The best and second best tuples form the best pair. For example, tuples A and B form the best pair AB.

[0160] The third and fourth best tuples form the second best pair. For example, tuples C and D form the second best pair CD.

[0161] The first line passes through the best pair of tuples. The second line passes through the second best pair of tuples.

[0162] The first and second lines intersect at the intersection point. The current value range of a particular hyperparameter is narrowed to extend only slightly beyond the intersection point from the optimal tuple. Therefore, the current value range of a particular hyperparameter should be narrower at the end of the epoch (at the end of step 304) than at the beginning of the epoch (at the beginning of step 302).

[0163] 4.3 Era Ordering

[0164] After step 304, control can proceed to step 306, or revisit steps 302 and 304 in the next epoch. In either case, the current epoch has ended.

[0165] Which step follows 304 depends on the epoch ordering criterion. If the current range of values ​​for a particular hyperparameter is still converging (improving), the next epoch begins by revisiting step 302.

[0166] Convergence occurs when any of the following termination conditions are met:

[0167] The first and second lines do not intersect. For example, the gradient is zero at the peak.

[0168] The intersection point falls outside the current value range.

[0169] The width of the current value range has been narrowed to an absolute threshold or a percentage threshold relative to the midpoint of the range.

[0170] If convergence occurs (e.g., because the best score obtained for a particular hyperparameter during the current epoch is not a (significant) improvement over the best score in the previous epoch), then the epoch sequence for that particular hyperparameter will terminate. The epoch sequences for some other hyperparameter(s) may have terminated earlier.

[0171] The epoch sequence of some other hyperparameter(s) may still be in progress. For example, when the first epoch of hyperparameter 122 has ended, the first epoch of hyperparameter 123 may still be in progress. This temporal skew may occur naturally, such as when hyperparameter 123 has a naturally wide range of many possible values ​​and hyperparameter 122 does not have a wide range.

[0172] Step 306 occurs only after the epoch sequence of all hyperparameters has ended, at which point computer 100 has a nearly globally optimal tuple. In step 306, the machine learning algorithm is configured based on the optimal, narrowed range of current values ​​for the algorithm's hyperparameters.

[0173] 4.4 Hill Climb After the Era Sequence

[0174] In this embodiment, the algorithm is configured with near-globally optimal tuples. In this embodiment, step 306 may ultimately use gradient ascent / descent (greedy climbing) to reach a globally optimal (best possible score) tuple from the near-globally optimal tuple.

[0175] In this embodiment, the greedy ramping is based on the optimal tuple for each numerical hyperparameter occurring independently to find tuples with even higher scores for that hyperparameter. Therefore, the greedy optimal value of the hyperparameter can be found after the epoch sequence of the hyperparameter has been completed.

[0176] A new tuple can be synthesized based on the greedy optimal values ​​of all hyperparameters. This tuple can then be scored to check if it exceeds the greedy optimal values ​​of any of the individual hyperparameters.

[0177] 4.5 Final Configuration

[0178] Step 306 can extensively train the algorithm with the optimal tuple configuration. In an embodiment, computer 100 repeatedly configures and extensively trains the algorithm with each of the best few tuples to empirically find which configuration of the algorithm is actually the best among all configurations.

[0179] The algorithm can be trained in both steps 302 and 306. However, step 302 can be efficient by using a small training dataset during exploration, while step 306 can instead be used for extensive training with a large dataset intended for production use.

[0180] At the end of step 306, the algorithm has been extensively trained and is ready for production deployment and use. The extensively trained algorithm is used during step 308.

[0181] In step 308, an extensively trained algorithm is invoked to obtain the result. For example, the algorithm could be deployed in a smartphone and subjected to natural stimulus patterns (such as photos).

[0182] Algorithms can infer / identify / classify photos to obtain conclusions. For example, the result might be that a photo contains or does not contain a face, which could be important for installed smartphone apps such as camera apps.

[0183] 5.0 Tracking Best Scores

[0184] Figure 4 This is a block diagram depicting an example computer 400 in an embodiment. Computer 400 combines epoch management with dynamic tracking of optimal scores and values ​​for numerical and / or categorical hyperparameters. Computer 400 may be an implementation of computer 100.

[0185] Epochs 411-412 explore hyperparameter 422. For all hyperparameters (such as 421-422), epoch 411 begins with the best values ​​so far (such as 441 and 442A).

[0186] For exploring hyperparameter 422, epoch 411 also begins with the best score so far, 472A. Because epoch 411 repeatedly samples and narrows the current value range of hyperparameter 422 (not shown), the best score 472A can be improved zero times, once times, several times, or even many times during epoch 411.

[0187] Whenever the optimal score 472A improves, the optimal value 442A also changes. A new optimal score and optimal value are recorded whenever the optimal score 472A improves, but this only applies to hyperparameter 422.

[0188] Other (e.g., concurrent) epochs (not shown) exploring other hyperparameters maintain their own dynamic optimal scores and optimal values ​​for their respective explored hyperparameters. Thus, the computer 400 actually tracks multiple optimal scores and optimal values ​​separately for each hyperparameter (i.e., each independent sequence of epochs).

[0189] When a new optimal score is achieved, the implementation can also record the current tuple as the new optimal tuple. Similarly, the dynamic optimal tuple is tracked individually for each hyperparameter.

[0190] By the end of epoch 411, the optimal value 442A and optimal fraction 472A have been (e.g., repeatedly) improved to the optimal value 442B and optimal fraction 472B. The improved values ​​can be reused as the optimal value 442C and optimal fraction 472C to begin the next epoch 412.

[0191] Additional processing can occur during epoch 411 or between epochs 411 and 412. Between epochs 411 and 412, the optimal value 442A can be made globally available for all hyperparameters to use in their next (non-current) epoch.

[0192] For example, another hyperparameter (not shown) may have its own epoch, which overlaps with epoch 411 in time and ends before epoch 411. Epoch 412 can use the globally available optimal value of this other hyperparameter.

[0193] 5.1 Optimal Combination

[0194] However, even though both epoch 411 and overlapping epochs can be accomplished by publishing corresponding new best values ​​(such as value A and value B (not shown)), the above techniques may not actually generate tuples that together have two values ​​A and B. For example, even if epoch 412 starts with the recorded best value 442C, the sampling technique may cause epoch 412 to never actually generate a tuple with the best value 442C in the tuples, but instead generate a tuple based on samples near (but not at) the best value 422C.

[0195] Therefore, between epochs 411 and 412, computer 400 can synthesize and score tuples with the best globally available values ​​for each hyperparameter. Even if the synthesized tuple reuses the best value 442B that initially produced the best score 472B, the synthesized tuple may sometimes still score higher than the best score 472B. Thus, although not shown, the best score 472C in the next epoch 412 can actually be a copy of one of the higher scores: the score of the synthesized tuple or the best score 472B.

[0196] The score of the synthesized tuple may also be better than the published best scores of (or more) other hyperparameters. In this embodiment, those published best scores are also updated and associated with the synthesized tuple.

[0197] The score of a synthesized tuple can also exceed the score of the globally best tuple, which the computer 400 also tracks. Therefore, the globally best tuple and score can reflect the tuples and scores generated during an epoch or synthesized between two epochs.

[0198] 5.2 Categorical Hyperparameters

[0199] Unlike numerical hyperparameters (such as 422), not all hyperparameters have reliable gradients. This is because some hyperparameters lack a natural relative ordering of values ​​that can provide a gradient.

[0200] Some hyperparameter types lack a monotonic range spanning from minimum to maximum. Therefore, some techniques based on gradients and intersection lines in this paper are not applicable to certain types of hyperparameters.

[0201] Categorical (i.e., non-numerical, such as text or symbols) hyperparameters, such as 421, are not suitable for narrowing the range and do not have their own epochs. For example, Boolean hyperparameters lack meaningful gradients, especially because it is impossible to obtain two optimal point pairs (i.e., four optimal values) from a range of values ​​that have only two values ​​(i.e., true and false, or 1 and 0).

[0202] Even multi-valued, signified hyperparameters (e.g., those with values ​​representing the seven geographical continents of Africa, Antarctica, Asia, Australia, Europe, North America, and South America) do not exhibit a natural relative ordering of values. Therefore, a special technique not based on gradients is needed to explore categorical hyperparameters.

[0203] As shown in the figure, the categorical hyperparameter 421 can provide its optimal value 441 as a constant for the epochs of other hyperparameters (such as 411). However, the categorical hyperparameter 421 does not have its own epoch.

[0204] Therefore, the categorical hyperparameter 421 is generally not explored during one or more epochs. However, a fundamental improvement in the best score of numerical hyperparameters (such as 422) during an epoch may indicate the discovery of a new subspace of the configuration hyperspace that imposes a new performance mechanism (regime).

[0205] Compared to the optimal value 441, the new performance mechanism may prefer (i.e., score higher) different values ​​of the categorical hyperparameter 421. When epoch 411 ends, if the new optimal score 472B of the numerical hyperparameter 422 exceeds the old optimal score 472A by a higher absolute threshold or percentage threshold 430, the computer 400 will detect that some or all categorical hyperparameters (such as 421) need to be explored spontaneously.

[0206] As shown in the figure, threshold 430 is applied (tested) between epochs 411 and 412. In an embodiment, threshold 430 can be applied dynamically / continuously during epoch 411.

[0207] When the threshold of 430 is exceeded, exploration of one or more categorical hyperparameters (or categorical hyperparameters) is triggered. Similar to epochs, tuples are generated as constants with optimal values ​​for the other hyperparameters.

[0208] Generate distinct tuples for each possible value of the categorical hyperparameter 421. Score these tuples, which may result in the optimal value 441 being surpassed by a distinct categorical value with a better score. This distinct categorical value is then exposed for use by the numerical hyperparameter at the start of its next epoch.

[0209] Other categorical hyperparameters can be explored in a similar manner. Therefore, the optimal value, score, and tuple for each categorical hyperparameter will also be tracked globally.

[0210] 5.3 Revisit

[0211] As discussed, tuples are generated or synthesized at different times, such as during an epoch, between epochs, and when triggered by a threshold. As already discussed, each of these cases of tuple creation can proceed in many different ways.

[0212] Therefore, the same tuple may be unnecessarily revisited, leading to redundant training and scoring, which is wasteful, especially in terms of time. This redundancy may be exacerbated by concurrent epochs targeting different hyperparameters.

[0213] Redundant training and scoring can be avoided by maintaining a lookup table (not shown) that maps each known (visited) tuple to its score. When a tuple is created and it is assumed that training and scoring of the tuple are required, the tuple can be used as a lookup key to attempt to retrieve a known score.

[0214] If a score is retrieved, that score will be used instead of redundantly training and scoring the tuple, since the tuple's score will never change. Otherwise, the tuple is indeed novel and is used for training and scoring, and the new score should be associated with that new tuple in the lookup table.

[0215] 6.0 Category-based optimization

[0216] Figure 5 This is a flowchart depicting the optimization of categorical hyperparameters by computer 400 in an embodiment. (Reference) Figure 4 discuss Figure 5 .

[0217] As mentioned above, categorical hyperparameters lack gradients. Therefore, categorical hyperparameters are explored in a special way that is not based on epochs.

[0218] Step 502 dynamically detects when to further optimize the categorical hyperparameters. In one embodiment, step 502 occurs at the end of an epoch. In another embodiment, step 504 can occur dynamically at any time during an epoch.

[0219] Step 502 detects whether the improvement in the best score for a specific hyperparameter of an epoch exceeds a threshold. This threshold can be the absolute or percentage difference between the best score at the beginning of an epoch and (depending on the embodiment) at the end of an epoch or at any time during an epoch where the best score has improved.

[0220] In one embodiment, the threshold is compared to the best score for a specific hyperparameter of an epoch. Alternatively, in another embodiment, the threshold is compared to the globally best score, regardless of the specific hyperparameter.

[0221] If no significant improvement is detected in step 502, optimization of the categorical hyperparameters will not occur at this point. If a significant improvement is indeed detected in step 502, optimization of the categorical hyperparameters will begin (using step 504).

[0222] Depending on the implementation, this optimization can be performed on one, some, or all categorical hyperparameters. As shown in the figure, all categorical hyperparameters are optimized.

[0223] 6.1 Category-based combinations

[0224] When multiple categorical hyperparameters are being optimized, in one embodiment shown, the exploration of each categorical hyperparameter may occur independently (e.g., a separate set of tuples), or in another embodiment (not shown), multiple categorical hyperparameters may be explored together (e.g., a shared set of tuples). Which of these two embodiments is better may depend on the number of categorical hyperparameters and the number of possible values ​​for these categorical hyperparameters.

[0225] For example, twenty categorical hyperparameters, each with two possible values, correspond to a hyperspace of 20 x 2 = 40 tuples. However, fifty categorical hyperparameters, each with thirty possible values, correspond to a hyperspace of 50 x 30 = 1,500 tuples.

[0226] In a dynamic implementation, when such combinations would produce fewer than a threshold number of tuples, multiple categorical hyperparameters are explored together using a shared set of tuples. In this case, tuples are generated by exhaustively enumerating all possible combinations of all possible values ​​for all multiple categorical hyperparameters.

[0227] In this dynamic embodiment, when such combinations exceed a threshold, each categorical hyperparameter has a separate set of tuples for exploration. In this case, the categorical hyperparameter values ​​in the tuples, except for the specific categorical hyperparameter being explored, are kept constant at their best values ​​so far. Only that specific hyperparameter has all the possible values ​​of those tuples enumerated in its separate set of tuples.

[0228] In all cases, the numerical (non-categorical) hyperparameter remains constant at its best value so far in all tuples generated for categorical exploration. Tuples are generated and used in step 504. Step 504 is repeated for each tuple generated during categorical exploration.

[0229] In step 504, a score is calculated based on the generated tuples. For example, tuples are generated, used to configure the algorithm, and then the configured algorithm is trained and scored.

[0230] Each categorical hyperparameter can have the best value, best score, and / or best tuple so far. These best data can be updated through categorical exploration.

[0231] Upon completion, categorical exploration can publish revised optimal values ​​to be used as constants in tuples for exploring numerical hyperparameters in future epochs.

[0232] 7.0 Horizontal scaling

[0233] Figure 6 This is a block diagram depicting an example computer 600 in an embodiment. Computer 600 is scaled horizontally. Computer 506 may be an implementation of computer 100.

[0234] 7.1 Multiprocessing

[0235] Computer 600 has multiple processors 611-616, each of which may be a networked computer, a central processing unit (CPU), a processor core such as for multi-core processing, hardware hyper-threading, a virtual machine, a heavyweight operating system process, or a lightweight thread. Depending on the embodiment, computer 600 may be a computer supporting symmetric multiprocessing (SMP) or a combination or cluster of multi-core computers.

[0236] 7.2 Concurrency

[0237] Each of processors 611-616 can provide one or more computational threads for coarse-grained (also known as task-based) parallelism. Coarse-grained parallelism can be heterogeneous, allowing processor 611 to train configured algorithms, while processor 612 manages epochs.

[0238] Processor 611 can use one thread to train the algorithm configuration and another thread to manage epochs simultaneously. Processor 611 can train one algorithm configuration for an epoch with one hyperparameter, while the same or different processors can simultaneously train another algorithm configuration for the same epoch or another epoch with a different hyperparameter.

[0239] Therefore, the computer 600 has various ways to scale horizontally as much as possible and / or overload the processor with additional tasks as needed.

[0240] 7.3 Load Balancing

[0241] An ideal work allocation is shown to achieve maximum horizontal scaling for a single epoch, allowing each processor to train and score an algorithm configuration in parallel. For example, processor 611 trains the algorithm configured by tuple 651, while processor 612 trains based on tuple 652.

[0242] Regardless of the number of processors available, the ideal work allocation is practically achievable as long as at least four processors are available to simultaneously train the two pairs (four points) required to generate the intersecting lines shown in the graph above. The ideal work allocation can be achieved with more than four processors as long as an even number of processors are available to train more than two pairs simultaneously, since each pair has an even number (two) points.

[0243] An odd number or mismatched number of processors can still scale horizontally, but the workload may be somewhat unbalanced, causing some processors (one or more) to finish early and then wait (idle). Even when an epoch is unbalanced, idleness can be more or less avoided through concurrent epochs with different hyperparameters.

[0244] To achieve ideal load distribution based on an arbitrary even number of processors, half the number of available processors should be sampled within the current value range of the hyperparameter. Therefore, as mentioned above, each sampling point has two processors. This is important because each sampling point appears in a pair of points, which also includes a composite point based on the sampled values ​​plus an offset, as described above.

[0245] With six available processors 611-616, then 6 / 2 = three (equally spaced) values ​​should be sampled. From these values, three pairs can be obtained by adding three collinear additional points based on offsets.

[0246] For example, the spaced tuple 651 is a sample, while the collinear tuple 652 is based on the offset, making tuples 651-652 a pair. The six points in each of the three pairs are shown as X in the graph.

[0247] Each processor 611-616 is trained based on the corresponding tuples from the corresponding points in the six points. Thus, the six tuples are used simultaneously on the six processors to train for detecting an intersection of two lines.

[0248] Therefore, even if six training iterations are actually possible over multiple epochs, a single iteration based on finding an intersection point and narrowing the value range may occur entirely within the time required for a single training algorithm. Thus, more or less linear speedup can be maintained within one epoch and across multiple epochs within the limits allowed by Amdahl's law.

[0249] 7.4 Synchronization

[0250] Because exploring a hyperparameter and / or exploring multiple hyperparameters simultaneously can occur in parallel, and because optimal values, scores, and tuples (e.g., global variables) are occasionally published, different processors may concurrently read or write to variables. Therefore, without additional thread safety, pathological race conditions leading to or caused by data inconsistency may occur.

[0251] For thread safety, processors 611-616, for example, can cache some data locally at the beginning of an epoch (e.g., read from a global data structure) and then flush dirty data (e.g., improved scores) back to that global data structure at the end of the epoch. Thus, thread synchronization can be more or less confined to between epochs.

[0252] Global data structures should be protected as needed, for example, to allow concurrent readers but serialize writers, such as by using read / write locks on critical sections with protection logic.

[0253] For example, Java has a more or less robust memory consistency model, which includes synchronization logic, volatile variables, and the java.util.concurrent package of utility classes. Remote synchronization may involve more spinlocks for maintaining cache consistency, test-and-set instructions, and memory fences.

[0254] 7.5 Cross-validation

[0255] Ideally, training with an infinitely large dataset might achieve near-perfect scores. However, that would require an infinite amount of time, making it impractical.

[0256] In practice, efficient training is achieved by sacrificing ideal scores using a limited training dataset. According to the holdout approach to training, a small portion (i.e., no more than one-third) of the training dataset is not used for training but instead for testing (e.g., scoring), because for optimal results, testing and training should not use the same data.

[0257] In the case of cross-validation, the original dataset will be divided equally in at least three ways, called folds. Empirically, five folds are optimal. The fold rows can be stratified, allowing for careful sorting of the original data so that each fold has an equal amount of data of a particular kind (e.g., category).

[0258] Ideally, directories can be reused to create as many training datasets as directories, but more or fewer are possible. Each dataset can be trained using some (ideally all but one) of these directories.

[0259] At least one (and usually only one) of the remaining directories is used for testing. Each of the different training datasets uses a different directory for testing.

[0260] For example, dataset 660 is divided into directories 1-5 of equal size for reuse during five-way cross-validation. Subsets of these reusable directories are used to create different training datasets for each of at least training runs 621-622 that produce at least the corresponding scores 631-632. Scores 631-632 can be averaged to compute individual scores for tuples configured with the algorithm.

[0261] Each training run round is trained using a separate training dataset for the same tuple / configuration. For example, according to Figure 640, the training dataset for run round 621 consists of directories 2-5, while directory 1 is used as the test dataset.

[0262] Training datasets from different run rounds can partially overlap. For example, catalogs 3-5 are shared by run rounds 621-622.

[0263] Ideally, each directory is used as the test dataset for exactly one training run. Therefore, ideally, and although not shown, directories 1-5 should be reused for five training runs, which in this embodiment may occur simultaneously.

[0264] For example, rounds 621-622 can occur simultaneously on the corresponding processors 611-612. Therefore, cross-validation is horizontally scalable.

[0265] 8.0 Hardware Overview

[0266] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute these techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) persistently programmed to execute these techniques, or may include one or more general-purpose hardware processors programmed to execute these techniques according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement these techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement these techniques.

[0267] For example, Figure 7 This is a block diagram illustrating a computer system 700 on which embodiments of the present invention may be implemented. The computer system 700 includes a bus 702 or other communication mechanism for transmitting information, and a hardware processor 704 coupled to the bus 702 for processing information. The hardware processor 704 may be, for example, a general-purpose microprocessor.

[0268] Computer system 700 also includes main memory 706, such as random access memory (RAM) or other dynamic storage devices, coupled to bus 702 for storing information and instructions to be executed by processor 704. Main memory 706 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 704. When these instructions are stored in non-transitory storage media accessible to processor 704, they make computer system 700 a dedicated machine customized to perform the operations specified in the instructions.

[0269] The computer system 700 also includes a read-only memory (ROM) 708 or other static storage device coupled to the bus 702 for storing static information and instructions for the processor 704. A storage device 710, such as a magnetic disk or optical disk, is provided and is coupled to the bus 702 for storing information and instructions.

[0270] Computer system 700 can be coupled to display 712, such as a cathode ray tube (CRT), via bus 702 for displaying information to the computer user. Input device 714 (which includes alphanumeric and other keys) is coupled to bus 702 for transmitting information and command selections to processor 704. Another type of user input device is cursor control 716, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to processor 704 and for controlling cursor movement on display 712. Such input devices typically have two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), which allows the device to specify a position in a plane.

[0271] Computer system 700 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic, combined with the computer system, enable or program the computer system 700 to be a special-purpose machine. According to one embodiment, the techniques described herein are executed by computer system 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 706. These instructions may be read into main memory 706 from another storage medium, such as storage device 710. Execution of the instruction sequence contained in main memory 706 causes processor 704 to perform the processing steps described herein. In alternative embodiments, hard-wired circuitry may be used instead of or in combination with software instructions.

[0272] As used herein, the term "storage medium" refers to any non-transient medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 76. Volatile media include dynamic memory, such as main memory 706. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassettes.

[0273] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 702. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0274] Various forms of media can involve carrying one or more sequences of instructions to processor 704 for execution. For example, instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them via a telephone line using a modem. A modem local to computer system 700 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 702. Bus 702 carries the data to main memory 706, from which processor 704 retrieves and executes the instructions. Instructions received from main memory 706 may optionally be stored on storage device 76 before or after execution by processor 704.

[0275] Computer system 700 also includes a communication interface 718 coupled to bus 702. Communication interface 718 provides bidirectional data communication coupled to network link 720, which is connected to local network 722. For example, communication interface 718 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 718 may be a LAN card providing data communication connectivity to a compatible local area network (LAN). A wireless link may also be implemented. In any such implementation, communication interface 718 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0276] Network link 720 typically provides data communication to other data devices via one or more networks. For example, network link 720 may provide a connection to host computer 724 or to data devices operated by Internet Service Provider (ISP) 726 via local network 722. ISP 726, in turn, provides data communication services through a worldwide packet data communication network now commonly referred to as the "Internet" 728. Both local network 722 and Internet 728 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, as well as signals on network link 720 and through communication interface 718, are example forms of transmission media in which information carries digital data to or from computer system 700.

[0277] Computer system 700 can send messages and receive data, including program code, through one or more networks, network links 720, and communication interfaces 718. In the Internet example, server 730 can transmit request codes to the application through the Internet 728, ISP 726, local network 722, and communication interface 718.

[0278] The received code can be executed by processor 704 when it is received, and / or stored in storage device 710 or other non-volatile memory for later execution.

[0279] 10.0 Software Overview

[0280] Figure 8 This is a block diagram of a basic software system 800 that can be used to control the operation of computer system 700. The software system 800 and its components, including their connections, relationships, and functions, are merely exemplary and are not intended to limit the implementation of one or more of the example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0281] Software system 800 is provided to guide the operation of computer system 700. Software system 800, which may be stored on system memory (RAM) 706 and fixed storage device (e.g., hard disk or flash memory) 76, includes a kernel or operating system (OS) 810.

[0282] OS 810 manages the low-level aspects of computer operations, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more applications, designated 802A, 802B, 802C...802N, can be "loaded" (e.g., transferred from fixed storage device 76 to memory 706) for execution by system 800. Applications or other software intended for use on computer system 700 can also be stored as downloadable computer-executable instruction sets, for example, for downloading and installing from internet locations (e.g., web servers, app stores, or other online services).

[0283] Software system 800 includes a graphical user interface (GUI) 815 for receiving user commands and data graphically (e.g., "click" or "touch gestures"). These inputs can then be manipulated by system 800 according to instructions from operating system 810 and / or (one or more) applications 802. GUI 815 also displays the results of operations from OS 810 and (one or more) applications 802, allowing the user to provide additional input or terminate the session (e.g., log off).

[0284] OS 810 can execute directly on the bare hardware 820 of computer system 700 (e.g., one or more processors 704). Alternatively, a hypervisor or virtual machine monitor (VMM) 830 can be inserted between the bare hardware 820 and OS 810. In this configuration, VMM 830 acts as a software “buffer” or virtualization layer between OS 810 and the bare hardware 820 of computer system 700.

[0285] VMM 830 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine includes a “guest” operating system (such as OS 810) and one or more applications (such as Application 802) designed to run on the guest operating system. VMM 830 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0286] In some instances, VMM 830 can allow a guest operating system to run as if it were running directly on the bare hardware 820 of the computer system 800. In these instances, the same version of the guest operating system configured to run directly on the bare hardware 820 can also run on VMM 830 without modification or reconfiguration. In other words, VMM 830 can provide complete hardware and CPU virtualization to a guest operating system in some situations.

[0287] In other instances, a guest operating system can be specifically designed or configured to run on VMM 830 for improved efficiency. In these instances, the guest operating system is "aware" that it is running on a virtual machine monitor. In other words, VMM 830 can provide paravirtualization to the guest operating system under certain circumstances.

[0288] Computer system processes include the allocation of hardware processor time, as well as the allocation of memory (physical and / or virtual), the allocation of memory for storing instructions executed by the hardware processor, the allocation of memory for storing data generated by the execution of instructions by the hardware processor, and / or the storage of hardware processor state (e.g., register contents) between hardware processor time allocations when the computer system process is not running. Computer system processes run under the control of the operating system and can also run under the control of other programs executing on the computer system.

[0289] 8.0 Cloud Computing

[0290] This article generally uses the term "cloud computing" to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0291] Cloud computing environments (sometimes called cloud environments or the cloud itself) can be implemented in various ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the public. In contrast, private cloud environments are generally used only by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.

[0292] Generally, cloud computing models enable some of the responsibilities that might have previously been provided by an organization's own IT department to be delivered as service layers within the cloud environment for consumer use (depending on the public / private nature of the cloud, within or outside the organization). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything in the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provide processes, storage devices, networks, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service where consumers use database servers or database management systems running on cloud infrastructure, while the DBaaS provider manages or controls the underlying cloud infrastructure and applications.

[0293] The basic computer hardware and software, as well as the cloud computing environment described above, are presented to illustrate the basic underlying computer components that can be used to implement one or more example embodiments. However, the one or more example embodiments are not necessarily limited to any particular computing environment or computing device configuration. Rather, according to this disclosure, the one or more example embodiments can be implemented in any type of system architecture or processing environment that a person skilled in the art will understand in accordance with this disclosure to be capable of supporting the features and functionality of the one or more example embodiments presented herein.

[0294] In the foregoing description, embodiments of the invention have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the description and drawings should be considered illustrative rather than restrictive. The unique and exclusive indication of the scope of the invention, and the content that the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the set of claims published from this application in the specific form of such claims, including any subsequent corrections.

Claims

1. A computer-implemented method, comprising: For each iteration in the first plurality of iterations: Choose multiple values ​​from the range of optimal values ​​for the first hyperparameter of the machine learning algorithm, where: When there is an even number of available processors, the selected multiple values ​​are grouped into one or more pairs, and the number of said one or more pairs is determined based on half of the number of available processors; The machine learning algorithm is trained based on the plurality of values ​​of the first hyperparameter of the machine learning algorithm. Multiple scores are calculated based on the training. Two lines are identified based on the aforementioned multiple scores. Detect the intersection of the two lines based on the plurality of scores, and In response to detecting the intersection of the two lines based on the plurality of scores, the range of values ​​for the optimal value of the first hyperparameter of the machine learning algorithm is narrowed, wherein the optimal value is an estimate of one item selected from a group consisting of: The minimum duration for training the machine learning algorithm, and The maximum applicability of the machine learning algorithm; In parallel with the first plurality of iterations, for each iteration of the second plurality of iterations: Narrowing the range of values ​​for a specific optimal value of the second hyperparameter of the machine learning algorithm, and When the optimal value of the first hyperparameter or the specific optimal value of the second hyperparameter is found, communication is performed between the first plurality of iterations and the second plurality of iterations; The machine learning algorithm is configured based on the optimal value of the first hyperparameter and the specific optimal value of the second hyperparameter; and The machine learning algorithm is invoked to obtain the results.

2. The method according to claim 1, wherein, The plurality of values ​​includes the six values ​​of the first hyperparameter.

3. The method according to claim 1, wherein, Selecting the plurality of values ​​for the first hyperparameter includes: selecting a first half of the plurality of values ​​and a second half of the plurality of values ​​based on the first half of the plurality of values.

4. The method according to claim 1, wherein, The two lines include at least one of the following: Four of the plurality of values ​​of the first hyperparameter, and Based on four values ​​from the plurality of scores used to train the machine learning algorithm.

5. The method according to claim 1, wherein, Identifying the two lines includes sorting the plurality of scores based on training the machine learning algorithm.

6. The method according to claim 1, further comprising: After the first plurality of iterations, the following operations are performed iteratively: Select a second plurality of values ​​from a subrange of the value range of the first hyperparameter, and Based on the second plurality of values ​​of the first hyperparameter, a specific optimal value of the first hyperparameter is estimated, which is better than the optimal value of the first hyperparameter from the first plurality of iterations.

7. The method according to claim 6, wherein, The subrange of the value range of the first hyperparameter is based on the optimal value of the first hyperparameter from the first plurality of iterations.

8. The method according to claim 6, wherein, The iteration following the first plurality of iterations is performed in response to obtaining a new optimal value for the second hyperparameter of the machine learning algorithm.

9. The method according to claim 6, further comprising: When the difference between the specific optimal value of the first hyperparameter and the optimal value of the first hyperparameter exceeds a threshold, a new optimal value for the categorical hyperparameter of the machine learning algorithm is detected.

10. A computer-readable nontransitory medium storing one or more instructions, which, when executed by one or more processors, cause the execution of the steps of any one of claims 1-9.

Images