Hyperparameter search system and its program

Abstract

To provide a new hyper parameter search method capable of reducing a calculation amount.SOLUTION: A solution candidate generation part 2 generates solution candidate groups about a hyper parameter of machine learning. Similarity calculation parts 6a to 6c calculates a similarity DS for each solution candidate about the inputted solution candidate groups. Hyper parameter search parts 7a to 7c perform hyper parameter search with each of the solution candidate groups as a processing object to select a solution candidate group. In that case, an accuracy score AS is calculated about a machine learning model prescribed by a processing object, and if the accuracy score does not satisfy prescribed reference, a solution candidate evaluated to be similar to the processing object on the basis of the similarity DS is excluded from its subsequent processing objects.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a hyperparameter search system and a program therefor. [Background technology]

[0002] In the field of machine learning, hyperparameter search is one of the important themes that influence the accuracy and performance of machine learning models. Here, hyperparameters refer to parameters that configure the behavior of machine learning algorithms, such as the number of hidden layers in a machine learning model, the number of neurons per layer, and the selection of loss functions. In addition to these, parameters related to image processing and parameters related to feature calculations such as higher-order local autocorrelation (HLAC) are also included in hyperparameters.

[0003] For example, Patent Document 1 describes an optimization method for optimizing machine learning parameters (including hyperparameters) with a small number of evaluations. Specifically, in a first step, an evaluation value, which is an index for evaluating a calculation result, is calculated using parameters that serve as search points and evaluation data. In a second step, the parameters are optimized. In a third step, the calculation of the evaluation value and the optimization of the parameters are repeated to output an optimal value of the parameter. Here, in the third step, first, a plurality of data points each consisting of a pair of a parameter and an evaluation value calculated using the parameter as a search point are stored. Next, based on the stored parameters, a plurality of search point candidates, which are parameters that can be candidates for search points, are generated. Then, for each of the plurality of search point candidates, it is determined whether or not the search point candidate should be used as a search point using the stored plurality of data points.

[0004] Furthermore, Patent Document 2 discloses a solution search method that can reliably calculate a solution within a specified calculation time without increasing memory usage when solving an optimization problem using an objective function, and can improve the accuracy of the solution. Specifically, in a solution search using simulation, a node to be simulated is first selected from nodes that are options in a search tree, and the simulation is executed from the selected node. Next, an evaluation value is calculated based on the simulation results using an evaluation function, and the evaluation values ​​of the selected node and its upper nodes are updated based on this evaluation value. Then, nodes with evaluation values ​​that do not satisfy a predetermined criterion are removed from the search tree. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-192160 [Patent Document 2] International Publication No. 2014 / 115232 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in hyperparameter search, the amount of calculation increases dramatically as the number of hyperparameters increases, and it is known that adjusting them takes a significantly long time. For example, if there are two hyperparameters a and b, each of which can take on 100 patterns, a grid search that verifies all possible combinations of patterns requires calculations for 100 x 100, or 10,000 patterns. If a hyperparameter c (100 patterns) is added, the number of patterns increases to 1,000,000.

[0007] As described in the aforementioned Patent Document 1, some kind of performance index is used in hyperparameter search. This performance index is often an accuracy index (e.g., classification accuracy of a classifier, prediction accuracy of a predictor, recognition accuracy of a recognizer) that indicates the relative accuracy of a machine learning model, such as accuracy rate. However, when actually using a machine learning model, it is necessary to also consider performance indexes that indicate the operational performance of the machine learning model, such as inference processing time. Because accuracy and operational performance are in a trade-off relationship, it is difficult to maximize both. Furthermore, because user preferences (task characteristics) and other factors are also reflected, it is difficult to integrate multiple performance indexes into a single index.

[0008] Therefore, an object of the present invention is to provide a novel hyperparameter search method that can reduce the amount of calculation. [Means for solving the problem]

[0009] To solve this problem, a first invention provides a hyperparameter search system having a solution candidate generation unit, a similarity calculation unit, and a hyperparameter search unit. The solution candidate generation unit generates a group of solution candidates for machine learning hyperparameters. The similarity calculation unit calculates a similarity for each of the solution candidates, representing the degree of similarity between the hyperparameter values ​​of the solution candidates. The hyperparameter search unit performs a hyperparameter search on each of the solution candidates as a processing target, and selects the solution candidates. Here, the hyperparameter search unit calculates a score of a first performance index representing the superiority or inferiority of a predetermined performance for a machine learning model defined by the processing target, and if the score of the first performance index does not satisfy a predetermined standard, excludes the solution candidate evaluated to be similar to the processing target based on the similarity from subsequent processing targets.

[0010] In the first invention, a performance estimation unit and a grouping unit may be provided. The performance estimation unit uses a pre-prepared estimation model to output a performance estimation value for each of a group of candidate solutions. This estimation model outputs a performance estimation value that estimates the performance of the machine learning model (related to performance different from the score of the first performance index) in response to an input of a set of hyperparameter values. The grouping unit classifies the group of candidate solutions into one of a plurality of groups by grouping solution candidates having similar performance estimation values. In this case, the similarity calculation unit calculates the similarity for each group, and the hyperparameter search unit performs hyperparameter search for each group.

[0011] In the first aspect of the present invention, a filter unit may be provided that excludes from the processing target those solution candidate groups whose performance estimate values ​​do not satisfy a predetermined standard. Furthermore, it is preferable that the performance estimate values ​​are values ​​that estimate the superiority or inferiority of the operating performance of the machine learning model.

[0012] In the first invention, the hyperparameter search unit may calculate a score of a second performance index that indicates the superiority or inferiority of performance, different from the score of the first performance index of the machine learning model. In this case, it is preferable that the first performance index indicates the superiority or inferiority of the accuracy of the machine learning model, and the second performance index indicates the superiority or inferiority of the operating performance of the machine learning model. Furthermore, the score of the first performance index and The score of the second performance index A result processing unit may be provided that stores the selected solution candidates in association with each other.

[0013] A second invention provides a hyperparameter search program that causes a computer to execute a process having the following steps: In the generation step, a group of candidate solutions is generated for machine learning hyperparameters; In the calculation step, a similarity indicating the degree of similarity between the hyperparameter values ​​of the candidate solutions is calculated for each candidate solution; In the search step, a hyperparameter search is performed with each candidate solution as a processing target to select the candidate solutions. Here, the search step includes a step of calculating a score of a first performance index indicating the superiority or inferiority of a predetermined performance for a machine learning model defined by the processing target; and a step of excluding a candidate solution that is evaluated to be similar to the processing target based on the similarity from subsequent processing targets if the score of the first performance index does not satisfy a predetermined criterion.

[0014] The second invention may further include an estimation step and a grouping step. In the estimation step, a performance estimate is output for each of a group of candidate solutions using a pre-prepared estimation model. This estimation model, in response to an input of a set of hyperparameter values, outputs a performance estimate that estimates performance different from the score of the first performance index in the machine learning model. In the grouping step, the group of candidate solutions is classified into one of a plurality of groups by grouping solution candidates having similar performance estimates. In this case, the calculation step calculates the similarity for each group, and the search step performs a hyperparameter search for each group.

[0015] In the second invention, a filtering step may be provided for excluding from the processing target those solution candidate groups whose performance estimate values ​​do not satisfy a predetermined standard. Furthermore, it is preferable that the performance estimate values ​​are values ​​that estimate the superiority or inferiority of the operating performance of the machine learning model.

[0016] In a second aspect of the present invention, the searching step may include a step of calculating a score of a second performance index that indicates the superiority or inferiority of performance, different from the score of the first performance index of the machine learning model. In this case, it is preferable that the first performance index indicates the superiority or inferiority of the accuracy of the machine learning model, and the second performance index indicates the superiority or inferiority of the operating performance of the machine learning model. Furthermore, the score of the first performance index and The score of the second performance index A storage step may be provided for storing the selected solution candidates in association with each other. [Effects of the Invention]

[0017] According to the present invention, if the score of the first performance index calculated for a candidate solution to be processed does not satisfy a predetermined standard, the candidate solution evaluated as being similar to the candidate solution to be processed is excluded from subsequent processing, thereby reducing the number of hyperparameter search attempts and effectively reducing the overall amount of calculations. [Brief explanation of the drawings]

[0018] [Figure 1] Block diagram of the hyperparameter search system [Figure 2] Hyperparameter search process flowchart [Figure 3] Illustration of filtering of solution candidates [Figure 4] Diagram of grouping of solution candidates [Figure 5] Similarity table illustration [Figure 6] Flowchart showing details of hyperparameter search [Figure 7] A diagram showing the data structure of the search results [Figure 8] Illustration of excluding solution candidates [Figure 9] Optimal solution for the entire system DETAILED DESCRIPTION OF THE INVENTION

[0019] 1 is a block diagram of a hyperparameter search system for machine learning according to this embodiment. This hyperparameter search system 1 is mainly composed of a solution candidate generation unit 2, a performance estimation unit 3, a filter unit 4, a grouping unit 5, multiple similarity calculation units 6a-6c, multiple hyperparameter search units 7a-7c, and a result processing unit 8. In addition, an estimation model 9 is prepared in advance so that the performance estimation unit 3 can acquire necessary information. The solution candidate generation unit 2 generates a group of solution candidates (multiple solution candidates) to be processed by this system 1 for machine learning hyperparameters.

[0020] The performance estimation unit 3 uses an estimation model 9 to output an estimated value (hereinafter referred to as "performance estimation value EV") that estimates a predetermined performance of the machine learning model for each solution candidate that constitutes the solution candidate group generated by the solution candidate generation unit 2. The estimation model 9 is a model that specifies the relationship between a set of hyperparameter values ​​(input) and the performance estimation value EV (output), and specifically, is constructed as a predetermined function or machine learning model.

[0021] The "predetermined performance" to be estimated here includes the operational performance of the machine learning model, such as the cycle time required to process one image unit or the memory usage during processing. Because these performance parameters are highly dependent on the hyperparameter values ​​themselves, they can be estimated in advance as approximate values ​​without performing hyperparameter search. The type of operational performance may be fixed by the system or may be selectable by the user from multiple options. A set of hyperparameter values ​​for a given solution candidate is input to the estimation model 9, and the value output from the estimation model 9 is uniquely determined as the performance estimate (EV) for that solution candidate. While stochastic gradient descent (SGD) in neural networks, for example, does not necessarily produce a definitive performance estimate (EV) that accurately reflects the actual performance due to the presence of probabilistic elements, factors such as the SGD learning rate and the kernel size of the CNN (convolutional neural network) still significantly contribute to operational performance. Therefore, even in such cases, it is possible to estimate a valid performance estimate (EV). Furthermore, the performance estimate (EV) may be calculated by taking into account probabilistic fluctuations. For example, the standard deviation σ is also estimated and the threshold value is set to ±3σ.

[0022] The filter unit 4 excludes from the group of solution candidates generated by the solution candidate generation unit 2 those whose performance estimate value EV does not satisfy a predetermined standard from further processing. This standard is set as a threshold value of the performance estimate value EV, and may be fixed in the system or may be arbitrarily set by the user. As a result, solution candidates whose operational performance does not satisfy the predetermined requirements are filtered out at a pre-stage without performing hyperparameter search.

[0023] The grouping unit 5 groups solution candidates having similar performance estimates EV from the solution candidate group filtered by the filter unit 4, thereby classifying the solution candidate group into one of multiple groups. Processing after grouping is basically performed on a group-by-group basis set by the grouping unit 5. For example, if three groups A to C are set, group A is processed by the similarity calculation unit 6a and the hyperparameter search unit 7a, group B is processed by the similarity calculation unit 6b and the hyperparameter search unit 7b, and group C is processed by the similarity calculation unit 6c and the hyperparameter search unit 7c. Note that the processing of groups A to C may be performed in parallel to improve processing speed.

[0024] The similarity calculation unit 6a calculates a similarity DS for each solution candidate in group A. The similarity DS represents the degree of similarity between hyperparameter values ​​between solution candidates, and can be calculated and evaluated using, for example, cosine similarity or Euclidean distance. Here, cosine similarity is an index that represents the degree to which a vector defined by one solution candidate (solution candidate vector) and a vector defined by another solution candidate (solution candidate vector) are oriented in the same direction in a k-dimensional vector space defined by k (k≧2) hyperparameters. Like the trigonometric function cosine, the closer the cosine similarity is to 1, the more similar the two candidates are evaluated to be. Furthermore, Euclidean distance is the distance between a point defined by one solution candidate (solution candidate point) and a point defined by another solution candidate (solution candidate point) in a k-dimensional space, and the closer the cosine distance is to 0, the more similar the two candidates are evaluated to be. The similarity DS is calculated for all combinations of solution candidates in group A. The multiple similarities DS calculated for group A are written to a similarity table stored in a storage device (not shown), as will be described later.

[0025] The hyperparameter search unit 7a performs hyperparameter search on each of the solution candidates belonging to group A as a processing target, i.e., performs learning using training data for a learning model defined by k hyperparameters and verification using test data for the trained learning model, to select the solution candidates. The hyperparameter search unit 7a also calculates a performance index score representing the superiority or inferiority of a predetermined performance for the machine learning model defined by the solution candidates. In this embodiment, an "accuracy index" is used as this performance index, and a score of the accuracy index (hereinafter referred to as "accuracy score AS") is calculated.

[0026] Here, "accuracy index" refers to an index that indicates the accuracy of a machine learning model. For example, in the case of a classification model, an index that indicates the accuracy of classification corresponds to an accuracy index. In the case of a prediction model, an index that indicates the accuracy of prediction corresponds to an accuracy index, and in the case of a recognition model, an index that indicates the accuracy of recognition corresponds to an accuracy index. Accuracy indexes are broadly divided into generalization accuracy indexes such as "ROC·AUC" (hereinafter abbreviated as "AUC") and maximum accuracy indexes such as the F-measure, and either can be used. The ROC (Receiver Operating Characteristic) in "ROC·AUC" is called an estimation curve, and is plotted with the ratio of TPR (True Positive Rate) on the vertical axis and FPR (False Positive Rate) on the horizontal axis. AUC (Area Under the Curve) refers to the surface under the ROC curve. A larger AUC area generally indicates better machine learning accuracy. The F-score is the harmonic mean of the recall rate (the proportion of predicted positive results among those that are actually positive) and the precision rate (the proportion of data predicted to be positive that are actually positive). In addition to these, AP (Average Precision) and other accuracy indices may also be used. AP is a value calculated by the area under the precision-recall curve in the same manner as AUC. The type of accuracy indices may be fixed in the system, or the user may be able to select from multiple options.

[0027] If the accuracy score AS does not satisfy a predetermined standard, the hyperparameter search unit 7a excludes the unprocessed solution candidate evaluated as similar to the processing target based on the similarity DS calculated by the similarity calculation unit 6a from subsequent processing targets. This reduces the number of hyperparameter search attempts. This is based on the idea that if the accuracy of a machine learning model defined by a certain solution candidate is low, the accuracy of a machine learning model defined by a solution candidate similar to this one will also be low.

[0028] Furthermore, the hyperparameter search unit 7a calculates a performance index score (hereinafter referred to as "performance score MS") that indicates the superiority or inferiority of the performance of the machine learning model defined by the solution candidate, as a performance index different from the accuracy index described above. Here, the performance index refers to an index that indicates the superiority or inferiority of the performance of the machine learning model, and is an actual measurement value obtained by actually measuring and calculating the performance equivalent to the performance estimate value EV described above. As described above, the hyperparameter search unit 7a selects solution candidates, and outputs the accuracy score AS and the performance score MS associated with each solution candidate. Note that in random search and Bayesian optimization, the results of this hyperparameter search are fed back to the solution candidate generation unit 2 as necessary, and a new group of solution candidates is generated.

[0029] On the other hand, similarity calculation units 6b and 6c perform the same processing as similarity calculation unit 6a, except that the processing targets are different from groups B and C. Moreover, hyperparameter search units 7b and 7c perform the same processing as hyperparameter search unit 7a, except that the processing targets are different from groups B and C.

[0030] The result processing unit 8 integrates information about the optimal solution output from the hyperparameter search units 7a to 7c, and stores and holds the information as the overall search result. The information held as the search result includes at least information about the accuracy of the solution candidates (accuracy score AS ) and information about the operational performance of the solution candidate (performance estimate EV and / or operational score MS).

[0031] Next, the details of the hyperparameter search process will be specifically explained with reference to FIG. 2. First, in step 1, multiple solution candidates are extracted from a set of k hyperparameters using an appropriate method, and a group of solution candidates to be processed is generated. This group of solution candidates does not necessarily include all solution candidates that exist within the search range set by the user. For example, in random search or Bayesian optimization, a portion of the search range is extracted to generate a group of solution candidates, and another group of candidates is generated depending on the results of the hyperparameter search. Next, in step 2, a performance estimate value EV is calculated for each of the group of solution candidates generated in step 1 using an estimation model 9.

[0032] In step 3, the solution candidate group is filtered by evaluating the performance estimate value EV calculated in step 2. In this embodiment, as an example of the estimation model 9, the following function with hyperparameters HP1 and HP2 as inputs is used.

[0033] Memory usage = HP1 x 3 + 0.5 Cycle time = HP2 x 0.5 + HP2 x 0.5 + 0.5

[0034] Specifically, as shown in Figure 3, the performance estimate value EV of each of the solution candidates is compared with a predetermined threshold value EVth1. Then, solution candidates c1 to c3 whose performance estimate value EV is lower than the predetermined threshold value EVth1 are excluded from processing as they do not satisfy the desired requirements (operational performance). By filtering in consideration of operational performance, the number of trials in the hyperparameter search is reduced.

[0035] In step 4, grouping is performed on the filtered solution candidate group based on the performance estimate value EV calculated in step 2. In this grouping, a predetermined boundary value that defines the boundary between adjacent groups is used. As shown in FIG. 4, when classifying the solution candidate group into three groups, two boundary values EVth2 and EVth3 are used. For solution candidates with EV ≤ EVth3, they are evaluated as having excellent operating performance and classified into group A. For solution candidates with EVth3 < EV ≤ EVth2, they are evaluated as having normal operating performance and classified into group B. Also, for solution candidates with EVth2 ≤ EV, they are evaluated as having poor operating performance and classified into group C. Steps 5 and 6 hereafter are performed in units of groups.

[0036] In step 5, for each of groups A to C set in step 4, the similarity DS is calculated in units of groups. Within the same group, basically, the similarity DS is calculated for all combinations of solution candidates (combinations that cross adjacent groups are excluded). By limiting the calculation target of the similarity DS within the same group, the combinations of solution candidates are significantly reduced, thus reducing the calculation amount. The similarity DS calculated in this step 5 is described and stored in the similarity table in the storage device. As shown in FIG. 5, in the similarity table, a pair of solution candidates specified by the ID and their similarity are described. For example, the description in the uppermost row indicates that the similarity between the solution candidate with ID "001" and the solution candidate with ID "002" is "0.89", showing relatively high similarity. Note that in the similarity table, it is sufficient to describe only those with a similarity DS of a predetermined threshold DSth or more, that is, those evaluated as having high similarity. Also, the comparison result between the similarity DS and the threshold DSth may be described as the similarity table. In this case, it is sufficient to describe only the IDs of the pairs of solution candidates evaluated as having similarity (the similarity item is unnecessary).

[0037] In step 6, a hyperparameter search is performed on a group-by-group basis, and a group of solution candidates is selected (optimal solutions are calculated) on a group-by-group basis. That is, for group A, a hyperparameter search is performed on the group of solution candidates belonging to group A, and at least one optimal solution (including its candidates; the same applies below) that represents group A is calculated. For group B, a hyperparameter search is performed on the group of solution candidates belonging to group B, and at least one optimal solution that represents group B is calculated. Then, for group C, a hyperparameter search is performed on the group of solution candidates belonging to group C, and at least one optimal solution that represents group C is calculated.

[0038] FIG. 6 is a flowchart showing the details of hyperparameter search focusing on one group. First, in step 11, one solution candidate (a set of hyperparameter values) is selected from a group of solution candidates belonging to a certain group and set as the processing target. Next, in step 12, learning is performed using pre-prepared learning data on the machine learning model defined by the solution candidate to be processed. Then, in step 13, verification is performed on this machine learning model using test data, and an accuracy score AS and an operation score MS are calculated. As described above, the accuracy score AS is a score of an accuracy index that indicates the superiority or inferiority of the accuracy of the machine learning model, and the operation score MS is a score of an operation index that indicates the superiority or inferiority of the operation performance of the machine learning model. These scores AS, MS are stored in a storage device as a predetermined data set. Fig. 7 is a diagram showing an example of the data structure of the search results. Each search result is labeled with an ID, and is associated with the values ​​of the hyperparameters HP1 to HPk, the performance estimate value EV (memory usage and cycle time), the operation score MS (memory usage and cycle time), the accuracy score AS, and the group to which it belongs.

[0039] In step 14, the accuracy score AS is compared with a predetermined threshold ASth to determine the quality of the solution candidate to be processed. If the determination in step 14 is positive, i.e., if the accuracy of the machine learning model defined by the solution candidate is evaluated to be low, the similarity table described above is referenced to determine whether a solution candidate similar to the solution candidate exists (step 15). If the determination in step 15 is positive, unprocessed solution candidates similar to the solution candidate are excluded from further processing (step 16). For example, as shown in FIG. 8, if solution candidate c4 in group B is the processing target and its accuracy score AS is low, unprocessed solution candidates c5 and c6 that exist within the similarity range of solution candidate c4 are excluded from further processing. The exclusion of solution candidates in step 16 is performed on the premise that the two conditions of the solution candidate being low in accuracy score AS and similar to the solution candidate are met. If either of these conditions is not met, the determination in either step 14 or 15 is negative, and step 16 is skipped, so that unprocessed solution candidates are not excluded.

[0040] In step 17, it is determined whether a predetermined search termination condition is met. Examples of search termination conditions include when a machine learning model that meets the desired requirements has been obtained, or when it is determined that a set of hyperparameters has been sufficiently explored. If the determination in step 17 is negative, the process returns to step 11, the next solution candidate is set as the processing target, and the same process is repeated. Then, if the search termination condition is met, the process exits the loop of steps 11 to 17 and proceeds to step 18, where solution candidates that meet the predetermined requirements (for example, a predetermined number of the top candidates in order of performance) are output as the optimal solution.

[0041] Referring back to FIG. 2, in step 7, the outputs of the hyperparameter search units 7a to 7c are integrated and saved as a single search result. These search results are compiled by the result processing unit 8 and presented to the user as an overall optimal solution integrating groups A to C. For example, as shown in FIG. 9, if the optimal solution for group A is c7, the optimal solution for group B is c8, and the optimal solution for group C is c9, three overall optimal solutions c7 to c9 are obtained (however, each group does not necessarily have one optimal solution). Based on these, an evaluation list screen is generated that lists evaluation scores associated with each solution candidate and is presented to the user. This evaluation score screen basically shows the same items as in FIG. 7, and for each "ID" assigned to each solution candidate, the scores of the performance indicators ("cycle time" and "memory usage") and the scores of the accuracy indicators are described. More preferably, a performance estimate value is also listed. In addition, instead of or in addition to the evaluation score screen, the user may be presented with a distribution of solution candidate points in a solution space, with the horizontal axis representing the performance index and the vertical axis representing the accuracy index, as shown in Figure 9.

[0042] As described above, according to this embodiment, if the accuracy score AS calculated for the solution candidate to be processed (e.g., c4 in FIG. 8) does not satisfy a predetermined standard, unprocessed solution candidates (c5 and c6 in FIG. 8) evaluated as being similar to the processing candidate c4 are excluded from subsequent processing targets. This reduces the number of hyperparameter search attempts without impairing the evaluation accuracy, and effectively reduces the overall amount of calculation. Note that this effect does not require grouping of the solution candidates, and can be obtained even if grouping is not performed.

[0043] Furthermore, according to this embodiment, by classifying the solution candidates into multiple groups and performing calculation of the similarity DS and hyperparameter search on a group-by-group basis, the number of combinations of solution candidates to be evaluated can be reduced, thereby effectively reducing the overall amount of calculation.

[0044] Furthermore, according to this embodiment, by filtering out (filtering) candidate solutions whose performance estimates EV do not satisfy a predetermined standard prior to the hyperparameter search, the number of trials of the hyperparameter search is reduced, thereby effectively reducing the overall amount of calculations. Note that this effect does not require grouping of candidate solutions, and can be obtained even if grouping is not performed.

[0045] Furthermore, according to this embodiment, by generating an evaluation score screen that shows an evaluation score including information on the accuracy of the machine learning model (accuracy score AS) and information on the operational performance of the machine learning model (operation score MS and / or performance estimate EV), it is possible to provide the user with information useful for identifying a Pareto-optimal solution that satisfies both accuracy and operational performance. This makes it easier for the user to consider operational policies, such as selecting three "models with 90% accuracy and requiring 200 MB of memory" and performing an ensemble over "models with 100% accuracy and requiring 10 GB of memory."

[0046] In the above-described embodiment, the calculation of the similarity DS (step 5) is performed before the hyperparameter search (step 6) is performed, but it may also be performed during the hyperparameter search, specifically after the calculation of the accuracy score AS in step 13. For example, if step 5 is provided between step 14 and step 15, the similarity DS is calculated only for processing targets with low accuracy scores AS, thereby reducing the amount of calculation.

[0047] Furthermore, the present invention can also be understood as a hyperparameter search program that causes a computer to execute the procedures shown in the flowcharts of FIGS. [Explanation of symbols]

[0048] 1 Hyperparameter search system 2 Solution candidate generator 3 Performance Estimation Section 4. Filter section 5 Grouping section 6a~6c Similarity calculation part 7a~7c Hyperparameter search section 8 Result Processing Section 9 Estimation Model

Claims

1. In a hyperparameter search system, a solution candidate generation unit that generates a group of solution candidates for machine learning hyperparameters; a similarity calculation unit that calculates, for each of the solution candidates, a similarity that indicates a degree of similarity between hyperparameter values ​​of the solution candidates with respect to the group of solution candidates; a hyperparameter search unit that performs a hyperparameter search on each of the solution candidate groups to select the solution candidate groups, The hyperparameter search unit Calculating a score of a first performance index representing the superiority or inferiority of a predetermined performance for the machine learning model defined by the processing object; A hyperparameter search system characterized in that, if the score of the first performance index does not meet a predetermined standard, a solution candidate evaluated to be similar to the processing target based on the similarity is excluded from subsequent processing targets.

2. a performance estimation unit that uses a pre-prepared estimation model to output a performance estimation value that estimates performance different from the score of the first performance index in the machine learning model in response to an input of a set of hyperparameter values, and outputs the performance estimation value for each of the group of candidate solutions; a grouping unit that classifies the group of solution candidates into one of a plurality of groups by grouping solution candidates having similar performance estimate values, the similarity calculation unit calculates the similarity on a group-by-group basis, 2. The hyperparameter search system according to claim 1, wherein the hyperparameter search unit performs the hyperparameter search in units of groups.

3. 3. The hyperparameter search system according to claim 2, further comprising a filter unit that excludes from the processing target those of the solution candidate group whose performance estimate values ​​do not satisfy a predetermined standard.

4. 4. The hyperparameter search system according to claim 2, wherein the performance estimate value is a value that estimates the relative merits of the operational performance of the machine learning model.

5. The hyperparameter search unit The hyperparameter search system according to claim 1, further comprising: a second performance index score representing the superiority or inferiority of performance of the machine learning model, the second performance index score being different from the first performance index score.

6. the first performance index represents the accuracy of the machine learning model; The hyperparameter search system according to claim 5 , wherein the second performance index indicates the superiority or inferiority of the operational performance of the machine learning model.

7. 7. The hyperparameter search system according to claim 5, further comprising a result processing unit that stores the score of the first performance index and the score of the second performance index in association with each of the selected solution candidates.

8. In the hyperparameter search program, A generation step of generating a group of candidate solutions for machine learning hyperparameters; a calculation step of calculating, for each of the solution candidates, a similarity that indicates a degree of similarity between hyperparameter values ​​of the solution candidates with respect to the group of solution candidates; a search step of selecting the solution candidate group by performing a hyperparameter search on each of the solution candidate group as a processing target; causing a computer to execute a process having the following steps: The searching step calculating a score of a first performance index representing a predetermined performance level for the machine learning model defined by the processing object; if the score of the first performance index does not satisfy a predetermined standard, excluding the solution candidate evaluated to be similar to the processing target based on the similarity from subsequent processing targets; A hyperparameter search program comprising:

9. an estimation step of outputting a performance estimate value for each of the group of candidate solutions using an estimation model that is prepared in advance to output a performance estimate value that estimates performance different from the score of the first performance index in the machine learning model in response to an input of a set of hyperparameter values; a grouping step of classifying the group of candidate solutions into one of a plurality of groups by grouping the group of candidate solutions having similar performance estimate values, the calculation step calculates the similarity on a group-by-group basis, 9. The hyperparameter search program according to claim 8, wherein the search step performs the hyperparameter search in units of groups.

10. 10. The hyperparameter search program according to claim 9, further comprising a filtering step of excluding from the processing target, among the group of solution candidates, those for which the performance estimate value does not satisfy a predetermined standard.

11. 11. The hyperparameter search program according to claim 9, wherein the performance estimate value is a value that estimates the relative merits of the operational performance of the machine learning model.

12. The searching step 9. The hyperparameter search program according to claim 8, further comprising a step of calculating a score of a second performance index that indicates superiority or inferiority of performance different from the score of the first performance index in the machine learning model.

13. the first performance index represents the accuracy of the machine learning model; The hyperparameter search program according to claim 12, wherein the second performance index indicates the superiority or inferiority of the operational performance of the machine learning model.

14. 14. The hyperparameter search program according to claim 12, further comprising a storage step of storing the score of the first performance index and the score of the second performance index in association with each of the selected solution candidates.

Images