Hyperparameter search system and its program

Abstract

To provide a new hyperparameter search method that can reduce an amount of calculation.SOLUTION: A case data processing unit 2a obtains a correlation coefficient group in which a correlation coefficient is calculated for each pair of a hyperparameter group for related case data related to k (k≥3) hyperparameters to be searched for of past case data related to hyperparameter search. A grouping unit 2 groups hyperparameters evaluated to have correlation for k hyperparameters based on the correlation coefficient group. A hyperparameter search unit 3 performs hyperparameter search using a predetermined search method in units of groups. An output unit 4 puts together optimum values obtained by the hyperparameter search in units of groups, and outputs k hyperparameters as a search result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a hyperparameter search system and a program therefor. [Background technology]

[0002] In the field of machine learning, hyperparameter search is one of the important themes that influence the accuracy and performance of machine learning models. Here, hyperparameters refer to parameters that configure the behavior of machine learning algorithms, such as the number of hidden layers in a machine learning model, the number of neurons per layer, and the selection of loss functions. In addition to these, parameters related to image processing and parameters related to feature calculations such as higher-order local autocorrelation (HLAC) are also included in hyperparameters.

[0003] For example, Patent Document 1 discloses a learning device capable of reducing the computational cost for determining hyperparameters. This learning device includes a calculation unit and a learning unit. The calculation unit calculates values ​​of a first objective function and a second objective function. The value of the first objective function includes smoothness, which indicates the smoothness of the local distribution of the model output, and is used to estimate first model parameters that define a model (neural network). The value of the second objective function is used to estimate second model parameters that are closer to the distance scale of the training data using the first objective function and hyperparameters of a learning method for training this model as second model parameters. The learning unit updates the first model parameters and the second model parameters so as to optimize the values ​​of the first objective function and the second objective function. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-46236 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in hyperparameter search, there is a known problem that the amount of calculation increases dramatically as the number of hyperparameters increases, and the adjustment takes a significantly long time. For example, if there are two hyperparameters a and b, each of which can take on 100 patterns, a grid search that verifies all possible combinations of patterns will require calculations for 100 x 100, or 10,000 patterns, and if a hyperparameter c (100 patterns) is added, the number of patterns will increase to 1,000,000.

[0006] In this regard, the above-mentioned Patent Document 1 introduces two types of objective functions, which allows the hyperparameters to be determined within the learning process, thereby reducing the time required. However, because it uses the characteristics of neural networks as prior knowledge, it cannot be applied to machine learning methods other than neural networks.

[0007] The present invention has been made in view of the above circumstances, and its object is to provide a novel hyperparameter search method that can reduce the amount of calculation. [Means for solving the problem]

[0008] To solve this problem, a first invention provides a hyperparameter search system having a grouping unit, a hyperparameter search unit, and an output unit. The grouping unit sets N (k>N≧2) groups for k (k≧3) hyperparameters to be searched based on the correlation between hyperparameters in past case data related to the hyperparameter search. The hyperparameter search unit performs hyperparameter search on a group-by-group basis using a predetermined search method. The output unit collects optimal values ​​obtained by the hyperparameter search on a group-by-group basis and outputs them as search results for the k hyperparameters.

[0009] Here, in the first invention, the grouping unit may have a case data processing unit. The case data processing unit calculates a correlation coefficient for each pair of hyperparameter groups based on related case data extracted from the case data as being related to the k hyperparameters, thereby obtaining a group of correlation coefficients for the related case data. In this case, it is preferable that the related case data has a group of hyperparameters that matches the k hyperparameters and is extracted based on predetermined search conditions.

[0010] In the first invention, the grouping unit may group the k hyperparameters based on a comparison result between each of the sets of correlation coefficients of the related case data and a predetermined threshold. The grouping unit may also group the k hyperparameters by clustering identical or similar correlation coefficients among the sets of correlation coefficients of the related case data. Furthermore, the grouping unit may generate a correlation screen for the related case data that associates the sets of hyperparameters with evaluation indicators of the machine learning model defined by the sets of hyperparameters and the sets of correlation coefficients, and may group the hyperparameters based on hyperparameters designated as correlated by a user viewing the correlation screen.

[0011] In the first invention, a performance evaluation unit and a user adjustment unit may be provided. The performance evaluation unit calculates an evaluation index of the machine learning model defined by k hyperparameters output as the search result. The user adjustment unit allows the user to adjust the k hyperparameters output as the search result in accordance with the evaluation index. In this case, it is preferable that the user adjustment unit newly registers the adjusted k hyperparameters and the evaluation index as case data.

[0012] A second invention provides a hyperparameter search program that causes a computer to execute a process having the following steps: In a first step, N (k>N≧2) groups are set for k (k≧3) hyperparameters to be searched, based on the correlations between hyperparameters in past case data related to the hyperparameter search; in a second step, hyperparameter search is performed for each group using a predetermined search method; and in a third step, optimal values ​​obtained by the hyperparameter search for each group are compiled and output as search results for the k hyperparameters.

[0013] Here, in the second invention, the first step may include a step of calculating a correlation coefficient for each pair of hyperparameter sets based on related case data extracted from the case data as being related to the k hyperparameters, thereby obtaining a correlation coefficient set of related case data. In this case, it is preferable that the related case data has a hyperparameter set that matches the k hyperparameters and is extracted based on predetermined search conditions.

[0014] In a second aspect of the present invention, the first step may group the k hyperparameters based on a comparison result between each of the correlation coefficients of the related case data and a predetermined threshold. The first step may also group the k hyperparameters by clustering identical or similar correlation coefficients among the correlation coefficients of the related case data. Furthermore, the first step may include the steps of: generating a correlation screen for the related case data that associates the hyperparameters, evaluation indicators of the machine learning model defined by the hyperparameters, and the correlation coefficients; and grouping the hyperparameters based on hyperparameters designated as correlated by a user viewing the correlation screen.

[0015] In the second invention, a fourth step of calculating an evaluation index of the machine learning model defined by the k hyperparameters output as the search results, and a fifth step of allowing the user to adjust the k hyperparameters output as the search results according to the evaluation index may be provided. In this case, it is preferable to provide a sixth step of newly registering the adjusted k hyperparameters and the evaluation index as case data. [Effects of the Invention]

[0016] According to the present invention, the k hyperparameters to be searched are grouped together using correlations between hyperparameters in past case data, and hyperparameter search is performed in divided groups. The optimal values ​​obtained by the hyperparameter search for each group are then compiled, and the k hyperparameters are output as the search results. This effectively reduces the amount of calculation required for hyperparameter search without compromising the accuracy of the machine learning model. [Brief explanation of the drawings]

[0017] [Figure 1] Block diagram of the hyperparameter search system [Figure 2] Hyperparameter search process flowchart [Figure 3] Case study database configuration diagram [Figure 4] An example of related case data extraction [Figure 5] A diagram showing an example of a group of correlation coefficients [Figure 6] Figure 5. Undirected graph showing correlations [Figure 7] Undirected graph with connections removed DETAILED DESCRIPTION OF THE INVENTION

[0018] 1 is a block diagram of a hyperparameter search system according to this embodiment. This hyperparameter search system 1 is mainly composed of a grouping unit 2, a hyperparameter search unit 3, an output unit 4, a performance evaluation unit 5, a user adjustment unit 6, and a case database 7.

[0019] The grouping unit 2 sets multiple groups for the hyperparameter sets input as search targets. This grouping uses the correlations between hyperparameters in past case data related to hyperparameter search, and groups correlated hyperparameters together. In this embodiment, past case data is centrally stored in a case database 7, and the case data processing unit 2a included in the grouping unit 2 extracts case data related to the search target as related case data to be referenced in the grouping. The case data processing unit 2a also calculates correlation coefficients for each pair of hyperparameter sets based on the related case data, thereby obtaining a set of correlation coefficients for the related case data. Then, the hyperparameter sets to be searched are grouped by evaluating the correlations between the hyperparameters based on this set of correlation coefficients.

[0020] The hyperparameter search unit 3 performs hyperparameter search on a group-by-group basis using a predetermined search method. The output unit 4 compiles the optimal values ​​obtained by the hyperparameter search on a group-by-group basis and outputs a set of hyperparameters as the search result. The performance evaluation unit 5 performs training using training data on a machine learning model defined by the set of hyperparameters output as the search result. At the same time, the performance evaluation unit 5 calculates an evaluation index indicating the performance superiority or inferiority of the trained model using the performance verification data. The user adjustment unit 6 allows the user to adjust the values ​​of the set of hyperparameters according to the evaluation index calculated by the performance evaluation unit 5 (fine tuning). The user adjustment unit 6 also newly registers the set of hyperparameters adjusted by the user and the evaluation index (either before or after adjustment) in the case database 7 as case data.

[0021] Next, the hyperparameter search process will be described in detail with reference to Fig. 2. First, in step 1, k (k≧3) hyperparameters to be searched are input to the grouping unit 2. In this embodiment, as an example, the objective is to build a learning model using higher-order local autocorrelation (HLAC) as a method for extracting image features, and the k hyperparameters include a set (k=5) of "noise removal filter size," "patch size," "correlation width," "number of feature dimensions," and "number of principal components."

[0022] In step 2, the case data processing unit 2a, which is part of the grouping unit 2, extracts case data stored in the case database 7 that is related to the k hyperparameters being searched as related case data. Figure 3 shows an example of the configuration of the case database 7. The case database 7 stores past case data related to hyperparameter search, and an "evaluation index" and a "hyperparameter group" are associated with each case using a "case ID," which is assigned to each case. The "evaluation index" is an evaluation index of the machine learning model defined by the hyperparameter group, and can be, for example, the "ROC·AUC." The "ROC·AUC" is one of the evaluation indexes for machine learning. The ROC (Receiver Operating Characteristic) is called an estimation curve, and is plotted with the TPR (True Positive Rate) on the vertical axis and the FPR (False Positive Rate) on the horizontal axis. On the other hand, the AUC (Area Under the Curve) refers to the surface under the ROC curve. A larger AUC area generally indicates better machine learning performance. In addition to ROC·AUC, other evaluation indices such as F-measure (F-measure) and AP (Average Precision) may also be used. For example, case ID "0001" indicates that the "performance index" (ROC·AUC) of the machine learning model defined by "noise reduction filter size" = 50, "patch size" = 10, "correlation width" = 5, "number of feature dimensions" = 128, and "number of principal components" = 6 is 0.97 (maximum is 1.0).

[0023] FIG. 4 is a diagram showing an example of related case data extraction. Related case data must satisfy the following two conditions (AND condition). The first condition is that the hyperparameter group of the case data matches the k hyperparameters to be searched. The second condition is that the data must satisfy predetermined search conditions set by the system or the user.

[0024] The first condition, that is, a match between the sets of hyperparameters, is ideally a perfect match, but partial match is also acceptable. For example, if the N hyperparameters are A, B, and C and the set of hyperparameters in the example data is A, B, and D, the first condition is evaluated as being met if the two hyperparameters A and B match. In this case, the concept of a degree of match indicating how many hyperparameters match may be introduced, and the first condition may be evaluated as being met if the degree of match between the two is equal to or greater than a predetermined threshold.

[0025] Regarding the second condition, any search condition may be used, but one example is a threshold value for evaluating the relative merits of evaluation indices. For example, when the threshold value is set to 0.9, of the five case IDs "0001" to "0005" shown in FIG. 3, the three case IDs "0001," "0002," and "0005," which are deemed to have superior evaluation indices based on the comparison results between the two, are extracted as related case data. By filtering the case data according to the relative merits of the evaluation indices, the accuracy of the machine learning model can be ensured with a small amount of calculation.

[0026] In step 3, the case data processing unit 2a calculates the correlation coefficient between hyperparameters based on the related case data extracted in step 2. Since the correlation coefficient is calculated for each pair of hyperparameters, k × k correlation coefficients are obtained for the entire group of hyperparameters. FIG. 5 shows an example of the group of correlation coefficients. A correlation coefficient is an index (a real value in the range of ±1) that measures the strength of the linear relationship between two pieces of data. Typically, the Pearson product-moment correlation coefficient expressed by the following formula can be used. X and Y are the data (pair of hyperparameters), cov(X, Y) is the covariance, and σX and σY are the standard deviations. To obtain an appropriate correlation coefficient, a sufficient number of samples (related case data) must be extracted in step 2. Other examples of correlation coefficients that can be used include Spearman's rank correlation coefficient and Kendall's rank correlation coefficient.

[0027]

number

[0028] In step 4, the grouping unit 2 sets N (k>N≧2) groups for the k hyperparameters to be searched. This grouping is performed, for example, based on the results of comparing each of the correlation coefficients of the related case data with a predetermined threshold. FIG. 6 shows an undirected graph representing the correlations in FIG. 5. In this undirected graph, each node corresponds to a specific hyperparameter shown in FIG. 5, and the lines connecting the nodes correspond to the correlation between a pair of hyperparameters (with the correlation coefficients in FIG. 5). Starting with this as the initial state, if the correlation coefficient is smaller than a predetermined threshold (e.g., 0.5), the correlation between the node pair (hyperparameter pair) connected by the line with this correlation coefficient is deemed to be low, and this line is deleted. As a result, as shown in FIG. 7, the line connecting the two nodes "patch size" and "correlation width" (correlation coefficient 0.7) and the line connecting the two nodes "number of feature dimensions" and "number of principal components" (correlation coefficient 0.8) remain as being highly correlated. Based on grouping rules such as grouping highly correlated nodes (hyperparameters) into the same group, for example, isolated "noise removal filter size" is classified into "Group 1," "patch size" and "correlation width" into "Group 2," and "number of principal components" and "number of feature dimensions" into "Group 3."

[0029] The grouping of hyperparameters may be performed by clustering the same or similar correlation coefficients among the correlation coefficients of the related case data. This grouping may also be performed based on a user instruction. In this case, the grouping unit 2 generates a correlation screen that associates the hyperparameters, the evaluation indexes of the machine learning model defined by the hyperparameters, and the correlation coefficients for the related case data, and presents this to the user. The user viewing the correlation screen specifies correlated items, in other words, hyperparameters to be grouped. The grouping unit 2 performs grouping of hyperparameters in response to the user's instruction.

[0030] In step 5, the hyperparameter search unit 3 sets the initial values ​​and search ranges for each of the N hyperparameters based on the case data (which may be related case data) stored in the case database 7. For example, the average value or most frequent value of the hyperparameters may be used as the initial value. Alternatively, the user may arbitrarily set the initial values ​​by referring to the case data.

[0031] In step 6, the loop variable n is set to 1. This loop variable n specifies the group number set in step 4, and is incremented sequentially until it reaches N (for example, 3). By setting n to 1, "group 1" is specified as the processing target.

[0032] In step 7, the hyperparameter search unit 3 performs a hyperparameter search using a predetermined search method for n=1, i.e., for the "noise reduction filter size" belonging to "group 1." When n=1, the hyperparameters belonging to "group 2" and "group 3" are fixed to the initial values ​​set in step 5. For the hyperparameter search, training data and evaluation data for evaluating the merits and demerits of hyperparameter values ​​are prepared in advance and used. Furthermore, grid search is typically used as a hyperparameter search method. Grid search is a method for building models of all possible value patterns for a given hyperparameter. Alternatively, Bayesian optimization may be used. Bayesian optimization is a type of optimization algorithm that utilizes uncertainty to find the next value to be searched, and a Gaussian process is used as a surrogate model for estimating the objective function. The optimal value (related to the "noise reduction filter size") obtained by the hyperparameter search is temporarily stored in a storage device, and the processing for "group 1" is then terminated.

[0033] In step 8, it is determined whether the loop variable n has reached N (=3). If n=1, the result of the determination in step 8 is negative, 1 is added to the loop variable n (step 9), and the process returns to step 7.

[0034] In step 7, the hyperparameter search unit 3 performs a hyperparameter search for n=2, i.e., for the "patch size" and "correlation width" that belong to "group 2." When n=2, the hyperparameters that belong to "group 1" are fixed to the optimal values ​​obtained in step 7 for n=1, and the hyperparameters that belong to "group 3" are fixed to the initial values ​​set in step 5. The optimal values ​​(related to "patch size" and "correlation width") obtained by the hyperparameter search are temporarily stored in a storage device, and the processing for "group 2" is then completed.

[0035] In step 8, it is determined whether the loop variable n has reached N (=3). If n=2, the result of the determination in step 8 is negative, 1 is added to the loop variable n (step 9), and the process returns to step 7.

[0036] In step 7, the hyperparameter search unit 3 performs a hyperparameter search for n=3, i.e., for the "number of feature dimensions" and "number of principal components" belonging to "group 3." When n=3, the hyperparameters belonging to "group 1" are fixed to the optimal values ​​obtained in step 7 for n=1, and the hyperparameters belonging to "group 2" are fixed to the optimal values ​​obtained in step 7 for n=2. The optimal values ​​obtained by the hyperparameter search (related to the "number of feature dimensions" and "number of principal components") are temporarily stored in the storage device, and processing for "group 3" is thereby completed.

[0037] In step 8, it is determined whether the loop variable n has reached N (=3). If n=3, the determination result in step 8 is affirmative, and the process proceeds to step 10.

[0038] In step 10, the output unit 4 outputs the search results of k (k=5) hyperparameters by combining the optimal value of the "noise removal filter size" obtained by the hyperparameter search for "group 1," the optimal value of the "patch size" and the optimal value of the "correlation width" obtained by the hyperparameter search for "group 2," and the optimal value of the "number of feature dimensions" and the optimal value of the "number of principal components" obtained by the hyperparameter search for "group 3."

[0039] In step 11, the performance evaluation unit 5 uses the above-mentioned training data to train a machine learning model defined by the k hyperparameters output as the search results. Then, in step 12, the performance evaluation unit 5 uses pre-prepared performance verification data to evaluate the training results of the machine learning model and calculates the above-mentioned evaluation indexes (ROC, AUC, etc.) as indices indicating its performance. In general, the performance verification data is often different from the data prepared when executing the grid search, but if the total amount of data is small, the same data may be used.

[0040] In step 13, the user adjustment unit 6 allows the user to adjust the k hyperparameters output as the search results according to the evaluation indexes, and accepts the adjusted values ​​(fine tuning). Then, in step 11, the user adjustment unit 6 assigns a new case ID and newly registers case data including the k hyperparameters adjusted in step 13 and the evaluation indexes calculated in step 12 (or evaluation indexes recalculated based on the adjusted values) in the case database 7. This completes the series of processes.

[0041] As described above, according to this embodiment, the hyperparameter search is divided into groups of k hyperparameters to be searched, utilizing correlations between hyperparameters in past case data. This is based on the knowledge that correlations observed in past case data also tend to be observed in similar k hyperparameters to be searched this time. The optimal values ​​obtained by the hyperparameter search for each group are then compiled, and the k hyperparameters are output as the search results. This effectively reduces the amount of computation required for hyperparameter search without compromising the accuracy of the machine learning model. For example, if two hyperparameters, b and c, out of three hyperparameters a, b, and c (each with 100 patterns), are grouped, the grid search only requires computation for 100 + (100 × 100), i.e., 10,100 patterns. This is a significant reduction in computational effort compared to the 1,000,000 patterns required without grouping.

[0042] Furthermore, according to this embodiment, by calculating a group of correlation coefficients using related case data as samples and using the results of comparing the correlation coefficients with a predetermined threshold, it becomes possible to perform grouping without compromising the accuracy of the machine learning model.

[0043] Furthermore, according to this embodiment, the method is not limited to neural networks as in Patent Document 1, but can be widely applied to various other machine learning methods, and has the advantage of being highly versatile.

[0044] In the above-described embodiment, an example has been described in which a case database 7 is provided and the correlation (correlation coefficient) between hyperparameter groups in past case data related to hyperparameter search is calculated each time. However, the present invention is not limited to this. For various combinations of hyperparameter groups, correlation coefficients between hyperparameters may be calculated in advance and compiled as a correlation coefficient table. In this case, the grouping unit 2 can obtain the information necessary for grouping simply by referring to the correlation coefficient table, without having to calculate the correlation coefficient each time. Furthermore, rules for grouping hyperparameters may be extracted based on the correlation between hyperparameter groups in past case data, and compiled as a rule database. In this case, the grouping unit 2 can obtain the information necessary for grouping simply by referring to the rule database.

[0045] Furthermore, the present invention can be understood as a hyperparameter search program that causes a computer to execute the procedure shown in the flowchart of FIG. [Explanation of symbols]

[0046] 1 Hyperparameter search system 2 Grouping section 2a Case data processing section 3 Hyperparameter search part 4 Output section 5 Performance Evaluation Department 6 User adjustment section 7. Case Database

Claims

1. In a hyperparameter search system, a grouping unit that sets N (k>N≧2) groups for k (k≧3) hyperparameters to be searched based on correlations between hyperparameters in past case data related to hyperparameter search; a hyperparameter search unit that performs hyperparameter search on a group basis using a predetermined search method; an output unit that collects the optimal values ​​obtained by the group-based hyperparameter search and outputs them as search results for the k hyperparameters; A hyperparameter search system comprising:

2. the grouping unit has a case data processing unit, 2. The hyperparameter search system according to claim 1, wherein the case data processing unit calculates a correlation coefficient for each pair of hyperparameters based on related case data extracted from the case data as being related to the k hyperparameters, thereby obtaining a group of correlation coefficients for the related case data.

3. 3. The hyperparameter search system according to claim 2, wherein the related case data is extracted based on predetermined search conditions and the group of hyperparameters matches the k hyperparameters.

4. 3. The hyperparameter search system according to claim 2, wherein the grouping unit groups the k hyperparameters based on a comparison result between each of the correlation coefficients of the related case data and a predetermined threshold value.

5. 3. The hyperparameter search system according to claim 2, wherein the grouping unit groups the k hyperparameters by clustering the same or similar correlation coefficients among the group of correlation coefficients of the related case data.

6. 3. The hyperparameter search system according to claim 2, wherein the grouping unit generates a correlation screen that associates the group of hyperparameters, the evaluation index of the machine learning model defined by the group of hyperparameters, and the group of correlation coefficients for the related case data, and groups the hyperparameters based on hyperparameters designated as having a correlation by a user viewing the correlation screen.

7. a performance evaluation unit that calculates an evaluation index of a machine learning model defined by the k hyperparameters output as the search results; a user adjustment unit that allows a user to adjust the k hyperparameters output as the search results according to the evaluation index; 2. The hyperparameter search system of claim 1, further comprising:

8. 8. The hyperparameter search system according to claim 7, wherein the user adjustment unit newly registers the adjusted k hyperparameters and the evaluation index as the case data.

9. In the hyperparameter search program, A first step of setting N (k>N≧2) groups for k (k≧3) hyperparameters to be searched based on correlations between hyperparameters in past case data related to hyperparameter search; a second step of performing a hyperparameter search on a group-by-group basis using a predetermined search method; a third step of summarizing the optimal values ​​obtained by the group-by-group hyperparameter search and outputting them as search results for the k hyperparameters; A hyperparameter search program that causes a computer to execute a process comprising:

10. The first step comprises:

10. The hyperparameter search program according to claim 9, further comprising a step of calculating a correlation coefficient for each pair of hyperparameters based on related case data extracted from the case data as being related to the k hyperparameters, thereby obtaining a group of correlation coefficients for the related case data.

11. 11. The hyperparameter search program according to claim 10, wherein the related case data is extracted based on predetermined search conditions and the group of hyperparameters matches the k hyperparameters.

12. The first step comprises: The hyperparameter search program according to claim 10, wherein the k hyperparameters are grouped based on the results of comparing each of the correlation coefficients of the related case data with a predetermined threshold value.

13. The first step comprises: The hyperparameter search program according to claim 10, wherein the k hyperparameters are grouped by clustering the same or similar correlation coefficients among the group of correlation coefficients of the related case data.

14. The first step comprises: generating a correlation screen in which the group of hyperparameters, evaluation indicators of a machine learning model defined by the group of hyperparameters, and the group of correlation coefficients are associated with each other for the related case data; A step of grouping the hyperparameters based on the hyperparameters designated as having correlation by a user who viewed the correlation screen; 11. The hyperparameter search program according to claim 10, further comprising:

15. a fourth step of calculating an evaluation index of the machine learning model defined by the k hyperparameters output as the search results; a fifth step of allowing a user to adjust the k hyperparameters output as the search results in accordance with the evaluation index; 10. The hyperparameter search program according to claim 9, further comprising:

16. 16. The hyperparameter search program according to claim 15, further comprising a sixth step of newly registering the adjusted k hyperparameters and the evaluation index as the case data.

Images