Learning device, learning method, and program

Abstract

This training device comprises: a reference setting means that uses the value of an evaluation function indicating an evaluation of the accuracy of estimation by a model generated in machine learning to set a reference value specifying how much improvement in the accuracy of estimation by the model is to be achieved through further machine learning; and a training means that trains the model so as to cause the output value of the evaluation function to approach the reference value.

Description

[Technical Field]

[0001] The present invention relates to a learning device. , studies Regarding learning methods and programs. [Background technology]

[0002] One method for setting hyperparameters of a machine learning model is to perform learning with each of multiple hyperparameter values ​​and select one of the multiple hyperparameter values ​​based on the learning results. For example, the parameter adjustment device described in Patent Document 1 sets multiple parameter values ​​for one of two hyperparameters, hyperparameter A, and sets a fixed value for the other hyperparameter, hyperparameter B. The parameter adjustment device transmits combinations of each value of hyperparameter A and a fixed value of hyperparameter B to the learning device, obtains the correct answer rate for each combination, and approximates the relationship between the value of hyperparameter A and the correct answer rate using a function. The parameter adjustment device also approximates the relationship between the value of hyperparameter A and the correct answer rate for other values ​​of hyperparameter B using a function, and finds the combination of the value of hyperparameter A and the value of hyperparameter B that results in the highest correct answer rate. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-15992 Summary of the Invention [Problem to be solved by the invention]

[0004] When training a model, it is possible to set hyperparameters to allow for a wide range of possible parameter values ​​to be explored, but it is difficult to set the hyperparameter values ​​appropriately.

[0005] An example of the object of this disclosure is a learning device that can solve the above-mentioned problems. , studies The purpose of this project is to provide learning methods and programs. [Means for solving the problem]

[0006] According to a first aspect of the present invention, a learning device includes a standard setting means for setting a standard value for the degree of estimation accuracy to be achieved in further machine learning based on an evaluation function value indicating an evaluation of the estimation accuracy of a model generated in machine learning, and a learning means for performing machine learning to update the model so as to bring the output value of the evaluation function closer to the standard value.

[0008] The present invention two According to this aspect, the learning method includes a computer setting a reference value for the degree of estimation accuracy to be achieved in further machine learning based on an evaluation function value indicating an evaluation of the estimation accuracy of a model generated in machine learning, and learning the model so that the output value of the evaluation function approaches the reference value.

[0009] The present invention three According to this aspect, the program causes a computer to set a reference value for the degree of estimation accuracy to be achieved in further machine learning based on an evaluation function value indicating an evaluation of the estimation accuracy of a model generated in machine learning, and to learn the model so as to bring the output value of the evaluation function closer to the reference value. [Effects of the Invention]

[0010] According to the present invention, it is possible to set the values ​​of hyperparameters for adjusting the breadth of the search for parameter values ​​relatively easily. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a learning device according to an embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of division of learning data by a data acquisition unit according to the embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of robustness against changes in parameter values. [Figure 4] FIG. 10 is a diagram illustrating a first example of a processing procedure in which the learning device according to the embodiment learns a model. [Figure 5] 10A and 10B are diagrams illustrating examples of start and end conditions for learning using a reference value according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating a second example of a processing procedure in which the learning device according to the embodiment learns a model. [Figure 7] FIG. 10 is a diagram illustrating a third example of a processing procedure in which the learning device according to the embodiment learns a model. [Figure 8] FIG. 1 is a diagram illustrating an example of the configuration of an estimation device according to an embodiment. [Figure 9] FIG. 10 is a diagram illustrating another example of the configuration of the learning device according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a processing procedure in a learning method according to an embodiment. [Figure 11] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Fig. 1 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. In the configuration shown in Fig. 1, learning device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a control unit 190. Storage unit 180 includes a model storage unit 181. Control unit 190 includes a data acquisition unit 191, a reference setting unit 192, and a learning unit 193.

[0013] The learning device 100 performs model learning. The learning device 100 may be configured using a computer such as a personal computer (PC) or a workstation. The communication unit 110 communicates with other devices. For example, the communication unit 110 may communicate with a device that stores learning data and receive the learning data.

[0014] The display unit 120 includes a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. The operation input unit 130 includes input devices such as a keyboard and a mouse, and receives user operations.

[0015] For example, for hyperparameters for which the user sets values, the display unit 120 may display a hyperparameter value input screen, and the operation input unit 130 may accept a user operation to input the hyperparameter value.

[0016] The storage unit 180 stores various data. The storage unit 180 is configured using a storage device included in the learning device 100. The model storage unit 181 stores a model to be learned. However, the model to be learned by the learning device 100 is not limited to the one stored in the model storage unit 181. For example, the model to be learned by the learning device 100 may be implemented using hardware and configured as a device separate from the learning device 100. In this case, the storage unit 180 may be configured without the model storage unit 181.

[0017] Control unit 190 performs various processes by controlling each unit of learning device 100. The functions of control unit 190 may be performed, for example, by a CPU (Central Processing Unit) included in learning device 100 reading and executing a program from storage unit 180.

[0018] The data acquiring unit 191 acquires learning data. For example, when the communication unit 110 receives learning data from another device, the data acquiring unit 191 extracts the learning data from the data received by the communication unit 110. The data acquiring unit 191 also divides the acquired learning data into training data, confirmation data, and test data. The following describes an example in which the data acquisition unit 191 acquires supervised sample data as learning data. The supervised sample data here can be a set whose elements are combinations of inputs to a model and correct answers that are the outputs of the model in response to those inputs.

[0019] Fig. 2 is a diagram showing an example of division of learning data by the data acquisition unit 191. In the example of Fig. 2, the data acquisition unit 191 divides the supervised sample data acquired as learning data into training data, confirmation data, and test data. Specifically, the data acquisition unit 191 divides multiple elements, which are combinations of inputs to a model and correct answers that are the output of the model for those inputs, included in the supervised sample data into training data elements, confirmation data elements, and test data elements.

[0020] Training data is used to adjust the parameter values ​​of a model. A parameter here refers to a variable included in the model whose value is updated by a learning algorithm. For example, when training a perceptron-type neural network using backpropagation, the weight coefficients between nodes and the bias at each node are examples of parameters.

[0021] The confirmation data is used to adjust the values ​​of hyperparameters in model training. Hyperparameters here refer to parameters related to model training, such as parameters for setting the behavior of the learning algorithm, other than parameters whose values ​​are subject to update by the learning algorithm. For example, when training a perceptron-type neural network using backpropagation, the learning rate is an example of a parameter. Furthermore, when the structure of a neural network is variable, values ​​related to the neural network structure, such as the number of hidden layers and the number of nodes per layer, can also be treated as hyperparameters.

[0022] For example, the learning device 100 may set multiple values ​​for the hyperparameters and use the confirmation data to select one of the hyperparameter values. In this case, the learning device 100 may adjust the parameter values ​​for each of the set hyperparameter values ​​using the training data, and then evaluate the model using the confirmation data. The learning device 100 may then select the hyperparameter value with the highest evaluation. The reference value of the evaluation function value, which will be described later, can also be treated as a hyperparameter. However, as will be described later, the learning device 100 sets only one value at a time for the reference value of the evaluation function value.

[0023] The test data is used as data for evaluating a model obtained by training. For example, training device 100 may evaluate a model obtained by training using the test data. Then, training device 100 or a user may decide, based on the evaluation results, whether to adopt the obtained model or to retrain the model.

[0024] Training data and validation data can be said to be the data used to build a model, and test data can be said to be the data used to evaluate a model. Furthermore, training data can be said to be data for updating parameter values ​​of a model, and verification data and test data can be said to be data other than data for updating parameter values ​​of a model.

[0025] However, the division of the supervised sample data performed by the data acquisition unit 191 is not limited to division into training data, verification data, and test data. For example, when a model is tested through trial operation in an actual usage environment, it is not necessary to secure test data from the supervised sample data. In this case, the data acquisition unit 191 may divide the acquired supervised sample data into training data and verification data.

[0026] The standard setting unit 192 sets a standard value of the evaluation function value for the evaluation function used in model learning. The standard setting unit 192 corresponds to an example of standard setting means. The evaluation function here is a function that indicates an evaluation of the estimation accuracy of the model. The estimation accuracy of the model indicated by the evaluation function can be said to be the degree of fit of the model to the data input to the model. A high evaluation indicated by the evaluation function can also be said to be a high degree of fit of the model to the input data.

[0027] Therefore, the evaluation function can also be said to be a function that indicates the degree of fit of the model to the data input to the model. For example, when training data is input to the model, the evaluation function value can be said to indicate the degree of fit of the model to the training data.

[0028] In the following, an example will be described in which the learning device 100 uses, as an evaluation function, a function in which the smaller the value, the higher the evaluation, such as an error function or a cross entropy loss function. In this case, the smaller the value of the evaluation function, the better the fit of the model to the input data.

[0029] In the following, an example in which the minimum value of the evaluation function value is 0 will be described. The error here may be a value indicated by "1-precision." Alternatively, the loss or error here may be a value based on the distance between the model's output value and the correct answer, such as L1 loss or L2 loss. However, there is no particular limitation to the evaluation function used by the learning unit 193. For example, the learning unit 193 may use, as an evaluation function, a function that indicates a higher evaluation as the evaluation function value increases.

[0030] The reference value here is a value that represents the standard for the degree of estimation accuracy to be achieved for training data in model learning. The reference value can also be said to be a value that specifies the degree of fit of the model to the training data. In other words, the reference value can be said to represent a standard for preventing overfitting. The closer the reference value is to 0, the more the model that is generated will fit the training data.

[0031] The learning unit 193 learns the model. In particular, the learning unit 193 learns the model so as to bring the evaluation function value as close as possible to the reference value. The learning unit 193 is an example of a learning means.

[0032] The standard setting unit 192 sets the evaluation function value indicating that the degree of fit of the model to the training data is more limited than the maximum degree of fit as the standard value. A smaller evaluation function value indicates a higher degree of fit, and when the minimum evaluation function value is 0, the standard setting unit 192 sets a value greater than 0 as the standard value of the evaluation function value. The learning unit 193 learns the model so that the evaluation function value approaches the standard value as much as possible.

[0033] The reference value set by the reference setting unit 192 is used as a hyperparameter for adjusting the breadth of the search for parameter values. When the reference value is relatively small (i.e., close to 0), it is considered that the learning unit 193 is relatively unlikely to search for another local solution after reaching a local solution where the evaluation function value is the reference value or a value close to it in the search for parameter values. On the other hand, when the reference value is relatively large, it is considered that the learning unit 193 is relatively likely to search for another local solution after reaching a local solution where the evaluation function value is the reference value or a value close to it in the search for parameter values. The learning unit 193 may use a solution search method that determines the next search point probabilistically, such as a stochastic gradient method, so that it can search for another local solution after reaching a local solution in the search for parameter values.

[0034] It is expected that overfitting can be prevented by the learning unit 193 learning a model using a reference value of the evaluation function value. By preventing overfitting, for example, it is expected that the learning unit 193 can obtain parameter values ​​that are relatively robust against changes in the parameter values. Here, being robust against changes in the parameter values ​​means that the evaluation function value does not increase significantly even if the parameter values ​​change slightly.

[0035] Fig. 3 is a diagram showing an example of robustness against changes in parameter values. The horizontal axis of the graph in Fig. 3 represents the parameter value, and the vertical axis represents the evaluation function value. Fig. 3 shows an example in which the evaluation function value is minimized when the value of the parameter w is w1. Line L11 shows an example where the parameter value w1 is relatively robust to changes in the parameter value, while line L12 shows an example where the parameter value w1 is relatively not robust to changes in the parameter value.

[0036] Comparing the case of line L11 with the case of line L12, the increase in the evaluation function value when the value of the parameter w changes slightly is smaller in the case of line L11. From this, it is expected that when learning progresses and new data is presented, it is easier to obtain parameter values ​​that are suitable for the new data when the learning unit 193 obtains the local solution for the parameter values ​​indicated by line L11 than when the local solution indicated by line L12 is obtained. In this way, parameter values ​​that are robust to changes in parameter values ​​are expected to result in a model with higher accuracy as learning progresses than parameter values ​​that are not robust to changes in parameter values.

[0037] It is expected that overfitting can be prevented by the learning unit 193 learning a model using the reference value of the evaluation function value. On the other hand, if the reference value is too large, it may be impossible to obtain a model with a high evaluation indicated by the evaluation function value, that is, the accuracy of the model obtained by learning may be low.

[0038] Regarding the setting of the reference value, the reference setting unit 192 sets the reference value based on the evaluation function value obtained by the learning up to that point. For example, the reference value for a new epoch may be set by the reference setting unit 192 to an evaluation function value obtained by applying the confirmation data to the model in a past epoch. The evaluation function value obtained by applying data such as the confirmation data to the model may be an average value of the evaluation function values ​​obtained for each element included in the data.

[0039] The epoch or one epoch referred to here is one model learning that the learning unit 193 repeatedly performs using the same supervised sample data. The number of times an epoch is repeated is also referred to as the number of epochs. An epoch corresponds to an example of one unit of model learning that the learning unit 193 repeatedly performs.

[0040] When the learning unit 193 uses a loss function as an evaluation function, the evaluation function value obtained by applying training data to the model is also referred to as training loss, the evaluation function value obtained by applying verification data to the model is also referred to as verification loss, and the evaluation function value obtained by applying test data to the model is also referred to as test loss.

[0041] When the learning unit 193 uses an error function as the evaluation function, the evaluation function value obtained by applying the training data to the model is also referred to as the training error, the evaluation function value obtained by applying the verification data to the model is also referred to as the verification error, and the evaluation function value obtained by applying the test data to the model is also referred to as the test error.

[0042] The criterion setting unit 192 may set any one of the training loss, validation loss, test loss, training error, validation error, or test error as the criterion value of the evaluation function. Alternatively, the reference value setting unit 192 may set a value obtained by calculation based on any of the training loss, validation loss, test loss, training error, validation error, or test error, such as a value obtained by multiplying the training loss by a predetermined coefficient, as the reference value of the evaluation function. Alternatively, the reference value setting unit 192 may set a value obtained by calculation based on a combination of two or more indicators of training loss, validation loss, test loss, training error, validation error, or test error, such as the average value of training loss and validation loss, as the reference value of the evaluation function.

[0043] The standard setting unit 192 may set an evaluation function including the standard value, thereby setting the standard value. For example, the standard setting unit 192 may set an evaluation function J * (g) may be set.

[0044]

number

[0045] g represents the model to be learned. J(g) represents the original evaluation function (the evaluation function without the reference value). b represents the reference value. "||" represents the absolute value. When J(g) ≥ b, J * (g) = J(g). In this case, the learning unit 193 performs the search for parameter values in the same way as when using J(g) as the evaluation function in the learning using the evaluation function J * (g).

[0046] On the other hand, when J(g) < b, J * (g) = 2b - J(g). In this case, the sign of the term "-J(g)" is negative, and the evaluation function J * (g) has a gradient opposite to that of J(g). In a solution search algorithm based on a gradient method such as the error backpropagation method, when using J(g) as the evaluation function, the learning unit 193 performs the search for parameter values so as to approximate the value of the evaluation function J(g) to the minimum value 0 as much as possible. On the other hand, when using J * (g) as the evaluation function, the learning unit 193 performs the search for parameter values so as to approximate the value of the evaluation function J * (g) to the reference value b as much as possible.

[0047] The evaluation function J * (g) outputs the same value as the output value of the evaluation function J(g) in the domain of the evaluation function J(g) where the output value of the evaluation function J(g) is equal to the reference value b or the output value of the evaluation function J(g) is greater than the reference value b. On the other hand, the evaluation function J * (g) outputs a value greater than the output value of the evaluation function J(g) in the domain of the evaluation function J(g) where the output value of the evaluation function J(g) is less than the reference value b. The evaluation function J * (g) is also referred to as a restricted evaluation function.

[0048] The standard setting unit 192 may set the evaluation function value of an epoch in which the evaluation function value satisfies a predetermined standard, among the epochs already executed by the learning unit 193, as the standard value for the next epoch. In this case, the evaluation function value referred to for selecting the epoch and the evaluation function value set as the standard value may be evaluation function values ​​of different data.

[0049] For example, the reference setting unit 192 may select an epoch whose confirmation error satisfies a predetermined criterion. In this case, the predetermined criterion may be that the confirmation error for each epoch is ranked within a predetermined order of precedence when the confirmation errors for the epochs are sorted in ascending order. The reference setting unit 192 may set the reference value based on the training error for the selected epoch. Alternatively, the reference setting unit 192 may set the reference value to a value obtained by calculation using the training error, such as adding a predetermined value to the training error or subtracting a predetermined value from the training error.

[0050] For example, the reference setting unit 192 may set the evaluation function value of the epoch with the smallest evaluation function value among the epochs already executed by the learning unit 193 as the reference value for the next epoch. In this case, the evaluation function value referred to for selecting the epoch and the evaluation function value set as the reference value may be evaluation function values ​​of different data.

[0051] Furthermore, for example, the reference setting unit 192 may select the epoch with the smallest training error from among the epochs already executed by the learning unit 193, and set the verification error in that epoch as the reference value for the next epoch. In this case, the reference setting unit 192 can set the reference value by referring to good learning results, in that it selects the epoch with the smallest training error. Also, since it is generally thought that the validation error is larger than the training error, the reference setting unit 192 sets a relatively large reference value. By having the learning unit 193 learn the model based on this reference value, it is expected that overfitting will be less likely to occur.

[0052] Alternatively, the reference setting unit 192 may select the epoch with the smallest verification error from among the epochs already executed by the learning unit 193, and set the training error in that epoch as the reference value for the next epoch. In an epoch with a small confirmation error, the generalization performance of the obtained model is expected to be relatively high. The reference setting unit 192 sets the training error in the epoch with the smallest confirmation error as the reference value, thereby setting the training error when a model with relatively high generalization performance is obtained as the reference value. By having the learning unit 193 learn the model based on this reference value, it is expected that it will be easier to search for a solution that will result in a model with relatively high generalization performance, and in this respect, it is expected that overfitting will be avoided.

[0053] Each time the learning unit 193 performs learning for one epoch, the reference setting unit 192 may determine whether or not to update the reference value. FIG. 4 is a diagram illustrating a first example of a processing procedure in which the learning device 100 learns a model. In the processing of FIG. 4, the learning unit 193 performs learning for the first epoch (step S101). In the learning for the first epoch, the learning unit 193 has not yet executed any epochs, and the reference setting unit 192 has not set a reference value. Therefore, the learning unit 193 performs model learning without setting a reference value. When no reference value is set, the learning unit 193 performs model learning so as to bring the evaluation function value closer to the minimum value.

[0054] When the learning unit 193 finishes learning in the first epoch, the reference setting unit 192 sets a reference value for the evaluation function value based on the learning result in the first epoch (step S102). The learning unit 193 performs learning for one epoch using the reference value set by the reference setting unit 192 (step S103).

[0055] Next, the reference setting unit 192 determines whether the index value in the epoch most recently executed by the learning unit 193 is the smallest value among the index values ​​in the epochs that the learning unit 193 has already executed (step S104). As described above, the index value here may be any of the training loss, validation loss, test loss, training error, validation error, or test error. Furthermore, the index value used by the reference setting unit 192 for the determination in step S104 may be different from the index value used to set the reference value.

[0056] If the learning unit 193 determines that the index value in the most recently executed epoch is the smallest value among the index values ​​in the epochs that the learning unit 193 has already executed (step S104: YES), the learning unit 193 updates the reference value to a value obtained from the learning result in the most recent epoch (step S111).

[0057] Next, the learning unit 193 determines whether a predetermined learning termination condition is met (step S112). The learning termination condition here is a condition for the learning unit 193 to determine whether to terminate model learning. The learning termination condition here is not limited to a specific condition. For example, the learning termination condition may be a condition that the learning unit 193 has completed learning for a predetermined number of epochs. Alternatively, the learning termination condition may be a condition that the error of the obtained model is equal to or less than a predetermined error threshold.

[0058] If the learning unit 193 determines that the learning end condition is not met (step S112: NO), the process returns to step S103. On the other hand, if the learning unit 193 determines that the learning end condition is met (step S112: YES), the learning device 100 ends the processing of FIG.

[0059] On the other hand, in step S104, if the standard setting unit 192 determines that the index value in the epoch most recently executed by the learning unit 193 is not the smallest index value among the index values ​​in the epochs that the learning unit 193 has already executed (step S104: NO), the processing proceeds to step S112.

[0060] It is conceivable that the performance of the model will not be stable in the early stages of model learning. Therefore, the learning unit 193 may perform learning using a reference value after learning has progressed to a certain extent. For example, as will be described later, until a predetermined reference value use start condition is met, the learning unit 193 may perform model learning so as to bring the evaluation function value closer to 0 without using a reference value. After the reference value use start condition is met, the learning unit 193 may perform model learning so as to bring the evaluation function value closer to the reference value. The reference value use start condition here is a condition for determining when the learning unit 193 starts learning using a reference value, or a condition for determining whether the learning unit 193 will perform learning using a reference value.

[0061] Furthermore, the learning unit 193 may perform learning without using a reference value in the final stage of model learning to improve the performance of the model. For example, the learning unit 193 may perform learning for the last 100 epochs without using a reference value. For example, as described below, the learning unit 193 may perform model learning so as to bring the evaluation function value closer to the reference value until a predetermined reference value use termination condition is met. After the reference value use termination condition is met, the learning unit 193 may perform model learning so as to bring the evaluation function value closer to 0 without using the reference value. The reference value use termination condition here is a condition for the learning unit 193 to determine when to terminate learning using a reference value, or a condition for the learning unit 193 to determine whether to terminate learning using a reference value.

[0062] Figure 5 shows examples of start and end conditions for learning using a reference value. The horizontal axis of the graph in Figure 5 represents the number of epochs. The vertical axis represents the error. Line L21 shows an example of the training error. Line L22 shows an example of the validation error. When the training error becomes equal to or smaller than the threshold value Et, the reference setting unit 192 may set the verification error in the epoch in which the training error is smallest as the reference value. The learning unit 193 may perform learning without setting a reference value in the epoch before the reference value is set by the reference setting unit 192, and may perform learning based on the reference value in the epoch in which the reference value is set by the reference setting unit 192. Alternatively, the learning unit 193 may perform learning without setting a reference value after the number of epochs reaches M, and terminate learning when the number of epochs reaches N. Here, M and N are both positive integers, and M <Nである。

[0063] FIG. 6 is a diagram illustrating a second example of the processing procedure in which the learning device 100 learns a model. In the process of FIG. 6, the learning unit 193 performs learning for one epoch without setting a reference value (step S201). Next, the reference setting unit 192 determines whether a predetermined reference value use start condition is satisfied (step S202). The reference value use start condition here is not limited to a specific condition. For example, as in the example of FIG. 5, the reference value use start condition may be a condition that the training error is equal to or less than a predetermined threshold, but is not limited thereto.

[0064] If the reference setting unit 192 determines that the reference value use start condition is not met (step S202: NO), the process returns to step S201. On the other hand, if it is determined that the reference value use start condition is met (step S202: YES), the reference setting unit 192 sets a reference value (step S211). Specifically, the reference setting unit 192 selects an epoch with the smallest index value from among the epochs already executed by the learning unit 193, and sets a reference value based on the learning result of that epoch. As described above, the index value here may be any of a training loss, a validation loss, a test loss, a training error, a validation error, or a test error. Furthermore, the index value used by the reference setting unit 192 to select an epoch may be different from the index value used to set the reference value. The learning unit 193 performs learning for one epoch based on the reference value set by the reference setting unit 192 (step S212).

[0065] Next, the reference setting unit 192 determines whether the index value in the epoch most recently executed by the learning unit 193 is the smallest value among the index values ​​in the epochs that the learning unit 193 has already executed (step S213). As described above, the index value here may be any of the training loss, validation loss, test loss, training error, validation error, or test error. Furthermore, the index value used by the reference setting unit 192 for the determination in step S213 may be different from the index value used to set the reference value.

[0066] If the learning unit 193 determines that the index value in the most recently executed epoch is the smallest value among the index values ​​in the epochs that the learning unit 193 has already executed (step S213: YES), the learning unit 193 updates the reference value to a value obtained from the learning result in the most recent epoch (step S221).

[0067] Next, the learning unit 193 determines whether a predetermined reference value use end condition is satisfied (step S222). The reference value use end condition here is not limited to a specific condition. For example, as in the example of FIG. 5, the reference value use end condition may be a condition that the learning unit 193 has completed learning for a predetermined number of epochs, but is not limited to this.

[0068] If the learning unit 193 determines that the reference value use end condition is not met (step S222: NO), the process returns to step S103. On the other hand, if it is determined in step S222 that the reference value use termination condition is not satisfied (step S222: NO), the learning unit 193 determines whether a predetermined learning termination condition is satisfied (step S231). The learning termination condition here is not limited to a specific condition. For example, the learning termination condition may be a condition that the learning unit 193 has completed learning for a predetermined number of epochs. Alternatively, the learning termination condition may be a condition that the error of the obtained model is equal to or less than a predetermined error threshold.

[0069] If the learning unit 193 determines that the learning end condition is not met (step S231: NO), the process returns to step S212. On the other hand, if the learning unit 193 determines that the learning end condition is met (step S231: YES), the learning device 100 ends the processing of FIG.

[0070] On the other hand, if it is determined in step S222 that the reference value use termination condition is met (step S222: YES), the learning unit 193 determines whether a predetermined learning termination condition is met (step S241). The determination made by the learning unit 193 in step S241 is the same as that in step S231.

[0071] If it is determined that the learning end condition is not met (step S241: NO), the learning unit 193 performs learning without setting a reference value for one epoch (step S251). After step S251, the process returns to step S241. On the other hand, if the learning unit 193 determines in step S241 that the learning end condition is met (step S241: YES), the learning device 100 ends the processing of FIG.

[0072] After the learning unit 193 has performed learning for a predetermined number of epochs, the reference setting unit 192 may set a reference value, and the learning unit 193 may perform learning for an additional predetermined number of epochs based on the reference value. For example, after the learning unit 193 has performed learning for 500 epochs, the reference setting unit 192 may set a reference value. Then, the learning unit 193 may perform learning for an additional 500 epochs based on the reference value.

[0073] In this case, the reference setting unit 192 may set the training loss at the epoch at which the error based on the confirmation data is smallest (i.e., the epoch at which the accuracy based on the confirmation data is largest) as the reference value. The epoch at which the error based on the confirmation data is smallest is considered to correspond to the point just before overfitting occurs, when the training loss becomes zero. By having the reference setting unit 192 set the training loss at this epoch as the reference value and having the learning unit 193 perform learning based on the reference value, it is expected that overfitting can be avoided.

[0074] FIG. 7 is a diagram illustrating a third example of the processing procedure in which the learning device 100 learns a model. In the process of FIG. 7, the learning unit 193 performs learning without setting a reference value for a predetermined number of epochs (step S301). Next, the reference setting unit 192 sets a reference value (step S302). Specifically, the reference setting unit 192 selects an epoch with the smallest index value from among the epochs already executed by the learning unit 193, and sets the reference value based on the learning result of that epoch. As described above, the index value here may be any of a training loss, a validation loss, a test loss, a training error, a validation error, or a test error. Furthermore, the index value used by the reference setting unit 192 to select an epoch may be different from the index value used to set the reference value. The learning unit 193 performs learning for a predetermined number of epochs based on the reference value set by the reference setting unit 192 (step S303). After step S303, the learning device 100 ends the processing in FIG.

[0075] Alternatively, after step S303, the learning device 100 may further set a reference value and perform learning based on the reference value. For example, as in step S302, the reference setting unit 192 selects the epoch with the smallest index value from the epochs already executed by the learning unit 193, and sets a reference value based on the learning result of that epoch. As in step S303, the learning unit 193 performs learning for a predetermined number of epochs based on the reference value set by the reference setting unit 192. For example, the user may instruct learning device 100 whether or not to further set a reference value and perform learning based on the reference value.

[0076] As described above, the standard setting unit 192 sets a standard value for the degree of estimation accuracy to be achieved in further machine learning, based on the evaluation function value indicating the estimation accuracy of the model generated in machine learning. The learning unit 193 learns the model so as to bring the output value of the evaluation function closer to the standard value.

[0077] According to the learning device 100, the use of evaluation function values ​​makes it relatively easy to set the values ​​of hyperparameters for adjusting the breadth of the search for parameter values. Furthermore, in the learning device 100, the learning unit 193 learns the model based on the reference value, which is expected to prevent overfitting. Furthermore, the learning device 100 can set a single reference value at a time, which corresponds to an example of a hyperparameter value for adjusting the breadth of parameter value search. In particular, the learning device 100 does not require the operation of setting multiple reference values, learning a model, and selecting one of the reference values ​​based on the learning results. In this respect, the learning device 100 is expected to be able to relatively shorten the time required for learning without requiring computational resources such as parallel processing. Furthermore, according to the learning device 100, the standard setting unit 192 sets a standard value based on the evaluation function value obtained by learning, so that the standard value can be set according to the model and learning situation, and in this respect, it is expected that a relatively appropriate standard value can be set.

[0078] Furthermore, the standard setting unit 192 sets the standard value to a value that indicates that the degree of fit of the model to the training data is more limited than the maximum degree of fit, among the values ​​that the evaluation function can take. According to the learning device 100, the possibility of overfitting can be reduced by learning using a reference value.

[0079] Furthermore, the reference setting unit 192 sets a reference value based on an evaluation function value obtained by applying confirmation data, which is data other than data (training data) for updating parameter values ​​of the model, to the model. For example, in an epoch in which the evaluation function value obtained by applying data other than the training data, such as the confirmation error, to the model is small, the generalization performance of the obtained model is expected to be relatively high. The reference setting unit 192 sets the training error in the epoch in which the confirmation error is small as the reference value, thereby setting the training error when a model with relatively high generalization performance is obtained as the reference value. By having the learning unit 193 learn the model based on this reference value, it is expected that it will be easier to search for a solution that will result in a model with relatively high generalization performance, and in this respect, it is expected that overfitting will be avoided.

[0080] Furthermore, after completing one epoch of model learning repeatedly performed by the learning unit 193, the standard setting unit 192 determines whether a learning result with a smaller evaluation function value than the learning result used to set the standard value has been obtained in the learning for that epoch. If it is determined that a learning result with a smaller evaluation function value has been obtained, the standard setting unit 192 updates the standard value based on the learning result for that epoch. This allows the standard setting unit 192 to update the standard value as the learning unit 193 progresses in learning the model, and is expected to be able to set an appropriate standard value according to the progress of learning.

[0081] Furthermore, the learning unit 193 performs model learning so as to bring the evaluation function value as close to 0 as possible until a predetermined reference value use start condition is met. This allows the standard setting unit 192 to set a standard value after learning has progressed to a certain extent and the accuracy of the model has stabilized. In this respect, the learning device 100 is expected to enable the standard setting unit 192 to set an appropriate standard value.

[0082] Furthermore, after a predetermined condition for ending the use of the reference value is met, the learning unit 193 performs model learning so as to bring the evaluation function value as close to 0 as possible. According to the learning device 100, learning can be performed without setting a reference value in the final stage of model learning, and in this respect, it is expected that the accuracy of the model can be made relatively high.

[0083] In addition, the standard setting unit 192 selects one of the epochs of the model learning performed by the learning unit 193 over a predetermined number of epochs based on the learning results, and sets a standard value based on the evaluation function value indicated in the learning results for the selected epoch. According to the learning device 100, the standard setting unit 192 only needs to set the standard value once, which is expected to make it possible to shorten the time required for learning relatively without requiring computational resources such as parallel processing. Also, since the standard setting unit 192 sets the standard value at a stage when model learning by the learning unit 193 has progressed to a certain extent, it is expected to be possible to set an appropriate standard value.

[0084] Furthermore, the criterion setting unit 192 generates a restricted evaluation function. The restricted evaluation function is a function that outputs a value equal to the output value of the evaluation function in a portion of the domain where the output value of the evaluation function is equal to or greater than the reference value, and outputs a value greater than the reference value in a portion where the output value of the evaluation function is smaller than the reference value. The learning unit 193 learns the model using the restricted evaluation function set by the criterion setting unit 192. In learning device 100, the reference value can be included in the restricted evaluation function, and learning unit 193 does not need to refer to the reference value separately from the evaluation function. In this respect, learning device 100 is expected to impose a relatively small load on learning unit 193.

[0085] FIG. 8 is a diagram illustrating an example of the configuration of an estimation device according to an embodiment. 8, the estimation device 200 includes a communication unit 210, a display unit 220, an operation input unit 230, a storage unit 280, and a control unit 290. The storage unit 280 includes a model storage unit 181. The control unit 290 includes a data acquisition unit 291 and an estimation unit 292.

[0086] The estimation device 200 performs estimation using a model trained by the learning device 100. The use of the estimation device 200 is not limited to a specific use. For example, the estimation device 200 may be configured as a face recognition device and may calculate the similarity between a face image to be authenticated and a registered face image using the model. Alternatively, the estimation device 200 may input a given sentence into the model and estimate the emotion indicated by the sentence. In this way, the estimation device 200 can be applied to various fields, such as computer vision or natural language processing.

[0087] The estimation device 200 may be configured using a computer such as a personal computer or a workstation. The estimation device 200 may be configured using the computer used as the learning device 100. Alternatively, the estimation device 200 may be configured using a computer different from the computer used as the learning device 100.

[0088] The communication unit 210 communicates with other devices. For example, the communication unit 210 may communicate with other devices to receive data to be estimated. The display unit 220 has a display screen such as a liquid crystal panel or an LED panel, and displays various images. For example, the display unit 220 may display the estimation result by the estimation device 200. The operation input unit 230 includes input devices such as a keyboard and a mouse, and receives user operations, including a user operation to instruct the start of estimation.

[0089] The storage unit 280 stores various data. The storage unit 280 is configured using a storage device included in the estimation device 200. The model storage unit 181 stores a model that has been learned by the learning device 100. In this respect, the model storage unit 181 of the estimation device 200 stores the same model as the model storage unit 181 of the learning device 100. For this reason, in FIG. 8, the reference numeral 181 is used for the model storage unit, as in FIG. 1.

[0090] Alternatively, when the model to be learned by learning device 100 is configured as a device separate from learning device 100, the model used by estimation device 200 may also be configured as a device separate from estimation device 200. In this case, storage unit 280 may be configured without including model storage unit 181.

[0091] The functions of the control unit 290, which controls each unit of the estimation device 200 and performs various processes, may be performed by, for example, a CPU included in the estimation device 200 reading and executing a program from the storage unit 280.

[0092] The data acquisition unit 291 acquires data to be inferred. For example, when the communication unit 210 receives data to be inferred from another device, the data acquisition unit 291 extracts the data to be inferred from the data received by the communication unit 210. The estimation unit 292 performs estimation on the estimation target acquired by the data acquisition unit 291. The estimation unit 292 inputs the data of the estimation target acquired by the data acquisition unit 291 into the model stored in the model storage unit 181, and acquires the output of the model as the estimation result.

[0093] As described above, the estimation unit 292 calculates an estimation value related to the estimation target using a trained model obtained by the learning device 100 learning the model. The trained model obtained by model training using the learning device 100 is set based on the value of an evaluation function that indicates an evaluation of the estimation accuracy of the model, and is an example of a model updated by machine learning that is performed so as to bring the output value of the evaluation function closer to a reference value that specifies the extent to which the estimation accuracy of the model should be achieved through further machine learning. It is expected that the model will not be overfitted in the estimation device 200. In this respect, it is expected that the estimation device 200 will be able to perform estimation with high accuracy.

[0094] 9 is a diagram showing another example of the configuration of the learning device according to the embodiment. In the configuration shown in FIG. In this configuration, the standard setting unit 611 sets a standard value for the degree of estimation accuracy to be achieved by further machine learning, based on the value of an evaluation function that indicates an evaluation of the estimation accuracy of a model generated in machine learning. The learning unit 612 updates the model by performing machine learning so that the output value of the evaluation function approaches the standard value. The reference setting unit 611 is an example of a reference setting means, and the learning unit 612 is an example of a learning means.

[0095] In the learning device 610, the learning unit 612 is expected to be able to avoid overfitting by learning the model based on the reference value. Furthermore, the learning device 610 can set a single reference value at a time, which corresponds to an example of a hyperparameter value for adjusting the breadth of parameter value search. In particular, the learning device 610 does not need to set multiple reference values, train a model, and select one of the reference values ​​based on the training results. In this respect, the learning device 610 is expected to be able to relatively shorten the time required for training without requiring computational resources such as parallel processing. Furthermore, according to the learning device 610, the standard setting unit 611 sets a standard value based on the evaluation function value obtained by learning, so that the standard value can be set according to the model and learning situation, and in this respect, it is expected that a relatively appropriate standard value can be set.

[0096] The reference setting unit 611 can be realized, for example, by using the function of the reference setting unit 192 in Fig. 1. The learning unit 612 can be realized, for example, by using the function of the learning unit 193 in Fig. 1.

[0097] FIG. 10 is a diagram illustrating an example of a processing procedure in the learning method according to the embodiment. The learning method shown in FIG. 10 includes setting a reference (step S611) and performing learning (step S612). In setting a standard (step S611), the computer sets a standard value for the degree of estimation accuracy of the model to be obtained through further machine learning, based on the value of an evaluation function that indicates an evaluation of the estimation accuracy of the model generated in machine learning. In learning (step S612), the computer learns the model so that the output value of the evaluation function approaches the reference value.

[0098] The learning method shown in Figure 10 is expected to avoid overfitting by learning the model based on a reference value. Furthermore, the learning method shown in Figure 10 allows for the setting of reference values ​​corresponding to example values ​​of hyperparameters for adjusting the breadth of parameter value search to be set to a single value at a time. In particular, the learning method shown in Figure 10 does not require the operation of setting multiple reference values, training a model, and selecting one of the reference values ​​based on the training results. In this respect, the learning method shown in Figure 10 is expected to be able to relatively shorten the time required for learning without requiring computational resources such as parallel processing. Furthermore, according to the learning method shown in Figure 10, by setting the reference value based on the evaluation function value obtained by learning, it is possible to set the reference value according to the model and learning situation, and in this respect, it is expected that a relatively appropriate reference value can be set.

[0099] FIG. 11 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 11, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0100] One or more of the learning device 100, the estimation device 200, and the learning device 610, or a part thereof, may be implemented in a computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by an interface 740 having a communication function and performing communication under the control of the CPU 710.

[0101] When learning device 100 is implemented in computer 700, the operations of control unit 190 and each of its components are stored in the form of a program in auxiliary storage device 730. CPU 710 reads the program from auxiliary storage device 730, loads it into main storage device 720, and executes the above-described processing in accordance with the program.

[0102] Furthermore, the CPU 710 allocates storage areas corresponding to the storage unit 180 and each unit thereof in the main storage device 720 in accordance with the program. Communication by the communication unit 110 is performed by the interface 740 having a communication function and performing communication under the control of the CPU 710. Display of images by the display unit 120 is performed by the interface 740 having a display device and displaying images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving the user operations.

[0103] When the estimation device 200 is implemented in a computer 700, the operations of the control unit 290 and each unit thereof are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0104] Furthermore, the CPU 710 allocates storage areas corresponding to the storage unit 280 and each unit thereof in the main storage device 720 in accordance with the program. Communication by the communication unit 210 is performed by the interface 740 having a communication function and performing communication under the control of the CPU 710. Display of images by the display unit 220 is performed by the interface 740 having a display device and displaying images under the control of the CPU 710. Reception of user operations by the operation input unit 230 is performed by the interface 740 having an input device and receiving user operations.

[0105] When the learning device 610 is implemented in the computer 700, the operations of the reference setting unit 611 and the learning unit 612 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0106] Furthermore, CPU 710 allocates a storage area in main memory 720 for the learning device 610 to perform processing in accordance with the program. Communication between learning device 610 and other devices is performed by interface 740, which has a communication function and operates under the control of CPU 710. Interaction between learning device 610 and a user is performed by interface 740, which has a display device and an input device, displaying various images under the control of CPU 710 and accepting user operations.

[0107] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. CPU 710 may then directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0108] Note that a program for executing all or part of the processing performed by learning device 100, estimation device 200, and learning device 610 may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to perform the processing of each unit. Note that the term "computer system" here includes hardware such as an OS (Operating System) and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The program may be one that realizes part of the aforementioned functions, or may be one that can realize the aforementioned functions in combination with a program already stored in the computer system.

[0109] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0110] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0111] (Appendix 1) a standard setting means for setting a standard value for the degree of estimation accuracy of the model to be obtained in further machine learning, based on a value of an evaluation function indicating an evaluation of the estimation accuracy of the model generated in machine learning; a learning means for updating the model by performing machine learning so as to bring the output value of the evaluation function closer to the reference value; A learning device comprising:

[0112] (Appendix 2) the standard setting means sets the standard value to a value that indicates that the degree of fit of the model to training data, which is data for updating parameter values ​​of the model, is more limited than the maximum degree of fit, among the values ​​that the evaluation function can take. 2. The learning device of claim 1.

[0113] (Appendix 3) the reference setting means sets the reference value based on an evaluation function value obtained by applying confirmation data, which is data other than data for updating parameter values ​​of the model, to the model. 10. The learning device according to claim 1 or 2.

[0114] (Appendix 4) the standard setting means, after completing one unit of learning of the model repeatedly performed by the learning means, determines whether or not a learning result indicating that the evaluation function value of the model has a higher degree of fit to the input data than the learning result used to set the standard value has been obtained in that one unit of learning, and updates the standard value based on the result of that one unit of learning if it is determined that a learning result indicating a higher degree of fit has been obtained. 4. A learning device according to any one of claims 1 to 3.

[0115] (Appendix 5) the learning means learns the model so as to bring the output value of the evaluation function closer to the evaluation function value showing the maximum degree of compatibility until a predetermined reference value use start condition is met. 5. The learning device of claim 4.

[0116] (Appendix 6) the learning means, after a predetermined condition for ending use of the reference value is satisfied, performs learning of the model so as to bring the output value of the evaluation function closer to an evaluation function value indicating a maximum degree of compatibility. 6. The learning device according to claim 4 or 5.

[0117] (Appendix 7) the reference setting means selects one of the epochs of the model learning performed by the learning means based on the learning result, and sets the reference value based on the evaluation function value indicated in the learning result of the selected epoch. 4. A learning device according to any one of claims 1 to 3.

[0118] (Appendix 8) the standard setting means generates a restricted evaluation function that outputs a value equal to the output value of the evaluation function in a portion of the domain of the evaluation function where the output value of the evaluation function is equal to the standard value or where the output value indicates that the fitness of the model to training data, which is data for updating parameter values ​​of the model, is smaller than the standard value, and that outputs a value indicating that the fitness is smaller than the standard value in a portion where the output value of the evaluation function indicates that the fitness is larger than the standard value; the learning means performs learning of the model using the restricted evaluation function. 8. A learning device according to any one of appendices 1 to 7.

[0119] (Appendix 9) An estimation unit that calculates an estimated value for an estimation target using the model that has been updated through machine learning, the estimate being set based on a value of an evaluation function that indicates an evaluation of the estimation accuracy of the model generated through machine learning, and that is performed so as to bring the output value of the evaluation function closer to a reference value that specifies the level of estimation accuracy to be achieved by further machine learning. An estimation device comprising:

[0120] (Appendix 10) The computer Based on the value of an evaluation function indicating an evaluation of the estimation accuracy of the model generated in machine learning, a reference value is set as to how much estimation accuracy of the model should be obtained in further machine learning; The model is trained so that the output value of the evaluation function approaches the reference value. A learning method that includes:

[0121] (Appendix 11) On the computer, Based on the value of an evaluation function indicating an evaluation of the estimation accuracy of the model generated in machine learning, a reference value is set for determining how much estimation accuracy the model should achieve in further machine learning; and learning the model so that the output value of the evaluation function approaches the reference value; A recording medium for recording a program for executing the above. [Industrial Applicability]

[0122] The present invention may be applied to a learning device, an estimation device, a learning method, and a recording medium. [Explanation of symbols]

[0123] 100, 610 Learning Device 110, 210 Communications Department 120, 220 display section 130, 230 Operation input section 180, 280 storage section 181 Model Memory Unit 190, 290 control section 191, 291 Data acquisition section 192, 611 Standard setting section 193, 612 Learning Department 292 Estimation Department

Claims

1. a standard setting means for setting a standard value for the degree of estimation accuracy of the model to be obtained in further machine learning, based on a value of an evaluation function indicating an evaluation of the estimation accuracy of the model generated in machine learning; a learning means for updating the model by performing machine learning so as to bring the output value of the evaluation function closer to the reference value; A learning device comprising:

2. the standard setting means sets the standard value to a value that indicates that the degree of fit of the model to training data, which is data for updating parameter values ​​of the model, is limited to less than the maximum degree of fit, among the values ​​that the evaluation function can take. The learning device according to claim 1 .

3. the reference setting means sets the reference value based on an evaluation function value obtained by applying confirmation data, which is data other than data for updating parameter values ​​of the model, to the model. The learning device according to claim 1 or 2.

4. the standard setting means, after completing one unit of learning of the model repeatedly performed by the learning means, determines whether or not a learning result indicating that the evaluation function value of the model has a higher degree of fit to the input data than the learning result used to set the standard value has been obtained in that one unit of learning, and updates the standard value based on the result of the one unit of learning if it is determined that a learning result indicating a higher degree of fit has been obtained. The learning device according to any one of claims 1 to 3.

5. the learning means learns the model so as to bring the output value of the evaluation function closer to the evaluation function value showing the maximum degree of compatibility until a predetermined reference value use start condition is met. The learning device according to claim 4 .

6. the learning means, after a predetermined condition for ending use of the reference value is satisfied, performs learning of the model so as to bring the output value of the evaluation function closer to an evaluation function value indicating a maximum degree of compatibility. The learning device according to claim 4 or 5.

7. the reference setting means selects one of the epochs of the model learning performed by the learning means based on the learning result, and sets the reference value based on the evaluation function value indicated in the learning result of the selected epoch. The learning device according to any one of claims 1 to 3.

8. The computer Based on the value of an evaluation function indicating an evaluation of the estimation accuracy of the model generated in machine learning, a reference value is set as to how much estimation accuracy of the model should be obtained in further machine learning; The model is trained so that the output value of the evaluation function approaches the reference value. A learning method that includes:

9. On the computer, Based on the value of an evaluation function indicating an evaluation of the estimation accuracy of the model generated in machine learning, a reference value is set for determining how much estimation accuracy the model should achieve in further machine learning; and learning the model so that the output value of the evaluation function approaches the reference value; A program to execute.

Images