Evaluation apparatus, evaluation method, and evaluation program
The evaluation device and method address the limitations of existing reliability evaluation by generating a second prediction function to assess the uncertainty of machine learning models, ensuring accurate reflection of confidence and prediction intervals.
Patent Information
- Application Number
- JP2023079867
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing methods for evaluating reliability in machine learning models fail to account for the true function representing the relationship between input data and correct data, as well as the noise contained in the data.
An evaluation device and method that utilizes data-based machine learning to generate a second prediction function, different from a first prediction function, to approximate a second variable, calculating index values to determine the uncertainty of the learning model, including the upper and lower limits of the prediction interval based on cumulative distribution functions.
Enables accurate evaluation of the uncertainty of a learning model, ensuring that the uncertainty correctly reflects the confidence and prediction intervals, even when the first prediction function is expressed with high explanatory properties at the expense of prediction accuracy.
Smart Images

Figure 2024164413000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an evaluation device, an evaluation method, and an evaluation program. [Background technology]
[0002] Patent Document 1 discloses a technology for evaluating the reliability of inference when learning using input data and correct answer data as input. In particular, according to this technology, multiple clusters are extracted by clustering the input data at the time of inference based on the result of clustering based on information on the input data. Then, a reliability evaluation value for evaluating the reliability of inference is determined based on the evaluation result and information on the multiple extracted clusters. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2022-088174 Summary of the Invention [Problem to be solved by the invention]
[0004] According to the technology described in Patent Document 1, reliability is calculated based on the distance between data in the input data. However, there is a problem in that the true function that expresses the relationship between the input data and the correct data, and the noise contained in the data are not evaluated.
[0005] The present disclosure has been made in consideration of the above problems. An object of the present disclosure is to provide an evaluation device, an evaluation method, and an evaluation program capable of evaluating the uncertainty of a learning model generated by any machine learning. [Means for solving the problem]
[0006] The evaluation device according to the present disclosure includes an input unit and a controller. The input unit acquires a plurality of data sets each including a first variable and a second variable, and a first prediction function generated based on the data and approximating the second variable corresponding to the first variable. The controller performs machine learning based on the data to generate a second prediction function different from the first prediction function and approximating the second variable corresponding to the first variable.
[0007] The first index value is an index indicating the error between the second variable and the second prediction function. The second index value is an index indicating the error between the second prediction function and the first prediction function. The controller calculates the second index value when the absolute difference between the first calculated value and the second calculated value is minimum when the second index value is varied. Here, the first calculated value is obtained by subtracting the value at the upper limit from the value at the lower limit of a cumulative distribution function for the probability distribution of the second variable for the upper and lower limits of a prediction interval at a predetermined significance level for the second variable, which is determined based on the first index value and the second index value. The second calculated value is obtained by subtracting the predetermined significance level from 1.
[0008] The first variable may be represented as x, the first prediction function as F(x), the second prediction function as G(x), and the predetermined significance level as α. The (1-α / 2) quantile in the probability distribution for variable t obtained by standardizing the second variable to have a standard deviation of 1 and a mean of 0 may be represented as k, and the cumulative distribution function for the probability distribution for variable t may be represented as Φ(t). The first index value may be represented as σ a (x), the second index value is σ e (x), with the upper limit set to PI + (x), with the lower limit being PI - It may also be written as (x).
[0009] The controller may calculate the upper and lower limits based on equation (1) and calculate the absolute difference based on equation (2).
[0010]
number
[0011] The first prediction function may be generated based on the data by machine learning other than deep learning.
[0012] The controller may perform deep learning based on the data to generate the second prediction function.
[0013] The controller may perform machine learning based on the data to generate the first index value.
[0014] The controller may generate as the first index value an unbiased standard deviation between the second variable and the second predictor function in a neighborhood of the value of the first variable.
[0015] The controller may calculate the first index value and the second index value when the absolute difference between the first index value and the second index value is minimum when the first index value and the second index value are varied.
[0016] The probability distribution may be a normal distribution.
[0017] The error between the second variable and the second prediction function may be smaller than the error between the second variable and the first prediction function.
[0018] The evaluation method according to the present disclosure controls a controller connected to an input unit. The input unit acquires a plurality of data sets each including a first variable and a second variable, and a first prediction function generated based on the data and approximating the second variable corresponding to the first variable. The controller performs machine learning based on the data to generate a second prediction function different from the first prediction function, which approximates the second variable corresponding to the first variable.
[0019] The first index value is an index indicating the error between the second variable and the second prediction function. The second index value is an index indicating the error between the second prediction function and the first prediction function. The controller calculates the second index value when the absolute difference between the first calculated value and the second calculated value is minimum when the second index value is varied. Here, the first calculated value is obtained by subtracting the value at the upper limit from the value at the lower limit of a cumulative distribution function for the probability distribution of the second variable for the upper and lower limits of a prediction interval at a predetermined significance level for the second variable, which is determined based on the first index value and the second index value. The second calculated value is obtained by subtracting the predetermined significance level from 1.
[0020] The evaluation program according to the present disclosure processes a plurality of data sets each including a first variable and a second variable, and a first prediction function generated based on the data and approximating a second variable corresponding to the first variable. The evaluation program causes a computer to execute a step of performing machine learning based on the data to generate a second prediction function different from the first prediction function that approximates the second variable corresponding to the first variable.
[0021] The first index value is an index indicating the error between the second variable and the second prediction function. The second index value is an index indicating the error between the second prediction function and the first prediction function. The computer executes a step of calculating a second index value when the absolute difference between the first calculated value and the second calculated value is minimum when the second index value is varied. Here, the first calculated value is obtained by subtracting the value at the lower limit from the value at the upper limit of a cumulative distribution function for the probability distribution of the second variable for the upper and lower limits of a prediction interval at a predetermined significance level for the second variable, which is determined based on the first index value and the second index value. The second calculated value is obtained by subtracting the predetermined significance level from 1. Effect of the Invention
[0022] According to the present disclosure, it is possible to provide an evaluation device, an evaluation method, and an evaluation program capable of evaluating the uncertainty of a learning model generated by any machine learning. [Brief description of the drawings]
[0023] [Figure 1] 1 is a block diagram showing a configuration of an evaluation device according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart showing a processing procedure of an evaluation device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] Hereinafter, some exemplary embodiments will be described with reference to the drawings. In addition, common parts in each drawing are given the same reference numerals, and duplicated explanations will be omitted.
[0025] [Evaluation device configuration] Fig. 1 is a block diagram showing a configuration of an evaluation device according to an embodiment of the present disclosure. As shown in Fig. 1, the evaluation device 20 includes an input unit 21 and a controller 25. In addition, the evaluation device 20 may include a database 23, an operation unit 27, and a display unit 29. The controller 25 is connected to the input unit 21, the database 23, the operation unit 27, and the display unit 29 so as to be able to communicate with them.
[0026] Additionally, the database 23 , the operation unit 27 , and the display unit 29 may be provided in the evaluation device 20 itself, or may be installed outside the evaluation device 20 and connected to the evaluation device 20 .
[0027] The input unit 21 acquires a plurality of data sets each including a first variable and a second variable. The input unit 21 also acquires a first prediction function that is generated based on the data and that approximates a second variable corresponding to the first variable. The evaluation device 20 may be shared with a computer that generates the plurality of data sets and the first prediction function. In that case, the evaluation device 20 may not include the input unit 21.
[0028] The first prediction function may be generated by machine learning based on a plurality of data. In particular, the first prediction function may be generated by machine learning different from deep learning. For example, the first prediction function that approximates a second variable corresponding to a first variable may be expressed by a learning model with high explanatory power. In this case, compared to a second prediction function described later, the first prediction function may be expressed by a learning model with high explanatory power at the expense of prediction accuracy.
[0029] The "explanatory ability" of a learning model means that the basis for associating a second variable with a first variable is expressed in a form that can be understood by humans in the learning model. For example, a learning model expressed by a neural network obtained by deep learning may have high prediction accuracy, but the meaning of the parameters included in the neural network may be unclear. Therefore, there may be cases where the "explanatory ability" is low in the neural network. In other words, according to the present disclosure, a first prediction function may be given that prioritizes the expression of the learning model in a form that can be understood by humans over prediction accuracy.
[0030] The database 23 stores the plurality of data acquired by the input unit 21 and the first prediction function. The database 23 may also store various parameters related to machine learning performed by the controller 25 described later, parameters defining a neural network expressing a learning model generated by machine learning, and the like.
[0031] The operation unit 27 is an input device that allows a user of the evaluation device 20 to perform operations. For example, the operation unit 27 is a keyboard, a mouse, a trackball, a touch panel, or the like. The operation unit 27 is not limited to the examples given here. The user's operation content input via the operation unit 27 is transmitted to the controller 25.
[0032] The display unit 29 displays the information received from the controller 25. For example, the display unit 29 may display an index related to machine learning performed by the controller 25, which will be described later.
[0033] The display unit 29 may be a display that displays figures and characters by combining a plurality of display pixels. The display unit 29 is not limited to the examples given here.
[0034] The controller 25 (control unit) is a general-purpose computer equipped with a CPU (Central Processing Unit), a memory, and an input / output unit. The controller 25 may be equipped with a GPU (Graphics Processing Unit). For example, the controller 25 may execute machine learning processing, which will be described later, using the GPU. A computer program (evaluation program) for functioning as the evaluation device 20 is installed in the controller 25. By executing the computer program, the controller 25 functions as a plurality of information processing circuits (251, 253, 255) equipped in the evaluation device 20.
[0035] In this disclosure, an example is shown in which multiple information processing circuits (251, 253, 255) are realized by software. However, it is also possible to configure the information processing circuits (251, 253, 255) by preparing dedicated hardware for executing each information process described below. Also, the multiple information processing circuits (251, 253, 255) may be configured by individual hardware.
[0036] As shown in FIG. 1, the controller 25 includes a prediction function generating unit 251, an index value setting unit 253, and an absolute difference calculating unit 255 as a plurality of information processing circuits (251, 253, 255).
[0037] The prediction function generating unit 251 performs machine learning based on data to generate a second prediction function that approximates a second variable corresponding to a first variable and is different from the first prediction function. For example, the prediction function generating unit 251 performs machine learning using data of a pair of the first variable and the second variable as teacher data to generate a learning model in which the first variable and the second variable are associated with each other. The prediction function generating unit 251 may perform deep learning based on data to generate the second prediction function.
[0038] For example, the learning model is configured by a neural network including an input layer and an output layer, and is learned by associating input data input to the input layer with output data output from the output layer. For example, a residual network is used as the neural network that configures the learning model.
[0039] Examples of techniques for generating a learning model include neural networks and support vector machines. Other examples include Random Forest, XGBoost, LightGBM, PLS regression, Ridge regression, and Lasso regression. In addition, a technique using one or a combination of two or more of the techniques listed here can be used. Techniques for generating a learning model by machine learning are not limited to the examples listed here.
[0040] A case will be described where the learning model is represented by a neural network. In this case, the prediction function generating unit 251 calculates the error or likelihood between the output obtained when input data is input to the neural network and the output data corresponding to the input data. Then, the prediction function generating unit 251 adjusts the parameters that define the neural network so that the error is minimized or the likelihood is maximized. As a result, the neural network learns the features that represent the teacher data.
[0041] The neural network includes an input layer to which input data is input, an output layer to which output values are output, and at least one or more hidden layers provided between the input layer and the output layer, and signals are propagated in the order of the input layer, hidden layer, and output layer. Each of the input layer, hidden layer, and output layer is composed of one or more units. The units between the layers are connected to each other, and each unit has an activation function (e.g., a sigmoid function, a normalized linear function, a softmax function, etc.). A weighted sum is calculated based on multiple inputs to the unit, and the value of the activation function with the sum value as a variable becomes the output of the unit.
[0042] The prediction function generating unit 251 adjusts the weights used in calculating the weighted sum for each unit among the parameters that define the neural network, thereby minimizing the error between the output of the neural network and the label, or maximizing the likelihood of the output. In order to minimize the error or maximize the likelihood of the output of the neural network for multiple teacher data, a maximum likelihood estimation method or the like can be applied.
[0043] In order to minimize an error related to the output of the neural network, for example, the prediction function generation unit 251 may use a gradient descent method, a stochastic gradient descent method, etc. The prediction function generation unit 251 may use an error backpropagation method for gradient calculation in the gradient descent method or the stochastic gradient descent method.
[0044] In machine learning using neural networks, generalization performance (the ability to discriminate against unknown data) and overfitting (the phenomenon in which a model adapts to training data but does not improve generalization performance) can become problems.
[0045] Therefore, in generating a learning model in the prediction function generating unit 251, a method such as regularization that restricts the degree of freedom of weights during learning may be used to alleviate overfitting. In addition, a method of probabilistically selecting units in a neural network and disabling other units may be used. Furthermore, in order to improve generalization performance, a method such as data regularization, data standardization, and data expansion that eliminate bias in teacher data may be used.
[0046] The prediction function generating unit 251 may generate a learning model using a support vector machine instead of a neural network. Machine learning using a support vector machine does not have the problem of local solution convergence and tends to improve generalization performance.
[0047] The learning model generated by the prediction function generation unit 251 is output to the index value setting unit 253 or the database 23.
[0048] When the prediction function generating unit 251 generates the second prediction function by performing deep learning, the generated second prediction function may have a higher prediction accuracy than the first prediction function. For example, the error between the second variable in the data and the second prediction function may be smaller than the error between the second variable in the data and the first prediction function.
[0049] The index value setting unit 253 sets a first index value which is an index indicating the error between the second variable and the second prediction function. Also, the index value setting unit 253 sets a second index value which is an index indicating the error between the second prediction function and the first prediction function.
[0050] It should be noted here that the first index value and the second index value fluctuate as the second prediction function changes. The index value setting unit 253 sets the first index value and the second index value based on the second prediction function generated by the prediction function generating unit 251.
[0051] When the prediction accuracy of the second prediction function is sufficiently high, the second prediction function indicates the true value of the second variable determined corresponding to the first variable. Therefore, when the prediction accuracy of the second prediction function is sufficiently high, the first index value is an index indicating the magnitude of data noise ("Aleatoric Uncertainty") contained in the second variable. In addition, the second index value is an index indicating the error between the "true function" representing the second variable determined corresponding to the first variable and the first prediction function. In other words, the second index value is an index indicating the uncertainty ("Epistemic Uncertainty") of the first prediction function.
[0052] In the present disclosure, since the purpose is to evaluate the uncertainty of a learning model generated by any machine learning, it is desirable for the second prediction function to have high prediction accuracy. Therefore, the second prediction function generated by the prediction function generating unit 251 may have a higher prediction accuracy than the first prediction function.
[0053] The index value setting unit 253 may perform machine learning based on data to generate the first index value. In other words, the index value setting unit 253 may generate the magnitude of data noise of the second variable determined corresponding to the first variable by machine learning.
[0054] More specifically, the index value setting unit 253 may perform machine learning using data of a pair of the first variable and the error between the second variable and the second prediction function as teacher data to generate the first index value. Here, the error between the second variable and the second prediction function may be evaluated as the difference between the second variable and the second prediction function, or may be evaluated as the square of the difference.
[0055] Alternatively, the index value setting unit 253 may generate an unbiased standard deviation between the second variable and the second prediction function in the vicinity of the value of the first variable as the first index value. When the "neighborhood" is set to include the entire range of the first variable included in the data, the calculated first index value approaches "Homoscedastic Uncertainty", which is a certain amount of noise. On the other hand, when the "neighborhood" is set to be sufficiently narrow, the calculated first index value approaches "Heteroscedastic Uncertainty", which is noise that changes depending on the first variable.
[0056] The index value setting unit 253 varies the second index value in a direction in which the absolute difference calculated by the absolute difference calculation unit 255 described later approaches a minimum (minimum). At that time, the first index value may be fixed. Also, the first index value may be varied when varying the second index value. As a method for minimizing the absolute difference (minimum), for example, the index value setting unit 253 may use the Nelder-Mead method.
[0057] Alternatively, the index value setting unit 253 may determine whether or not the absolute difference has converged near the minimum value due to fluctuations in the first index value and the second index value.
[0058] The absolute difference calculation unit 255 calculates the upper and lower limits of a prediction interval at a predetermined significance level for the second variable, which is determined based on the first index value and the second index value. The absolute difference calculation unit 255 calculates a first calculated value by subtracting the value at the lower limit from the value at the upper limit of a cumulative distribution function for the probability distribution of the second variable. The absolute difference calculation unit 255 also calculates a second calculated value by subtracting the predetermined significance level from 1. Then, the absolute difference calculation unit 255 calculates the absolute difference between the first calculated value and the second calculated value.
[0059] More specifically, the process of the absolute difference calculation unit 255 will be described using formulas. Here, the first variable is denoted as x, the first prediction function as F(x), the second prediction function as G(x), and the predetermined significance level as α. Also, the (1-α / 2) quantile in the probability distribution for the variable t obtained by standardizing the second variable so that the standard deviation is 1 and the mean value is 0 is denoted as k, and the cumulative distribution function for the probability distribution for the variable t is denoted as Φ(t). The first index value is denoted as σ a (x), the second index value is σ e (x), with the upper limit set to PI + (x), with the lower limit being PI - This is written as (x).
[0060] Under the above notation, the absolute difference calculation section 255 calculates the upper and lower limits of the prediction interval based on the following formula (1), and calculates the absolute difference based on formula (2).
[0061]
number
[0062] The absolute difference calculation unit 255 may calculate the upper and lower limits of the prediction interval based on an equation equivalent to equation (1) in addition to equation (1). The absolute difference calculation unit 255 may calculate the absolute difference between the first calculated value and the second calculated value based on an equation equivalent to equation (2) in addition to equation (2).
[0063] A normal distribution may be assumed as the probability distribution of the second variable. That is, data noise included in the second variable determined corresponding to the first variable may be assumed to be distributed according to a normal distribution. In addition, the probability distribution of the second variable may be various probability distributions such as exponential distribution, gamma distribution, beta distribution, uniform distribution, hypergeometric distribution, binomial distribution, Poisson distribution, and discrete uniform distribution. Uniform distribution / discrete uniform distribution can be used as long as it has a defined range of values. The probability distribution of the second variable may be a continuous type probability distribution or a discrete type probability distribution. The probability distribution of the second variable may be any one that can define a cumulative distribution function, and is not limited to the examples given here.
[0064] The absolute difference calculated by the absolute difference calculation unit 255 is transmitted to the index value setting unit 253 .
[0065] [Evaluation device processing procedure] Next, a processing procedure of the evaluation device will be described with reference to the flowchart of Fig. 2. Fig. 2 is a flowchart showing a processing procedure of the evaluation device according to an embodiment of the present disclosure.
[0066] For example, the process of the flowchart shown in FIG. 2 may be started by an instruction from a user.
[0067] In step S101, the input unit 21 acquires a plurality of data sets each including a first variable and a second variable, and a first prediction function.
[0068] In step S103, the prediction function generating unit 251 performs machine learning based on data to generate a second prediction function that approximates a second variable corresponding to the first variable and is different from the first prediction function.
[0069] In step S105, index value setting unit 253 sets a first index value and a second index value.
[0070] In step S107, absolute difference calculation section 255 calculates the absolute difference between the first calculated value and the second calculated value.
[0071] In step S109, in order to minimize (minimize) the absolute difference, index value setting unit 253 varies the first index value and the second index value.
[0072] In step S111, absolute difference calculation section 255 calculates the absolute difference between the first calculated value and the second calculated value (the absolute difference after variation) based on the first index value and the second index value after variation.
[0073] In step S113, the index value setting unit 253 calculates the amount of change in the absolute difference before and after the first index value and the second index value change, and determines whether the magnitude of the amount of change in the absolute difference is equal to or smaller than a predetermined threshold value, i.e., whether the absolute difference has converged near the minimum value.
[0074] If the magnitude of the amount of variation in the absolute difference is not equal to or less than the predetermined threshold (NO in step S113), the process returns to step S109.
[0075] On the other hand, if the magnitude of the fluctuation amount of the absolute difference is equal to or less than the predetermined threshold value (YES in step S113), in step S115, controller 25 outputs the first index value and the second index value at the time when the absolute difference converges near the minimum value.
[0076] [Effects of the embodiment] As described above in detail, the evaluation device, evaluation method, and evaluation program according to the present disclosure relate to an evaluation device including an input unit and a controller. The input unit acquires a plurality of data sets each including a first variable and a second variable, and a first prediction function generated based on the data and approximating the second variable corresponding to the first variable. The controller performs machine learning based on the data to generate a second prediction function different from the first prediction function, approximating the second variable corresponding to the first variable.
[0077] The first index value is an index indicating the error between the second variable and the second prediction function. The second index value is an index indicating the error between the second prediction function and the first prediction function. The controller calculates the second index value when the absolute difference between the first calculated value and the second calculated value is minimum when the second index value is varied. Here, the first calculated value is obtained by subtracting the value at the upper limit from the value at the lower limit of a cumulative distribution function for the probability distribution of the second variable for the upper and lower limits of a prediction interval at a predetermined significance level for the second variable, which is determined based on the first index value and the second index value. The second calculated value is obtained by subtracting the predetermined significance level from 1.
[0078] This makes it possible to evaluate the uncertainty (second index value) of a learning model (first prediction function) generated by any machine learning. In particular, it is possible to ensure that the obtained uncertainty correctly reflects the properties of the confidence interval and prediction interval.
[0079] Here, when the prediction accuracy of the second prediction function is sufficiently high, the second prediction function indicates the true value of the second variable determined corresponding to the first variable. Therefore, when the prediction accuracy of the second prediction function is sufficiently high, the second index value becomes an index indicating the error between the true function representing the second variable determined corresponding to the first variable and the first prediction function. Therefore, by using the second prediction function as a reference instead of the true function, the uncertainty of the first prediction function can be evaluated by the second index value.
[0080] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the first variable may be represented as x, the first prediction function as F(x), the second prediction function as G(x), and the predetermined significance level as α. The (1-α / 2) quantile in the probability distribution for variable t obtained by standardizing the second variable to have a standard deviation of 1 and a mean of 0 may be represented as k, and the cumulative distribution function for the probability distribution for variable t may be represented as Φ(t). The first index value may be represented as σ a (x), the second index value is σ e (x), with the upper limit set to PI + (x), with the lower limit being PI - It may also be written as (x).
[0081] The controller may then calculate the upper and lower limits based on equation (1), and calculate the absolute difference based on equation (2).
[0082] This ensures that the value of the second variable that appears in the data is included within the range of the prediction interval with a probability of (1-α), and therefore ensures that the uncertainty (second index value) of the calculated learning model correctly reflects the properties of the confidence interval and prediction interval.
[0083] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the first prediction function may be generated by machine learning different from deep learning based on data. This makes it possible to evaluate the uncertainty of the first prediction function even for the first prediction function expressed by a learning model with high explanatory power at the expense of prediction accuracy.
[0084] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the controller may perform deep learning based on data to generate a second prediction function, whereby a true function not including data noise can be accurately expressed by the second prediction function, and the uncertainty of the first prediction function can be evaluated.
[0085] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the controller may perform machine learning based on data to generate a first index value. This allows the error between the second variable and the second prediction function to be evaluated. In particular, even in an area where the distribution of the value of the first variable is sparse in a plurality of data, the deterioration of the error evaluation due to the small number of data is suppressed.
[0086] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the controller may generate an unbiased standard deviation between the second variable and the second prediction function in the vicinity of the value of the first variable as the first index value. This makes it possible to accurately evaluate the error when the noise changes gradually depending on the first variable.
[0087] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the controller may calculate the first index value and the second index value when the absolute difference is minimum when the first index value and the second index value are varied. This makes it possible to search for a case where the absolute difference is minimum by varying the first index value and the second index value simultaneously. As a result, it is possible to accurately evaluate the magnitude of data noise contained in the second variable in addition to the uncertainty of the first prediction function.
[0088] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the probability distribution may be a normal distribution. This makes it possible to evaluate the uncertainty of the first prediction function when the data noise included in the second variable is normally distributed. In particular, the nature of the distribution of the data noise can be used, and the uncertainty of the first prediction function can be evaluated with higher accuracy.
[0089] In the evaluation device, evaluation method, and evaluation program according to the present disclosure, the error between the second variable and the second prediction function may be smaller than the error between the second variable and the first prediction function, thereby ensuring that the second prediction function is a model that better describes the true function that does not include data noise compared to the first prediction function.
[0090] Each of the functions described in the above embodiments may be implemented by one or more processing circuits, including programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), or circuit components arranged to perform the described functions.
[0091] Although several embodiments have been described, the embodiments can be modified or modified based on the above disclosure. All components of the above embodiments and all features described in the claims may be individually extracted and combined as long as they are not mutually inconsistent. [Explanation of symbols]
[0092] 20 Evaluation Equipment 21 Input section 23 Database 25 Controller 27 Control section 29 Display section 251 Prediction Function Generator 253 Index value setting unit 255 Absolute Difference Calculation Unit
Claims
1. An evaluation device comprising an input unit and a controller, The input unit includes: A plurality of data sets each including a first variable and a second variable; a first prediction function generated based on the data and approximating the second variable corresponding to the first variable; Get The controller: performing machine learning based on the data to generate a second prediction function that is different from the first prediction function and that approximates the second variable corresponding to the first variable; a first index value being an index indicating an error between the second variable and the second prediction function; A second index value is an index indicating an error between the second prediction function and the first prediction function, When the second index value is varied, a first calculated value obtained by subtracting a value at the lower limit from a value at the upper limit of a cumulative distribution function for a probability distribution of the second variable, the first calculated value being determined based on the first index value and the second index value, for an upper limit and a lower limit of a prediction interval at a predetermined significance level for the second variable; A second calculated value obtained by subtracting the predetermined significance level from 1. and calculating the second index value when the absolute difference between the first and second index values is minimum.
2. The first variable is x, The first prediction function is F(x), The second prediction function is G(x), The predetermined significance level is α, k is the (1-α / 2) quantile in the probability distribution for the variable t obtained by standardizing the second variable to have a standard deviation of 1 and a mean of 0; The cumulative distribution function for the probability distribution of the variable t is Φ(t), The first index value is σ a (x), The second index value is σ e (x), The upper limit is PI + (x), the lower limit is PI - When written as (x), The controller: Calculating the upper limit and the lower limit based on formula (1); The evaluation device according to claim 1 , wherein the absolute difference is calculated based on the formula (2). [0010]
3. The evaluation device according to claim 1 , wherein the first prediction function is generated by machine learning other than deep learning based on the data.
4. The evaluation device according to claim 1 , wherein the controller performs deep learning based on the data to generate the second prediction function.
5. The evaluation device according to claim 1 , wherein the controller performs machine learning based on the data to generate the first index value.
6. The evaluation device according to claim 1 , wherein the controller generates, as the first index value, an unbiased standard deviation between the second variable and the second prediction function in a neighborhood of a value of the first variable.
7. The evaluation device according to claim 1 , wherein the controller calculates the first index value and the second index value when the absolute difference is minimized when the first index value and the second index value are varied.
8. The evaluation device according to claim 1 , wherein the probability distribution is a normal distribution.
9. The evaluation device according to claim 1 , wherein an error between the second variable and the second prediction function is smaller than an error between the second variable and the first prediction function.
10. A plurality of data sets each including a first variable and a second variable; a first prediction function generated based on the data and approximating the second variable corresponding to the first variable; An evaluation method for controlling a controller connected to an input unit that acquires The controller: performing machine learning based on the data to generate a second prediction function that is different from the first prediction function and that approximates the second variable corresponding to the first variable; a first index value being an index indicating an error between the second variable and the second prediction function; A second index value is an index indicating an error between the second prediction function and the first prediction function, When the second index value is varied, a first calculated value obtained by subtracting a value at the lower limit from a value at the upper limit of a cumulative distribution function for a probability distribution of the second variable, the first calculated value being determined based on the first index value and the second index value, for an upper limit and a lower limit of a prediction interval at a predetermined significance level for the second variable; A second calculated value obtained by subtracting the predetermined significance level from 1. and calculating the second index value when the absolute difference between the first index value and the second index value is minimum.
11. A plurality of data sets each including a first variable and a second variable; a first prediction function generated based on the data and approximating the second variable corresponding to the first variable; An evaluation program for processing On the computer, performing machine learning based on the data to generate a second prediction function that is different from the first prediction function and that approximates the second variable corresponding to the first variable; a first index value being an index indicating an error between the second variable and the second prediction function; A second index value is an index indicating an error between the second prediction function and the first prediction function, When the second index value is varied, a first calculated value obtained by subtracting a value at the lower limit from a value at the upper limit of a cumulative distribution function for a probability distribution of the second variable, the first calculated value being determined based on the first index value and the second index value, for an upper limit and a lower limit of a prediction interval at a predetermined significance level for the second variable; A second calculated value obtained by subtracting the predetermined significance level from 1. Calculating the second index value when the absolute difference between the An evaluation program that executes the following:
Citation Information
Patent Citations
Reliability assessment device
JP2022088174A
