Processing apparatus, processing method, and program

Abstract

To more appropriately evaluate a learning model.SOLUTION: A processing device 100 uses a learning model indicating a relation between an explanatory variable and an objective variable explained by the explanatory variable, and first learning data used for calculation of the learning model to generate second learning data for calculating a model evaluation regression equation, which is a linear regression equation indicating a relation between a correct answer value yt of the objective variable in the learning model and an estimated value ye of the objective variable in the learning model, and calculate the model evaluation regression equation using the second learning data. The processing device 100 calculates a variation evaluation index V, which is an index representing a variation of the second learning data from the model evaluation regression equation, and a gradient evaluation index grad, which is an index representing a gradient of the model evaluation regression equation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processing apparatus, a processing method, and a program. [Background technology]

[0002] In various fields, learning models that show the relationship between explanatory variables (independent variables) and the dependent variable that explains the explanatory variables are used to calculate the value of the dependent variable. Patent Document 1 discloses a method for identifying microbial groups involved in changes in the amount of a specific substance, in which the content of the microbial group is used as the explanatory variable and the rate of change of the amount of the specific substance is used as the dependent variable, and a penalty-based regression analysis is performed which can reduce the regression coefficient to 0 (zero), and microbial groups corresponding to the explanatory variable selected based on the regression coefficient calculated by the penalty-based regression analysis are selected.

[0003] Here, the coefficient of determination R is used as an evaluation metric to assess the accuracy of the learning model's estimation of the dependent variable (an evaluation metric to assess how accurately the dependent variable is explained (determined) by the explanatory variables). 2 There is. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-36579 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, the coefficient of determination R 2 However, there are cases where the regression equation cannot be properly evaluated. For example, if the regression equation is a nonlinear regression equation, the coefficient of determination R 2 There is a risk that it will be a negative value. In this case, the coefficient of determination R 2 The value of becomes meaningless, and the coefficient of determination R 2This makes it impossible to evaluate the regression equation. Therefore, there is a problem in that learning models, including the regression equation, cannot be properly evaluated.

[0006] This invention has been made in view of the above-mentioned problems, and aims to enable more appropriate evaluation of learning models. [Means for solving the problem]

[0007] The present invention relates to a processing apparatus that performs processing including evaluating a learning model that shows the relationship between one or more explanatory variables and a target variable explained by the explanatory variables, comprising: first learning data acquisition means for acquiring multiple sets of first learning data, each set consisting of a data set containing data of the explanatory variables and ground truth data of the target variable explained by the explanatory variables; second learning data creation means for creating multiple sets of second learning data as second learning data for calculating a model evaluation regression equation, which is a linear regression equation showing the relationship between a ground truth value, which is the value of the ground truth data of the target variable, and an estimated value, which is the value of the target variable calculated by the learning model, using the first learning data and the learning model; and model evaluation regression equation calculation means for calculating the model evaluation regression equation using the second learning data. The second learning data creation means includes an evaluation index calculation means for calculating an evaluation index for the learning model, wherein the second learning data creation means calculates an estimated value of the target variable in the learning model by providing the data values ​​of the explanatory variables included in the first learning data to the learning model, and creates a set of data as the second learning data that includes the estimated value of the target variable in the learning model and the ground truth value, which is the value of the ground truth data of the target variable that is paired with the data of the explanatory variables used to calculate the estimated value of the target variable in the first learning data, wherein the evaluation index includes a variability evaluation index and a gradient evaluation index, wherein the variability evaluation index is an index that represents the variability of the second learning data from the regression equation for model evaluation, and the gradient evaluation index is an index that represents the gradient of the regression equation for model evaluation.

[0008] The present invention relates to a processing method that includes evaluating a learning model that shows the relationship between one or more explanatory variables and a target variable explained by the explanatory variables, comprising: a first learning data acquisition step of acquiring multiple sets of first learning data, each set consisting of a data set containing data of the explanatory variables and ground truth data of the target variable explained by the explanatory variables; a second learning data creation step of creating multiple sets of second learning data as second learning data for calculating a model evaluation regression equation, which is a linear regression equation showing the relationship between the ground truth value, which is the value of the ground truth data of the target variable, and the estimated value, which is the value of the target variable calculated by the learning model, using the first learning data and the learning model; and a model evaluation regression equation calculation step of calculating the model evaluation regression equation using the second learning data. The second learning data creation step includes an evaluation index calculation step for calculating an evaluation index for the learning model, wherein the second learning data creation step calculates an estimated value of the target variable in the learning model by providing the data values ​​of the explanatory variables included in the first learning data to the learning model, and creates a set of data as the second learning data that includes the estimated value of the target variable in the learning model and the ground truth value, which is the value of the ground truth data of the target variable that is paired with the data of the explanatory variables used to calculate the estimated value of the target variable in the first learning data, and the evaluation index includes a variability evaluation index and a gradient evaluation index, wherein the variability evaluation index is an index that represents the variability of the second learning data from the regression equation for model evaluation, and the gradient evaluation index is an index that represents the gradient of the regression equation for model evaluation.

[0009] The program of the present invention causes a computer to function as one of the means of the processing apparatus. [Effects of the Invention]

[0010] According to the present invention, learning models can be evaluated more appropriately. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows a first example of the functional configuration of the processing unit. [Figure 2] This flowchart illustrates one example of a process for evaluating a learning model. [Figure 3] This is a conceptual diagram illustrating an example of the first training data. [Figure 4] This figure illustrates examples of variability evaluation indices and gradient evaluation indices. [Figure 5] This figure shows the first specific example of a variation evaluation index and a gradient evaluation index. [Figure 6] This figure shows a second specific example of the variability evaluation index and the gradient evaluation index. [Figure 7] This figure shows an example of the configuration of a biological processing device. [Figure 8A] This figure shows an example of the relationship between the thiocyanate removal rate per day of operation and the number of operating days. [Figure 8B] This figure shows an example of the relationship between the phenol removal rate per day of operation and the number of operating days. [Figure 8C] This figure shows an example of the relationship between the thiosulfate removal rate per day of operation and the number of operating days. [Figure 9] This figure shows a second example of the functional configuration of the processing unit. [Figure 10A] This flowchart illustrates the first example of a processing method for creating a learning model. [Figure 10B] This flowchart illustrates the first example of a method for calculating the estimated value of the dependent variable. [Figure 11] This figure shows an example of the regression equation for model evaluation and the second training data obtained for each selection process. [Figure 12] This figure shows an example of the regression equation for model evaluation and the second training data obtained in each selection process following Figure 11. [Figure 13] This figure shows the regression equation for model evaluation obtained in the first selection process and the first example of the second training data. [Figure 14] This figure shows the regression equation for model evaluation obtained from the second and subsequent selection processes, and a second example of the second training data. [Figure 15] This figure shows the first example of the relationship between the number of iterations of the selection process and the variability evaluation index for candidate learning models that have the explanatory variables selected in the selection process. [Figure 16] This figure shows an example of the group of explanatory variables (OTU) selected in each selection process. [Figure 17] This figure shows an example of the group of explanatory variables (OTU) selected in the first selection process. [Figure 18] This figure shows the regression equation for model evaluation obtained in the first selection process and the first example of the second training data. [Figure 19] This figure shows the regression equation for model evaluation obtained from the second and subsequent selection processes, and a second example of the second training data. [Figure 20] This figure shows a second example of the relationship between the number of iterations of the selection process and the variability evaluation index for candidate learning models that have the explanatory variables selected in that selection process. [Figure 21] This figure shows a second example of the functional configuration of the processing unit. [Figure 22A] This flowchart illustrates a second example of the process for creating a learning model. [Figure 22B] This flowchart illustrates a second example of a method for calculating the estimated value of the dependent variable. [Figure 23] This figure shows an example of an object that can be subjected to numerical analysis. [Figure 24] This is a diagram illustrating an example of a learning model. [Figure 25] This figure shows an example of the relationship between the average estimated diffusive substance concentration at all coordinates and the ground truth data value (ground truth value) for diffusive substance concentration. [Figure 26] This figure shows an example of the relationship between the average estimated diffusive substance concentration at a single coordinate and the ground truth value of the diffusive substance concentration. [Figure 27] This figure shows a first example of the diffusive substance concentration and gradient evaluation index at the bottom surface of the material. [Figure 28]This figure shows a second example of the diffusive substance concentration and gradient evaluation index at the bottom surface of the material. [Figure 29] This figure shows a comparative example of the diffusive substance concentration and gradient evaluation index at the bottom surface of the material. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the drawings. Note that when comparing objects with the same length, position, size, spacing, etc., this includes not only cases where they are strictly identical, but also cases where they differ within a range that does not depart from the spirit of the invention (for example, differences within the tolerance range determined during design).

[0013] (First Embodiment) First, the first embodiment will be described. Figure 1 shows an example of the functional configuration of the processing unit 100 of this embodiment. The processing unit 100 has, for example, one or more hardware processors such as a CPU (Central Processing Unit) and one or more memory such as RAM (Random Access Memory) and ROM (Read Only Memory) as hardware, and performs various calculations by executing one or more programs stored in memory using one or more hardware processors. Furthermore, the processing unit 100 has input devices and output devices as hardware.

[0014] The processing unit 100 of this embodiment performs a process that includes evaluating a learning model that shows the relationship between one or more explanatory variables and the target variable explained by those explanatory variables. The learning model may be a linear regression equation, a nonlinear regression equation, another machine learning model (such as a neural network), or AI (artificial intelligence). The learning model may also be a surrogate model or a model that estimates phenomena other than physical phenomena. The number of explanatory variables may be one or multiple.

[0015] The explanatory and dependent variables are not limited. For example, the explanatory variables may include factors used in the treatment process, and the dependent variable may include factors obtained in the treatment process by performing the treatment using the factors used in the treatment process. The treatment process is, for example, a biological treatment process of water to be treated. In this case, the explanatory variables are, for example, factors relating to the amount of microbial communities present in and used in the treatment process. The dependent variable is, for example, a factor relating to the rate of change in the amount of a specific substance before and after treatment (change in the amount of a specific substance). The water to be treated is treated using microorganisms present in the water, and the amount of a specific substance in the water changes (decreases or increases). That is, the rate of change in the amount of a specific substance before and after treatment is influenced by the microbial communities present in the treatment process. Here, the microbial community refers to a group of microorganisms classified based on a certain trait or genotype, and includes, but is not limited to, genera, species, and subspecies. The genotype may also include simply the base sequence of the microbial genome. The number of microbial communities can be, for example, between 100 and 1 million species, or between 1,000 and 10,000 species.

[0016] Furthermore, the treatment process is not limited to a biological treatment process for the water to be treated. For example, the treatment process may be a product manufacturing process. For example, the treatment process may be a hot-rolled coil manufacturing process. In this case, the explanatory variables are, for example, physical quantities used as manipulated quantities for the rolled material (e.g., reduction amount of the finishing rolling mill, flow rate of cooling water in the cooling spray, power consumption of the heating device, conveying speed of the rolled material). The dependent variable is, for example, the quality of the hot-rolled coil (e.g., tensile strength). That is, the quality of the hot-rolled coil is affected by the physical quantities used as manipulated quantities for the rolled material. The explanatory variables may be state factors that determine the characteristics of the objective variable during processing (uncontrollable factors whose attributes (quantity, etc.) cannot be controlled during the processing process), or they may be controllable factors whose attributes can be controlled during the processing process.

[0017] Furthermore, explanatory variables are not limited to factors used in the processing steps. For example, explanatory variables may be factors relating to the number of various genes in a person. In this case, the dependent variable may be, for example, a factor relating to a test item for a specific human disease. That is, the values ​​of test items for a specific disease are influenced by a person's genes. Furthermore, the explanatory variables and the dependent variable are not limited to those mentioned above, as long as the dependent variable is explained by the explanatory variables (i.e., there is a causal relationship between the explanatory variables and the dependent variable).

[0018] In Figure 1, the processing unit 100 includes a first training data acquisition unit 110, a second training data creation unit 120, a regression equation calculation unit for model evaluation 130, an evaluation index calculation unit 140, and an output unit 150. Figure 2 is a flowchart illustrating an example of a processing method for evaluating a training model. The processing method of this embodiment will be described below with reference to the flowchart in Figure 2, along with an example of the functions of the processing unit 100 shown in Figure 1.

[0019] [First learning data acquisition process (S201), first learning data acquisition unit 110] The first learning data acquisition unit 110 acquires multiple sets of first learning data, each set consisting of a data set containing data for explanatory variables in the learning model and the correct data for the target variable explained by those explanatory variables. The first learning data is the learning data used when calculating (learning) the learning model. As mentioned above, the learning model may be a linear regression equation, a nonlinear regression equation, another machine learning model (such as a neural network), or AI (artificial intelligence). However, in this embodiment, for the sake of simplicity, we will illustrate the case where the learning model is a linear regression equation as shown in equation (1) below.

[0020]

number

[0021] Here, y eis the target variable (estimated value). Note that y e The e in it represents that it is an estimated value. Also, the estimated value is the value of the target variable calculated by the learning model (regression equation). x j is an explanatory variable. a j is the explanatory variable x j is the regression coefficient with respect to x. p is the number of explanatory variables. When p = 1, Equation (1) is a simple regression equation. When p ≥ 2, Equation (1) is a multiple regression equation. b is the intercept.

[0022] Figure 3 conceptually shows an example of the first learning data. In Figure 3, data of p explanatory variables x1 to x p and the target variable y p explained by the p explanatory variables x1 to x t are exemplified as Q sets of the first learning data which consists of a set of data including the correct answer data. Note that the t in y t represents that it is the correct value (the value of the correct answer data (correct label)). As a form of obtaining the first learning data, for example, at least one of transmission from an external device, an input operation of the first learning data to the user interface by an operator, and reading from a portable storage medium is adopted.

[0023] Also, in this embodiment, a case where what explanatory variables are to be used as the explanatory variables x1 to x p of the learning model is predetermined is exemplified. Also, in this embodiment, a case where the regression coefficient a j is calculated in advance and stored in the processing device 100 using the first learning data acquired by the first learning data acquisition unit 110 is exemplified. Note that the calculation method of the regression coefficient a j using the learning data is realized by using a known regression analysis method such as the least squares method, so detailed description is omitted here. Also, the processing device 100 may calculate the regression coefficient a j using the first learning data acquired by the first learning data acquisition unit 110.

[0024] [Second training data creation process (S202), second training data creation unit 120] The second training data creation unit 120 uses the first training data and the training model to create multiple sets of second training data for calculating a regression equation for model evaluation. The regression equation for model evaluation is, for example, the ground truth value y, which is the ground truth value of the target variable in the training model. t And the estimated value y, which is the value of the target variable calculated by the learning model. e This is a linear regression equation that shows the relationship between and . The regression equation for model evaluation can be expressed, for example, by equation (2) below. In the following explanation, the ground truth value y is the value of the ground truth data for the target variable in the learning model. t If necessary, the correct value y of the target variable in the learning model. t , or the correct value y of the dependent variable t This is referred to as [the model name]. Furthermore, in the following explanation, the estimated value y is the value of the target variable calculated by the learning model. e If necessary, the estimated value of the target variable y in the learning model e It is called that.

[0025]

number

[0026] Here, c is the regression coefficient, and d is the intercept. The intercept d does not have to be 0 (zero), but in this embodiment, when the intercept d is 0 (zero), (i.e., the regression equation for model evaluation is y) e =cy t This example illustrates the case where...

[0027] The second training data creation unit 120 generates explanatory variables x1 to x included in the first training data. p By providing the data values ​​to the learning model, the estimated value of the target variable y can be obtained. e The following is calculated. In the example shown in Figure 3, the estimated value of the dependent variable y e The estimated values ​​of Q dependent variables y e The following is calculated. Then, the second learning data creation unit 120 calculates the estimated value y of the target variable. eAnd, in the first training data, the estimated value of the target variable y e Explanatory variables x1~x used in the calculation p The correct value of the target variable y, paired with the data. t A second set of training data, set Q, is created, consisting of a set of data including and .

[0028] [Model evaluation regression equation calculation process (S203), Model evaluation regression equation calculation unit 130] The regression equation calculation unit 130 for model evaluation calculates the second training data (estimated value of the target variable y) e and the correct value y t The regression equation for model evaluation (regression coefficient c) is calculated using the set of (2). Note that the method for calculating the regression coefficient c using the training data is achieved by using known regression analysis methods such as the least squares method, so a detailed explanation is omitted here. Here, the case where the intercept d of equation (2) is 0 (zero) is given as an example, so the regression equation calculation unit 130 for model evaluation performs regression analysis with the intercept d fixed at 0 (zero). If the intercept d of equation (2) is not 0 (zero), the regression equation calculation unit 130 for model evaluation also calculates the value of the intercept d.

[0029] [Evaluation index calculation process (S204), evaluation index calculation unit 140] The evaluation index calculation unit 140 uses the second training data and the regression equation for model evaluation to calculate a variability evaluation index V and a gradient evaluation index grad as evaluation indices for evaluating the learning model.

[0030] The variability evaluation index V is an index that represents the variability of the second training data from the regression equation used for model evaluation. The gradient evaluation index grad is an index that represents the gradient (slope) of the regression equation used for model evaluation. Figure 4 illustrates an example of the variability evaluation index V and the gradient evaluation index grad.

[0031] Figure 4(a) illustrates the case where the second training data is represented by points (black circles (●)) on a two-dimensional Cartesian coordinate system. The horizontal axis (first axis) of the two-dimensional Cartesian coordinate system represents the correct value y of the target variable in the learning model. tThis axis represents the y-axis. The vertical axis (second axis) of the 2D Cartesian coordinate system represents the estimated value of the target variable in the learning model. e This is the axis that indicates [something].

[0032] The black circles (●) in Figure 4(a) represent the correct values ​​of the target variable y in the learning model. t And, in the first training data, the correct value y t The explanatory variables x1~x are paired with this. p By substituting the data values ​​into the learning model, the estimated value of the target variable y is calculated by the learning model. e This is the point that is determined from that.

[0033] Here, the estimated value y of the target variable in the learning model is included in the i-th set of the second training data from set Q. e to y e1_i This will be the notation. Also, the correct value y of the target variable in the learning model, which is included in the second training data of the i-th set. t to y t_i This is expressed as y-, where y- is the mean of the Q correct values ​​of the target variable in the learning model. t This will be the notation. Also, the correct value y of the target variable in the learning model, which is included in the second training data of the i-th set. t_i The estimated value of the dependent variable y is calculated by substituting this into the regression equation for model evaluation. e to, y e2_i We will use this notation. Then, the variability evaluation index V can be expressed, for example, by equation (3) below. Equation (3) illustrates a case where a larger value of the variability evaluation index V (closer to 1) indicates a smaller degree of variability in the second training data from the regression equation used for model evaluation. y- t In equation (3), y t This corresponds to the symbol with a hyphen above the 't' (this notation is the same for other symbols as well).

[0034]

number

[0035] Here, the content inside the parentheses in the denominator of Equation (3) is an example of the first variation, and the content inside the parentheses in the numerator is an example of the second variation. The first variation is represented, for example, by the difference between the estimated value of the target variable included in the second training data and the average value of the correct answer value of the target variable. The second variation is represented, for example, by the difference between the estimated value of the target variable included in the second training data and the estimated value of the target variable calculated by substituting the correct answer value of the target variable, which is paired with the estimated value of the target variable in the second training data, into the regression equation for model evaluation, and one of the differences between the estimated value of the target variable calculated by substituting the correct answer value of the target variable into the regression equation for model evaluation and the average value of the correct answer value of the target variable. More specifically, in Equation (3), a case is exemplified in which the variation evaluation index V is represented by the ratio of the value obtained by integrating the square of the second variation over all sets (=Q) of the second training data to the value obtained by integrating the square of the first variation over all sets (=Q) of the second training data. However, it is not always necessary to represent the variation evaluation index V in this way. For example, instead of the square of the first variation and the square of the second variation, the square root of the square of the first variation and the square root of the square of the second variation may be used, respectively. The slope evaluation index grad is represented by the gradient of the regression equation 411 for model evaluation (the regression coefficient c in Equation (2)).

[0036] FIG. 4(b) is a diagram for explaining the coefficient of determination R 2 FIG. 4(b) exemplifies a case where the learning model 421 is a linear simple regression equation represented by the following Equation (4) for the sake of simplicity of explanation.

[0037]

Equation

[0038] In FIG. 4(a), an example is shown in which the second learning data is represented by points (black circles (●)) on a two-dimensional orthogonal coordinate system. On the other hand, in FIG. 4(b), an example is shown in which the first learning data is represented by points (black circles (●)) on a two-dimensional orthogonal coordinate system. The horizontal axis (first axis) of the two-dimensional orthogonal coordinate system is an axis showing the data value of the explanatory variable x1 in the learning model. The vertical axis (second axis) of the two-dimensional orthogonal coordinate system is an axis showing the estimated value y of the objective variable in the learning model. e It is an axis showing.

[0039] The black circles (●) shown in FIG. 4(b) are the data values of the explanatory variable x1 included in the first learning data and the correct value y of the objective variable in the learning model that is paired with the data of the explanatory variable x1. t It is a point determined from.

[0040] Here, let the data value of the explanatory variable x1 in the learning model included in the i-th set of the Q sets of the first learning data be denoted as x. 1_t_i Also, let the correct value y of the objective variable in the learning model included in the i-th set of the first learning data be denoted as y. t be denoted as y. t_i Also, let the correct value of the objective variable in the learning model included in the i-th set of the first learning data be denoted as y. t_i be denoted as y. 2 Then, the coefficient of determination R is represented by, for example, the following equation (5).

[0041]

Equation

[0042] The denominator of equation (5) represents the total variation. Also, the middle numerator of equation (5) represents the residual variation. The last numerator of equation (5) represents the regression variation. As described above, the coefficient of determination R is an index representing the variation of the first learning data with respect to the learning model 421, while the variation evaluation index V is an index representing the variation of the second learning data with respect to the regression equation 411 for model evaluation. 2

[0043] ​ Figure 5 shows the first specific example of the variability evaluation index V and the gradient evaluation index grad. Figure 5(a) shows an example of the regression equation 511 for model evaluation when the gradient evaluation index grad (regression coefficient c) is a positive value. Figure 5(b) shows an example of the regression equation 521 for model evaluation when the gradient evaluation index grad (regression coefficient c) is a negative value. The learning model used to calculate the regression equations 511 and 521 for model evaluation shown in Figure 5 is a linear regression equation.

[0044] In the example shown in Figure 5(a), the variability evaluation index V was 0.3293 and the gradient evaluation index grad was 0.337. On the other hand, the coefficient of determination R calculated from the first training data and the trained model used to calculate the regression equation 511 for model evaluation was 0.337. 2 The value was 0.329. In the example shown in Figure 5(b), the variability evaluation index V is 0.5249 and the gradient evaluation index grad is -0.022, quantitatively capturing how the data is scattered around a negative line. On the other hand, the coefficient of determination R calculated from the first training data and the trained model used to calculate the regression equation 521 for model evaluation is 2 The coefficient of determination R is -0.045. 2 A negative value indicates that statistical regression is not possible, and therefore the coefficient of determination R 2 This shows that it cannot be used as an indicator for evaluating variability.

[0045] The inventors have found that when the learning model is a linear regression equation, if the gradient evaluation index grad is non-negative (or positive), the variability evaluation index V is equal to the coefficient of determination R 2 Since this is in close agreement, we found that we can evaluate the learning model by using the variability evaluation index V. That is, under the condition that the gradient evaluation index grad is non-negative (or positive), the larger the value of the variability evaluation index V, the better the estimated value of the target variable y in the learning model. e The reliability is high (the dependent variable is well explained by the explanatory variables).

[0046] On the other hand, if the gradient evaluation index grad is negative (or less than or equal to 0 (zero)), the coefficient of determination R 2Since it tends to be a negative value, if the gradient evaluation index grad is negative (or less than or equal to 0 (zero)), the coefficient of determination R 2 While is not a variability evaluation index, we found that by using the variability evaluation index V, it is possible to quantitatively evaluate the variability of the learning model.

[0047] Figure 6 shows a second specific example of the variability evaluation index V and the gradient evaluation index grad. The regression equation used to calculate the regression equation 611 for model evaluation shown in Figure 6 is a nonlinear regression equation. In the example shown in Figure 6, the variability evaluation index V is 0.999, indicating that there is almost no variability in the data, and the gradient evaluation index grad at this time was 0.300. On the other hand, the coefficient of determination R calculated from the first training data and the trained model used to calculate the regression equation 611 for model evaluation is... 2 The result is -1218.252, which means it cannot be used as an index for evaluating variability.

[0048] Thus, the inventors have determined the coefficient of determination R 2 While it is not possible to evaluate the nonlinear regression equation using the method described above, we found that by using the variability evaluation index V and the gradient evaluation index grad, if the variability of the second training data is small relative to the regression equation used for model evaluation (nonlinear regression equation), the variability evaluation index V approaches 1. Therefore, the nonlinear regression equation can be evaluated using the variability evaluation index V.

[0049] [Output process (S205), output unit 150] The output unit 150 outputs information on evaluation indices (variability evaluation index V and gradient evaluation index grad) calculated by the evaluation index calculation unit 140. The output format of the evaluation index information is at least one of the following: display on a computer display, transmission to an external device, and storage on a portable storage medium.

[0050] [summary] As described above, in this embodiment, the processing unit 100 uses the explanatory variables, a learning model showing the relationship between the explanatory variables and the target variable explained by the explanatory variables, and the first learning data used to calculate the learning model to obtain the correct value y of the target variable in the learning model. t And the estimated value of the target variable y in the learning model e The system creates a second set of training data to calculate a model evaluation regression equation, which is a linear regression equation showing the relationship between and , and calculates the model evaluation regression equation using the second set of training data. Then, the processing unit 100 calculates a variability evaluation index V, which is an index representing the variability from the model evaluation regression equation of the second set of training data, and a gradient evaluation index grad, which is an index representing the gradient of the model evaluation regression equation. Therefore, the trained model can be evaluated more appropriately. For example, the coefficient of determination R 2 Even learning models that cannot be evaluated due to a negative value can be evaluated.

[0051] [First training data] As mentioned above, the dependent and independent variables of the learning model are not limited, but we will explain an example of how to acquire the first training data by taking the case where the learning model is used to estimate the microbial species that affect the rate of change in the amount of a specific substance before and after treatment in a biological wastewater treatment process. In this case, the independent variable of the learning model is the microbial species, and the dependent variable of the learning model is the rate of change in the amount of a specific substance before and after treatment in a biological wastewater treatment process.

[0052] First, a Moving Bed Biofilm Reactor (MBBR) is used to collect sample data of the rate of change in the amount of specific substances and the content of microbial communities in each sample of artificial wastewater (treated water). In the following explanation, the MBBR that treats the artificial wastewater will be referred to as MBBR Run3 as needed. Below is an example of how to actually acquire the sample data (first training data).

[0053] <Collection of data on the rate of change in the amount of a specific substance> In a solvent obtained by mixing industrial water and natural seawater at a volume ratio of 2:3, the solutes shown in Table 1 were dissolved at the concentrations shown in Table 1 to prepare artificial wastewater (the water to be treated).

[0054]

Table 1

[0055] The rate of change in the amount of a specific substance was measured, for example, as shown in FIG. 7, using an integrated biological treatment apparatus 20 having a structure in which a biological treatment region 20a and a sedimentation region 20b are separated from each other by a partition wall 23 in one tank and communicate with each other below the partition wall 23.

[0056] First, a sponge carrier 21 [fluid carrier (AQ-1 manufactured by Kanto Denka Kogyo Co., Ltd.)] having a size of 10 mm × 10 mm × 10 mm and high-concentration activated sludge as a microbial seeding source were put into a plastic bottle, and the plastic bottle was kneaded well by hand, covered, and immersed overnight to attach microorganisms to the sponge carrier 21. The activated sludge to be put in here is preferably a seeding source rich in thiocyanate-decomposing microorganisms, phenol-decomposing microorganisms, and thiosulfate-decomposing microorganisms.

[0057] The sponge carrier 21 (500 pieces) and the activated sludge prepared in this way were put into the biological treatment region 20a of the biological treatment apparatus 20 to prepare the biological treatment apparatus 20.

[0058] The water to be treated defined above was made to flow into the biological treatment apparatus 20 prepared in this way, and activated sludge was put in as a microbial seeding source. At the time of the first-stage treatment, the water to be treated 24 was made to flow into the biological treatment apparatus 20 so that the hydraulic retention time in the biological treatment region 20a would be 24 hours. Further, air aeration 22 was performed on the water to be treated 24 in each biological treatment apparatus 20 to form an aerobic fluidized bed and acclimatize the microorganisms. Further, the treatment was carried out while adjusting the pH to around 7.5 using a 5 wt% aqueous sodium hydroxide solution. Then, the treated water 25 treated by the biological treatment apparatus 20 was discharged from the biological treatment apparatus 20.

[0059] The thiocyanate ion concentration, phenol concentration, and thiosulfate ion concentration were measured in the treated water within the biological treatment area 20a of each biological treatment device 20 to monitor thiocyanate ions, phenol, and thiosulfate ions. Monitoring was performed approximately twice a week.

[0060] After the first stage of treatment stabilized, the treated water 24 was introduced into the biological treatment device 20 so that the daily inflow concentrations of thiocyanate ions, phenol, and thiosulfate ions were as shown in Table 2. Each stage was operated for at least two weeks to confirm that the fluctuations in the treated water concentrations at each stage had stabilized before moving on to the next stage. Monitoring was performed approximately twice a week.

[0061] [Table 2]

[0062] Figures 8A, 8B, and 8C show the daily thiocyanate removal rate, phenol removal rate, and thiosulfate removal rate per day of operation in the MBBR Run3 biological treatment, respectively. Figure 8A shows the relationship between the daily thiocyanate ion removal rate and the number of operation days, obtained from the monitoring results of thiocyanate ions. Figure 8B shows the relationship between the daily phenol removal rate and the number of operation days, obtained from the monitoring results of phenol. Figure 8C shows the relationship between the daily thiosulfate removal rate and the number of operation days, obtained from the monitoring results of thiosulfate.

[0063] <Collection of microbial community content> DNA extraction, sequencing, determination of microbial communities, and collection of microbial community content were performed in this order. Note that the method for collecting microbial community content can be achieved using known techniques, as described in Patent Document 1, and is not limited to the method described in this section.

[0064] DNA extraction and next-generation sequencing microbial community analysis from the sponge carrier 21 to which microorganisms were attached within the biological treatment area 20a of the biological treatment apparatus 20 were performed by outsourcing (Nippon Steel Environmental Co., Ltd.). At each time point corresponding to the measurement of the rate of change in the amount of a specific substance, a sponge carrier 21 with microorganisms attached was collected. After dividing the collected sponge carrier 21 into four parts, DNA extraction and purification were performed using Extra Soil DNA Plus ver.2 (Nippon Steel Environmental). The DNA concentration in the purified DNA solution was measured using the PicoGreen dsDNA Assay Kit (Invitrogen). PCR amplification targeting the V4 and V5 regions of the 16S rRNA gene of eubacteria was performed using the primers shown in Table 3.

[0065] [Table 3]

[0066] The PCR products were analyzed using a next-generation sequencer (MiSeq) to determine their base sequences. The obtained nucleotide sequences were analyzed using the QIIME (Quantitative Insights Into Microbial Ecology) pipeline. First, the data quality and chimericity were checked, and only sequence data that met the criteria were filtered. For sequence data that met the criteria, highly similar sequence data (homology of 97% or more) were grouped into clusters. The most frequently occurring sequence within each cluster was designated as the representative OTU (Operational Taxonomic Unit) sequence, and subsequent analyses were performed using this representative sequence. In other words, the presence and quantity of each detected OTU were treated as indicating the presence and quantity of a single microbial community. These OTUs varied in that some were detected multiple times in each sample, while others were detected in only one sample. Furthermore, the relative proportion of each OTU to the total OTU was calculated from the number of times each OTU was detected.

[0067] The number of genes in eubacteria attached to the sponge carrier 21 was quantified using the QP-PCR method (Nippon Steel Environmental Co., Ltd.), a real-time PCR method. Specifically, after appropriately diluting the purified DNA solution described above, a reaction solution was prepared using the primers and QProbe shown in Table 4, and the number of genes was quantified using Rotor-Gene Q (QIAGEN).

[0068] [Table 4]

[0069] The total amount of microorganisms attached to the sponge carrier 21 differs depending on the sampling date. Therefore, in order to accurately understand the fluctuations of each OTU during the operation period of the biological wastewater treatment process, the amount of each OTU was determined by multiplying the relative ratio of each OTU to the total OTU mentioned above by the number of genes of eubacteria attached to the sponge carrier 21. The number of OTUs was 3752.

[0070] In the specific examples described above, the target variable of the learning model is, for example, at least one of the thiocyanate removal rate, phenol removal rate, and thiosulfate removal rate. The explanatory variables of the learning model are, for example, the amounts of each OTU. In the following explanation, OTU will be referred to as denovo as needed. Note that if there are multiple target variables in the learning model, the target variables may be represented as vectors, the regression coefficients as matrices, and the intercept as vectors.

[0071] (Second Embodiment) Next, a second embodiment will be described. In the first embodiment, the explanatory variables x1~x of the learning model p The example given was that the variable x1 is fixed. In contrast, in this embodiment, the explanatory variables x1 to x included in the first training data are pThis example illustrates the case where the explanatory variables used as explanatory variables for the learning model are determined using the variability evaluation index V and the gradient evaluation index grad. Thus, this embodiment differs from the first embodiment mainly in its configuration and processing, which are determined by using the variability evaluation index V and the gradient evaluation index grad to determine the explanatory variables used as explanatory variables for the learning model. Therefore, in the description of this embodiment, detailed explanations of parts that are the same as those in the first embodiment will be omitted by using the same reference numerals as those used in Figures 1 to 8D. Also, in this embodiment, for the sake of simplicity, the example will illustrate the case where the learning model is a linear regression equation.

[0072] Figure 9 shows an example of the functional configuration of the processing unit 900 in this embodiment. The hardware of the processing unit 900 is implemented using, for example, the same hardware as the processing unit 100 in the first embodiment.

[0073] In Figure 9, the processing unit 900 includes a first learning data acquisition unit 110, a determination unit 910, a storage unit 920, an estimation data acquisition unit 930, a target variable calculation unit 940, and an output unit 950.

[0074] The decision unit 910 determines the explanatory variables of the learning model. The decision unit 910 determines the explanatory variables x1 to x included in the first learning data. p The first training data is used to calculate multiple candidates for a learning model that shows the relationship between some of the explanatory variables and the target variable explained by those explanatory variables. The decision unit 910 then uses the multiple candidates for the learning model (as if they were learning models) to calculate evaluation indices (variability evaluation index V and gradient evaluation index grad) as described in the first embodiment. The decision unit 910 then determines the explanatory variables in the learning model based on the evaluation indices for each of the multiple candidates for the learning model. Specifically in this embodiment, an example is given in which the decision unit 910 further comprises an explanatory variable selection unit 911, a learning model candidate calculation unit 912, a second training data creation unit 913, a regression equation calculation unit for model evaluation 914, an evaluation index calculation unit 915, and an explanatory variable determination unit 916.

[0075] The memory unit 920 stores the learning model having the explanatory variables determined by the decision unit 910. The estimation data acquisition unit 930 acquires estimation data, including the data of the explanatory variables in the learning model. The estimation data is used by the learning model to calculate an estimated value of the target variable. The target variable calculation unit 940 calculates an estimated value of the target variable by providing the explanatory variable data acquired by the estimation data acquisition unit 930 to the learning model. The output unit 950 outputs information on the estimated value of the target variable corrected by the target variable correction unit 2110.

[0076] In this embodiment, we illustrate a case where the first learning data acquisition unit 110, the determination unit 910, the storage unit 920, the estimation data acquisition unit 930, the target variable calculation unit 940, and the output unit 950 are all provided in a single processing unit 900. However, this is not necessarily required. For example, the processing unit comprising the first learning data acquisition unit 110, the determination unit 910, and the storage unit 920, and the processing unit comprising the estimation data acquisition unit 930, the target variable calculation unit 940, and the output unit 950 may be separate processing units. Furthermore, a separate processing unit may also be equipped with the storage unit 920. In addition, multiple processing units may communicate with each other by wired communication or wireless communication.

[0077] Figure 10A is a flowchart illustrating an example of the processing method for creating the learning model of this embodiment. Figure 10B is a flowchart illustrating an example of the processing method for calculating the estimated value of the target variable using the learning model of this embodiment.

[0078] [Creating a learning model] First, referring to the flowchart in Figure 10A, an example of the processing method for creating the learning model in this embodiment will be explained along with an example of the functions of the processing unit 900 shown in Figure 9.

[0079] [First learning data acquisition process (S1001), first learning data acquisition unit 110] The first learning data acquisition unit 110 acquires multiple sets (Q sets) of first learning data, similar to the first embodiment. In the specific example described in the [First Learning Data] section, the explanatory variables x1~x shown in Figure 3 are used. p This represents the quantities of 3752 types of OTU (p=3752). Also, the dependent variable y shown in Figure 3. t For example, this is the phenol removal rate.

[0080] [Explanatory variable selection process (S1002), learning model candidate calculation process (S1003), second training data creation process (S1004), regression equation calculation process for model evaluation (S1005), evaluation index calculation process (S1006), explanatory variable selection unit 911, learning model candidate calculation unit 912, second training data creation unit 913, regression equation calculation unit for model evaluation 914, evaluation index calculation unit 915]

[0081] The explanatory variable selection unit 911 selects the explanatory variables x1 to x included in the first training data. p From this selection, we choose explanatory variables for the learning model. In this embodiment, the explanatory variables x1 to x included in the first training data are selected. p As an example of a method for narrowing down the explanatory variables to be used in the learning model, we will use stepwise forward selection (variable augmentation), a type of sequential feature selection method. However, the explanatory variables x1~x included in the first training data... pThe method for narrowing down the explanatory variables to be used in the learning model is not limited to stepwise forward selection (variable augmentation). For example, variable decrement, variable increment / decrement, or variable decrement / increment may be used. However, in sequential feature selection, factors (explanatory variables in this embodiment) that have been selected once are not re-evaluated. Therefore, the factors selected initially may greatly influence the factors selected in the end. In this case, the factors selected in the end may be a local optimum. Therefore, in this embodiment, we illustrate a case in which the factors (explanatory variables in this embodiment) are selected using evaluation indices (variability evaluation index V and gradient evaluation index grad) as described in the first embodiment, thereby suppressing the convergence of the final selected factors to an undesirable local optimum. That is, in this embodiment, we illustrate a case in which the selection conditions for factors (explanatory variables in this embodiment) in algorithms such as variable augmentation are conditions using evaluation indices (variability evaluation index V and gradient evaluation index grad). Below, we describe a specific example of the selection process for sequentially selecting explanatory variables in a learning model.

[0082] <First selection process> First, the explanatory variable selection unit 911 selects the explanatory variables x1 to x included in the first training data. p Select one of these as a candidate explanatory variable in the learning model and output it to the learning model candidate calculation unit 912, based on the explanatory variables x1 to x included in the first learning data. p This is done for each of the above. Thus, in this embodiment, we illustrate the case where the number of explanatory variables selected in the first selection process (initial value of the number of explanatory variables) is 1.

[0083] Then, the learning model candidate calculation unit 912 calculates a learning model candidate using the first training data, assuming that the learning model candidate is a simple linear regression equation, by considering the explanatory variables x1~x included in the first training data. p This is done individually for each of them. This results in p candidate learning models. In this case, the explanatory variables x1 to x included in the first training data are... p Each of these is a candidate explanatory variable (an explanatory variable in a candidate learning model).

[0084] Note that the first learning data includes data of p explanatory variables x1 to x p and the correct value y of the objective variable explained by the p explanatory variables x1 to x p . When calculating a candidate for a learning model having a candidate x t (in the first selection process, r is an integer from 1 to p) as an explanatory variable, the learning model candidate calculation unit 912 uses the data of the candidate x r of the explanatory variable and the correct value y of the objective variable paired with the data in the first learning data r as a set of learning data to calculate a candidate for the learning model. t

[0085] Note that the candidate for the learning model in the first selection process is represented by, for example, the following equation (6). As shown in the following equation (6), in this embodiment, the case where the candidate for the learning model in the first selection process is a simple regression equation is exemplified.

[0086]

Equation

[0087] <0OO1088>The second learning data creation unit 913 performs a process of replacing "learning model" with "candidate for learning model" and replacing "first learning data" with "learning data used in the calculation of the candidate for learning model" in the section of [second learning data creation step (S202), second learning data creation unit 120], thereby creating second learning data for each of the p candidates for the learning model calculated by the learning model candidate calculation unit 912.

[0088] Specifically, the second learning data creation unit 913 substitutes the value of the data of the explanatory variable x r included in the first learning data used in the calculation of the candidate for the learning model into the candidate for the learning model (equation (6)), thereby obtaining the estimated value y eCalculate it. Then, the second learning data creation unit 913 uses the estimated value y of the target variable in the candidate of the learning model e and, in the first learning data used when calculating the candidate of the learning model, the estimated value y of the target variable e of the explanatory variable x used for the calculation r and the correct value y of the target variable that is paired with the data t to create Q sets of second learning data as learning data consisting of a set of data including them. The second learning data creation unit 913 creates such Q sets of second learning data for each of the p candidates of the learning model calculated by the learning model candidate calculation unit 912 (for each r = 1 to p). As a result, p second learning data groups are created as a second learning data group consisting of Q sets of second learning data.

[0089] Similar to the regression equation calculation unit 130 for model evaluation, the regression equation calculation unit 914 for model evaluation calculates a regression equation for model evaluation (regression coefficient c) using the second learning data (second learning data group). However, the regression equation calculation unit 914 for model evaluation calculates the regression equation for model evaluation by individually using each of the p second learning data groups. As a result, one regression equation for model evaluation for each of the p candidates of the learning model is calculated (p regression equations for model evaluation are calculated). The regression equation for model evaluation for a candidate of the learning model is a regression equation for model evaluation calculated using the second learning data group created for the candidate of the learning model. The regression equation calculation unit 914 for model evaluation stores the candidate of the learning model and the regression equation for model evaluation for the candidate of the learning model in association with each other.

[0090] The evaluation index calculation unit 915 calculates a variability evaluation index V and a gradient evaluation index grad using the second training data (second training data group) and the regression equation for model evaluation, similar to the evaluation index calculation unit 140. However, the evaluation index calculation unit 915 calculates the variability evaluation index V and the gradient evaluation index grad for each of the p second training data groups, using one of the p second training data groups and the regression equation for model evaluation calculated using that second training data group. As a result, one set of variability evaluation index V and gradient evaluation index grad is calculated for each of the p candidate learning models (p sets of variability evaluation index V and gradient evaluation index grad are calculated). The variability evaluation index V and gradient evaluation index grad for a candidate learning model are the variability evaluation index V and gradient evaluation index grad calculated using the regression equation for model evaluation for that candidate learning model. The evaluation index calculation unit 915 stores candidate learning models and the variability evaluation index V and gradient evaluation index grad for those candidate learning models in relation to each other.

[0091] Once the p sets of variability evaluation index V and gradient evaluation index grad are calculated as described above, the explanatory variable selection unit 911 selects from the p sets of variability evaluation index V and gradient evaluation index grad a variability evaluation index V that is paired with a gradient evaluation index grad which indicates that the gradient of the regression equation for model evaluation is non-negative, and which represents the degree of variability of the second training data with respect to the regression equation for model evaluation (the variability evaluation index V with the maximum value in equation (3)). The explanatory variable selection unit 911 then selects an explanatory variable (candidate explanatory variable) included in the candidate learning model stored in association with the selected variability evaluation index V as one of the explanatory variables of the learning model.

[0092] FIG. 11 is a diagram showing an example of a regression equation for model evaluation and second learning data (black circles (●)) obtained in each selection process. FIG. 11(a) is a diagram showing an example of a regression equation 1111 for model evaluation and second learning data (black circles (●)) obtained in the first selection process. Here, a graph showing the regression equation for model evaluation is denoted as the regression equation 1111 for model evaluation (such notation is the same in other figures). Above the graph of FIG. 11(a), a variance evaluation index V (= 0.3293) and a gradient evaluation index grad (= 0.341) for candidates of learning models having explanatory variables selected in the first selection process are shown. Note that the variance evaluation index V for candidates of learning models having explanatory variables selected in each selection process usually becomes the maximum value of the variance evaluation index V calculated in that selection process.

[0093] <Subsequent selection processes after the second time> As described above, in the present embodiment, a case where the stepwise forward selection method (variable addition method) is used as a method for narrowing down explanatory variables to be used in a learning model is exemplified. Therefore, in the s-th selection process after the second time, the number of explanatory variables in the learning model is set to s (1 is added to the number of explanatory variables in the learning model in the (s - 1)-th selection process). In the present embodiment, a case where one explanatory variable is selected in the first selection process is exemplified. Therefore, in the second selection process, the number of explanatory variables in the learning model is 2.

[0094] First, the explanatory variable selection unit 911 individually selects one of the explanatory variables x1 to x p included in the first learning data, excluding the already selected explanatory variables (the explanatory variables selected in <the (s - 1)-th selection process>), as a candidate for the explanatory variable in the learning model, and outputs it to the learning model candidate calculation unit 912. This is done individually for each of the explanatory variables included in the first learning data x1 to x p excluding the already selected explanatory variables. In the s-th selection process, among the explanatory variables x1 to x p included in the first learning data, the explanatory variables excluding the already selected explanatory variables are p - s + 1 explanatory variables.

[0095] Then, the learning model candidate calculation unit 912 calculates that the explanatory variables of the learning model candidate are the s-1 explanatory variables x that have already been selected. r' Then, the explanatory variable selection unit 911 selects a new candidate explanatory variable x r Assuming that this is the case, the calculation of candidate learning models using the first training data is performed using the explanatory variables x1~x included in the first training data. p Of these, s-1 explanatory variables x have already been selected. r' This process is performed for each of the explanatory variables except for . This generates p-s+1 candidate learning models. In the second s-th selection process, r is an integer between 1 and p-s+1 (inclusive). In this case, the s-1 explanatory variables x that have already been selected are used. r' Each of the p-s+1 explanatory variables, excluding the one specified, is a candidate explanatory variable (an explanatory variable in the candidate learning model).

[0096] Note that s-1 explanatory variables x have already been selected. r' And, one candidate explanatory variable x r When calculating candidate learning models having and , the learning model candidate calculation unit 912 calculates the already selected s-1 explanatory variables x r' and candidate explanatory variable x r The data and the correct value y of the target variable paired with that data in the first training data. t By using this as a set of training data, a candidate learning model is calculated. Furthermore, the candidate learning models in the s-th selection process from the second onward are represented, for example, by equation (7) below. As shown in equation (7) below, in this embodiment, we illustrate the case where the candidate learning models in the s-th selection process from the second onward are multiple regression equations.

[0097]

number

[0098] In the section [Second Learning Data Creation Process (S202), Second Learning Data Creation Unit 120], the second learning data creation unit 913 replaces "learning model" with "candidate learning model" and replaces "first learning data" with "learning data used when calculating the candidate learning model," thereby creating second learning data for each of the p-s+1 candidate learning models calculated by the candidate learning model calculation unit 912.

[0099] Specifically, the second training data creation unit 913 generates the explanatory variables x in the training model included in the training data used when calculating candidate training models. r' , x r By substituting the data values ​​into the candidate learning model (equation (7)), we can obtain the estimated value of the target variable y in the candidate learning model. e The second training data creation unit 913 then calculates the estimated value y of the target variable in the candidate training model. e And, in the training data used when calculating the candidate learning model, the estimated value of the target variable y e Explanatory variable x used in the calculation r' , x r The correct value of the target variable y in the learning model paired with the data. t The second set of Q training data is created as training data consisting of sets of data including and . The second training data creation unit 913 creates such a set of Q second training data for each of the p-s+1 candidate learning models (for each of r=1 to p-s+1) calculated by the candidate learning model calculation unit 912. As a result, p-s+1 second training data sets are created as second training data sets consisting of the second set of Q second training data.

[0100] The model evaluation regression equation calculation unit 914 calculates a model evaluation regression equation (regression coefficient c) using the second training data (second training data group), similar to the model evaluation regression equation calculation unit 130. However, the model evaluation regression equation calculation unit 914 calculates the model evaluation regression equation using each of the p-s+1 second training data groups individually. As a result, one model evaluation regression equation is calculated for each of the p-s+1 candidate learning models (p-s+1 model evaluation regression equations are calculated). The model evaluation regression equation calculation unit 914 stores the candidate learning models and the model evaluation regression equation for each candidate learning model in relation to each other.

[0101] The evaluation index calculation unit 915 calculates a variability evaluation index V and a gradient evaluation index grad using the second training data (second training data group) and the regression equation for model evaluation, similar to the evaluation index calculation unit 140. However, the evaluation index calculation unit 915 calculates the variability evaluation index V and the gradient evaluation index grad for each of the p-s+1 second training data groups, using one of the p-s+1 second training data groups and the regression equation for model evaluation calculated using that second training data group. As a result, one set of variability evaluation index V and gradient evaluation index grad is calculated for each of the p-s+1 candidate learning models (p-s+1 sets of variability evaluation index V and gradient evaluation index grad are calculated). The evaluation index calculation unit 915 stores the candidate learning model and the variability evaluation index V and gradient evaluation index grad for that candidate learning model in relation to each other.

[0102] Once the p-s+1 pairs of variability evaluation index V and gradient evaluation index grad are calculated as described above, the explanatory variable selection unit 911 selects from the p-s+1 pairs of variability evaluation index V and gradient evaluation index grad a variability evaluation index V that is paired with a gradient evaluation index grad which indicates that the gradient of the regression equation for model evaluation is non-negative, the variability evaluation index V that indicates that the degree of variability of the second training data with respect to the regression equation for model evaluation is minimal (the variability evaluation index V with the maximum value in equation (3)). Then, the explanatory variable selection unit 911 selects one of the explanatory variables (candidate explanatory variable) among the explanatory variables included in the candidate learning model stored in association with the selected variability evaluation index V, excluding the explanatory variables that have already been selected, as one of the explanatory variables of the learning model.

[0103] Figures 11(b) and 11(c) show examples of regression equations 1112 and 1113 for model evaluation and the second training data (black circles (●)) obtained in the second and third selection processes, respectively. Above the graphs in Figures 11(b) and 11(c), the variability evaluation index V (=0.5935, 0.766) and gradient evaluation index grad (=0.627, 0.778) for candidate learning models with explanatory variables selected in the second and third selection processes are shown.

[0104] Figure 12 shows an example of the regression equations for model evaluation and the second training data (black circles (●)) obtained in each selection process following Figure 11. Figures 12(a), 12(b), and 12(c) show an example of the regression equations 1211, 1212, and 1213 for model evaluation and the second training data (black circles (●)) obtained in the 4th, 5th, and 6th selection processes, respectively. Above the graphs in Figures 12(a), 12(b), and 12(c), the variability evaluation index V (=0.8648, 0.9209, 0.9573) and gradient evaluation index grad (=0.862, 0.925, 0.957) for candidate learning models with explanatory variables selected in the 4th, 5th, and 6th selection processes are shown.

[0105] As shown in Figures 11 and 12, it can be seen that by repeating the selection process, the variability evaluation index V and the gradient evaluation index grad for the candidate learning models approach 1, respectively. Therefore, by repeating the selection process, the estimated value y of the target variable in the candidate learning model can be obtained. e This can increase the reliability of the model (it can ensure that the dependent variable is well explained by the explanatory variables).

[0106] <Conditions for ending the selection process> Once the explanatory variable selection unit 911 has selected explanatory variables for the learning model as described above, it determines whether or not the search termination condition is met. The search termination condition may be defined, for example, using a variability evaluation index V for candidate learning models having the explanatory variables selected in each selection process. Specifically, the explanatory variable selection unit 911 may define the condition as the variability evaluation index V for candidate learning models having the explanatory variables selected in each selection process exceeding a threshold. Alternatively, the search termination condition may be defined using a histogram of the variability evaluation index V calculated in each selection process (one selection process). For example, the search termination condition may be that the variance of the histogram of the variability evaluation index V calculated in each selection process (one selection process) is less than or equal to a predetermined value. Alternatively, the search termination condition may be defined using the maximum value of the variability evaluation index V calculated in each selection process and at least one of the variability evaluation index V other than the maximum value. In this case, for example, the search termination condition could be that the difference between the maximum value of the variability evaluation index V calculated in each selection process and the second largest value of the variability evaluation index V is less than or equal to a predetermined value. Alternatively, the search termination condition could be that the number of selection process iterations is a predetermined value.

[0107] The process described in the section "Selection Process from the Second Time Onward" is executed repeatedly until the search termination condition is met. As mentioned above, the explanatory variables x1~x included in the first training data pThe method for narrowing down the explanatory variables to be used in the learning model from among them is not limited to stepwise forward selection (variable increment method). For example, explanatory variables x1 to x included in the first training data p When using variable reduction or variable addition as a method to narrow down the explanatory variables to be used in the learning model, the number of explanatory variables selected in the first selection process (initial value of the number of explanatory variables) is p. Also, the explanatory variables x1 to x included in the first training data p When using the variable reduction method to narrow down the explanatory variables to be used in the learning model, in the sth selection process from the second time onward, the number of explanatory variables in the candidate learning model is set to p-s+1 (the number of explanatory variables in the candidate learning model is obtained by subtracting 1 from the number of explanatory variables in the learning model in the s-1th selection process). Also, the explanatory variables x1~x included in the first training data p When using the variable addition / subtraction method or the variable subtraction / addition method as a technique to narrow down the explanatory variables to be used in the learning model, the number of explanatory variables in the learning model in the sth selection process from the second onward is set to the number of explanatory variables in the learning model in the (s-1)th selection process plus or minus 1.

[0108] In the s-th selection process, if the number of explanatory variables is to be reduced, for example, the explanatory variable selection unit 911 selects the explanatory variables obtained by removing one explanatory variable from the candidate explanatory variables of the learning model obtained in the (s-1)th selection process as candidate explanatory variables and outputs them to the learning model candidate calculation unit 912. The one explanatory variable to be removed from the candidate explanatory variables of the learning model obtained in the (s-1)th selection process is selected one by one from the candidate explanatory variables of the learning model obtained in that (s-1)th selection process.

[0109] The learning model candidate calculation unit 912 calculates a learning model having candidate explanatory variables output from the explanatory variable selection unit 911 as a candidate learning model for the s-th selection process. If the number of explanatory variables for the candidate learning model obtained in the s-1th selection process is c, then c candidate learning models are obtained as candidates for the s-th selection process. Subsequently, the second learning data creation unit 913, the regression equation calculation unit 914 for model evaluation, and the evaluation index calculation unit 915, as described in the section <Selection Processes from the Second Time Onward>, perform a process in which "p-s+1 candidate learning models" are replaced with "c candidate learning models".

[0110] [Explanatory variable determination process (S1007), explanatory variable determination unit 916] The explanatory variable determination unit 916 determines the explanatory variables selected by the explanatory variable selection unit 911 at the time the search termination condition is met as explanatory variables in the learning model. For example, if the search termination condition is met in the 6th selection process, the explanatory variable determination unit 916 determines the explanatory variables selected by the explanatory variable selection unit 911 in the 1st to 6th selection processes as explanatory variables in the learning model. When using the variable augmentation method, if the search termination condition is met in the 6th selection process, the number of explanatory variables in the learning model (the number of explanatory variables selected by the explanatory variable selection unit 911 in the 1st to 6th selection processes) will be 6.

[0111] [Storage process (S1008), storage section 920] The memory unit 920 stores information about a learning model that has the explanatory variables determined by the decision unit 910 (explanatory variable determination unit 916) as explanatory variables and the target variable included in the first training data as the target variable.

[0112] [Estimation of the dependent variable] Next, referring to the flowchart in Figure 10B, an example of a processing method for calculating an estimated value of the target variable using the learning model of this embodiment will be explained, along with an example of the functions of the processing unit 900 shown in Figure 9. The processing shown in the flowchart in Figure 10B starts after the learning model has been stored in the storage unit 920 in the processing shown in the flowchart in Figure 10A.

[0113] [Data acquisition process for estimation (S1011), Data acquisition unit for estimation 930] The estimation data acquisition unit 930 acquires estimation data that includes the data of the explanatory variables determined by the determination unit 910 (explanatory variable determination unit 916). In this embodiment, an example is given in which the estimation data does not include the data of the target variable.

[0114] As a method for acquiring estimation data, at least one of the following is employed: transmission from an external device, input of estimation data by an operator through a user interface, and reading from a portable storage medium.

[0115] [Dependent variable calculation process (S1012), dependent variable calculation unit 940] The target variable calculation unit 940 calculates an estimated value of the target variable by providing estimation data to the learning model stored in the memory unit 920.

[0116] [Output process (S1012), output section 950] The output unit 950 outputs information about the estimated value of the dependent variable calculated by the dependent variable calculation unit 940. The output format of the dependent variable information is at least one of the following: display on a computer display, transmission to an external device, and storage on a portable storage medium.

[0117] [summary] As described above, in this embodiment, the processing unit 900 processes the explanatory variables x1~x included in the first training data. pUsing some of the explanatory variables and candidate learning models that show the relationship between the explanatory variables and the target variable explained by those explanatory variables, evaluation indices (variability evaluation index V and gradient evaluation index grad) are calculated as described in the first embodiment, and the explanatory variables in the learning model are determined based on the evaluation indices for each of the multiple candidates. Therefore, explanatory variables that explain the target variable well can be selected. For example, by using the variability evaluation index V and gradient evaluation index grad as the selection conditions for factors (explanatory variables) in the sequential feature selection method, it is possible to suppress the convergence of the finally selected explanatory variables to an undesirable local optimum. Furthermore, when selecting explanatory variables as described in Patent Document 1, if there are many mutually correlated explanatory variables, the number of explanatory variables that can be selected may be limited, and therefore the estimation accuracy of the target variable may not be sufficiently improved. In contrast, in this embodiment, since explanatory variables are selected using the variability evaluation index V and gradient evaluation index grad, the estimation accuracy of the target variable can be improved compared to the method described in Patent Document 1, even when there are many mutually correlated explanatory variables.

[0118] (Third embodiment) Next, a third embodiment will be described. In the second embodiment, in the first selection process, a variation evaluation index V that represents the degree of variation of the second training data with respect to the regression equation for model evaluation is selected from p sets of variation evaluation index V and gradient evaluation index grad, which are paired with a variation evaluation index V that represents the non-negative gradient of the regression equation for model evaluation. An explanatory variable included in the candidate training model stored in association with the selected variation evaluation index V is then selected as one of the explanatory variables of the training model. Therefore, if the first training data is the same, the explanatory variable selected in the first selection process is invariant. In contrast, this embodiment exemplifies a case where the explanatory variable selected in the first selection process is variable, even if the first training data is the same. Thus, this embodiment and the second embodiment differ mainly in the first selection process. Therefore, in the description of this embodiment, parts that are the same as those in the first and second embodiments will be denoted by the same reference numerals as those in Figures 1 to 12, and detailed explanations will be omitted. Furthermore, in this embodiment, for the sake of simplicity, we will illustrate the case where the learning model is a linear regression equation.

[0119] <First selection process> In this embodiment, the explanatory variable selection unit 911 selects the explanatory variables x1 to x included in the first training data. p An example is given where one of the options is randomly selected using a random number generator or the like. In this embodiment, as in the second embodiment, the calculation of candidate learning models, the creation of second training data using the candidate learning models, and the calculation of evaluation metrics using the second training data may or may not be performed in the first selection process. Subsequent selection processes are as described in the section "Subsequent Selection Processes" of the second embodiment.

[0120] Figure 13 shows an example of the regression equation 1311 for model evaluation and the second training data (black circles (●)) obtained in the first selection process. Above the graph in Figure 13, the variability evaluation index V (=0.0219) for candidate learning models with explanatory variables selected in the first selection process is shown. Figure 14 shows an example of the regression equations 1411, 1412 for model evaluation and the second training data (black circles (●)) obtained in the second and subsequent selection processes (the upper figure shows the second selection process, and the lower figure shows the seventh selection process). Above the respective graphs in Figure 14, the variability evaluation index V (=0.3202, 0.9084) for candidate learning models with explanatory variables selected in the second and seventh selection processes is shown. Note that the variability evaluation index V for candidate learning models with explanatory variables selected in each selection process is usually the maximum value of the variability evaluation index V calculated in that selection process.

[0121] Figure 15 shows an example of the relationship between the number of iterations of the selection process and the variability evaluation index V for candidate learning models that have the explanatory variables selected in the selection process. Note that the vertical axis of V in Figure 15 max This indicates the maximum value of the variability evaluation index V in each selection process. As shown in Figures 13 to 15, even if one explanatory variable is randomly selected in the first selection process, by repeating the selection process, the estimated value of the target variable y in the learning model can be obtained. e This can increase the reliability of the model (it can ensure that the dependent variable is well explained by the explanatory variables).

[0122] For example, as explained in the section on [First Training Data], when estimating microbial species that influence the rate of change in the amount of a specific substance before and after treatment in a biological wastewater treatment process using the method of this embodiment, the results show that by changing the OTU (denovo) selected in the first selection process, the explanatory variables (OTU) selected in the second to seventh selection processes change, as shown in Figure 16. Therefore, multiple explanatory variables can be obtained as explanatory variables that are ultimately selected as explanatory variables in the learning model.

[0123] (Fourth Embodiment) Next, a fourth embodiment will be described. In the third embodiment, in the first selection process, the explanatory variables x1~x included in the first training data are selected. p An example was given of randomly selecting one of them. In contrast, in this embodiment, in the first selection process, the explanatory variables x1~x are selected using the method described in Patent Document 1. p This example illustrates the case where at least one (one or more) of the following is selected. Thus, this embodiment differs from the third embodiment mainly in the first selection process. Therefore, in the description of this embodiment, parts that are the same as those in the first to third embodiments will be denoted by the same reference numerals as those used in Figures 1 to 16, and detailed explanations will be omitted. Also, in this embodiment, for the sake of simplicity, the example will illustrate the case where the learning model is a linear regression equation.

[0124] <First selection process> In this embodiment, the explanatory variable selection unit 911 creates a sample by resampling using the first training data, calculates the regression coefficients of the learning model (regression equation) by performing a penalty regression analysis that can reduce the regression coefficients to 0 (zero) using the sample, and selects explanatory variables (group of explanatory variables) in the learning model based on the calculated regression coefficients (i.e., in this embodiment, at least the first selection step described in Patent Document 1 is adopted as an example). One group of explanatory variables consists of explanatory variables x1 to x included in the first training data. p It consists of one or more explanatory variables from the following list.

[0125] As described in Patent Document 1, methods such as the Bootstrap method and the Jackknife method can be used as resampling techniques. Furthermore, as a penalized regression analysis that can reduce regression coefficients to zero, regression equations equipped with a regularization term of the L1 norm, such as Lasso (Least Absolute Shrinkage and Selection Operator), Elastic net, and SCAD (Smoothly Clipped Absolute Deviation), can be used.

[0126] Alternatively, the confidence level can be calculated from the selection frequency of the explanatory variables selected based on the regression coefficients in the sample created by resampling, and the explanatory variables can be further selected based on this confidence level (i.e., the second selection step described in Patent Document 1 can be adopted). If B sets of samples are created by resampling and analyzed, and the number of selections for each group of explanatory variables is U, then the selection frequency of each group of explanatory variables selected based on the regression coefficients can be expressed as U / B, and this value of U / B can be used as the confidence level.

[0127] Alternatively, a group of explanatory variables selected based on regression coefficients, etc., may be used as explanatory variables (independent variables), and the target variable included in the first training data may be used as the target variable (dependent variable). A regression analysis may then be performed using the first training data to further select a group of explanatory variables that show either a positive or negative correlation (i.e., the third selection step described in Patent Document 1 may be adopted).

[0128] Furthermore, a principal component regression analysis with a regularization term may be performed using a group of explanatory variables selected based on regression coefficients, etc., as explanatory variables (independent variables), and the target variable included in the first training data as the target variable (dependent variable), and a group of explanatory variables showing at least a positive or negative correlation may be further selected (i.e., the fourth selection step described in Patent Document 1 may be adopted). As described in Patent Document 1, it is preferable to have about 1 to 5 principal component axes. In addition, it is preferable to use a one-step principal component regression model with sparse regularization (sparse principal component regression (SPCR) model) for principal component regression (PCR) analysis.

[0129] Alternatively, a regression analysis may be performed using a group of explanatory variables selected based on regression coefficients, etc., as explanatory variables (independent variables), and the target variable included in the first training data as the target variable (dependent variable), and a further group of explanatory variables may be selected based on the p-value (i.e., the fifth selection step described in Patent Document 1 may be adopted). Alternatively, a calculation of the Akaike Information Criterion may be performed using a group of explanatory variables selected based on regression coefficients, etc., as explanatory variables (independent variables), and the target variable included in the first training data as the target variable (dependent variable), and a further group of explanatory variables may be selected based on the obtained standard value (AIC) (i.e., the fifth selection step described in Patent Document 1 may be adopted). The selection process from the second time onward is as described in the section "Selection Process from the Second Time Onward" of the second embodiment.

[0130] Figure 17 shows an example of the explanatory variables (OTU) selected in the first selection process. Figure 17 illustrates the case where five explanatory variables (OTU) are selected in order of their confidence level, as mentioned above. Figure 18 shows an example of the regression equation 1811 for model evaluation and the second training data (black circles (●)) obtained in the first selection process. Above the graph in Figure 18, the variability evaluation index V (=0.8313) and gradient evaluation index grad (=0.85) for the candidate learning model with the explanatory variables selected in the first selection process are shown.

[0131] Figure 19 shows examples of regression equations 1911, 1912, and 1913 for model evaluation, and the second training data (black circles (●)) obtained from the second and subsequent selection processes (the top figure shows the second selection process, the middle figure shows the third selection process, and the bottom figure shows the sixth selection process). Above each graph in Figure 19 are the variability evaluation index V (=0.9136, 0.9538, 0.9911) and gradient evaluation index grad (=0.919, 0.956, 0.986) for candidate learning models with explanatory variables selected in the second, third, and sixth selection processes. Note that the variability evaluation index V for candidate learning models with explanatory variables selected in each selection process is usually the maximum value of the variability evaluation index V calculated in that selection process.

[0132] Figure 20 shows an example of the relationship between the number of iterations of the selection process and the variability evaluation index V for candidate learning models that have the explanatory variables selected in the selection process. Note that, as with Figure 15, the vertical axis of V in Figure 20 is max This indicates the maximum value of the variability evaluation index V in each selection process. As shown in Figures 18 to 20, even if the explanatory variables in the learning model are selected using the method described in Patent Document 1 during the first selection process, by repeating the selection process, the estimated value of the target variable y in the learning model can be obtained. e The reliability can be increased (the dependent variable can be better explained by the explanatory variables). Furthermore, by changing parameters such as the regularization parameter, the explanatory variables selected in the first selection process can be changed in this embodiment as well, similar to the third embodiment.

[0133] (Fifth embodiment) Next, a fifth embodiment will be described. In this embodiment, the case in which the learning model has a nonlinear kernel function is illustrated. Thus, this embodiment differs from the first to fourth embodiments mainly in its configuration and processing due to the different learning model. Therefore, in the description of this embodiment, parts that are the same as those in the first to fourth embodiments will be denoted by the same reference numerals as those used in Figures 1 to 20, and detailed explanations will be omitted.

[0134] Figure 21 shows an example of the functional configuration of the processing unit 2100 in this embodiment. The hardware of the processing unit 2100 is implemented using, for example, the same hardware as the processing unit 100 in the first embodiment.

[0135] In Figure 21, the processing unit 2100 includes a first learning data acquisition unit 110, a determination unit 910, a storage unit 920, an estimation data acquisition unit 930, a target variable calculation unit 940, a target variable correction unit 2110, and an output unit 950.

[0136] The functions of the first learning data acquisition unit 110 are the same as those of the first to fourth embodiments, for example. However, the first learning data acquisition unit 110 of this embodiment acquires first learning data corresponding to the learning model of this embodiment, which will be described later.

[0137] Furthermore, the functions of the determination unit 910 are the same as those of the determination unit 910 described in the second to fourth embodiments, for example. In this case, the determination unit 910 further comprises an explanatory variable selection unit 911, a learning model candidate calculation unit 912, a second learning data creation unit 913, a regression equation calculation unit for model evaluation 914, an evaluation index calculation unit 915, and an explanatory variable determination unit 916. However, the determination unit 910 of this embodiment performs the processing described in the second to fourth embodiments using the learning model of this embodiment, which will be described later.

[0138] The memory unit 920 has the same functions as the memory unit 920 described in the second to fourth embodiments, for example, and stores information about the learning model. However, the memory unit 920 of this embodiment stores information about the learning model of this embodiment, which will be described later.

[0139] The functions of the estimation data acquisition unit 930 are the same as those described in the second to fourth embodiments, for example, and it acquires estimation data. However, the estimation data acquisition unit 930 of this embodiment acquires estimation data that includes data for explanatory variables in the learning model of this embodiment, which will be described later.

[0140] The functions of the target variable calculation unit 940 are the same as those described in the second to fourth embodiments, for example, and it calculates an estimated value of the target variable by providing the explanatory variable data acquired by the estimation data acquisition unit 930 to the learning model. However, the target variable calculation unit 940 in this embodiment provides the explanatory variable data acquired by the estimation data acquisition unit 930 to the learning model of this embodiment, which will be described later.

[0141] The dependent variable correction unit 2110 corrects the estimated value of the dependent variable calculated by the dependent variable calculation unit 940. An example of a method for correcting the estimated value of the dependent variable will be described later. The output unit 950 has the same functions as the output unit 950 described in the second to fourth embodiments, for example, and outputs information on the estimated value of the target variable. However, the output unit 950 in this embodiment outputs information on the estimated value of the target variable corrected by the target variable correction unit 2110.

[0142] Figure 22A is a flowchart illustrating an example of the processing method for creating the learning model of this embodiment. Figure 22B is a flowchart illustrating an example of the processing method for calculating the estimated value of the target variable using the learning model of this embodiment.

[0143] [Learning Model] First, an example of the learning model in this embodiment will be described. In this embodiment, we will illustrate the case where the learning model has a nonlinear kernel function. The nonlinear kernel function is not limited, but in this embodiment, we will illustrate the case where the nonlinear kernel function is a Gaussian kernel.

[0144] Furthermore, this embodiment illustrates a case where the learning model has the concentration of a diffusive substance at one or more locations within an object as an explanatory variable, and the concentration of the diffusive substance at other locations within the object as an objective variable. Thus, this embodiment illustrates a case where the learning model is a learning model that performs data interpolation (at least one of interpolation and extrapolation). For example, data interpolation may be performed by calculating a physical quantity at other locations in a target region (space or plane) based on a known physical quantity at multiple locations in the target region (space or plane). Alternatively, data interpolation may be performed by calculating a physical quantity at other times at one or more locations in a target region (space or plane) based on a known physical quantity at multiple times.

[0145] Furthermore, this embodiment illustrates a case where the correct value of the target variable in the learning model is calculated by numerical analysis and included in the first training data. Figure 23 shows an example of an object to be subjected to numerical analysis. In this embodiment, the example shows the object to be subjected to numerical analysis as a solid such as a material. However, the object to be subjected to numerical analysis may also be a liquid or a gas.

[0146] Figure 23(a) is a front view, and Figure 23(b) is a top view. In Figure 23, material 2300 has a rectangular parallelepiped with a width of 1 m, a depth of 1 m, and a height of 0.32 m, with a recess measuring 0.2 m in width, 1 m in depth, and 0.14 m in height formed at the upper left end of the rectangular parallelepiped.

[0147] Below is an example of the conditions used in the actual numerical analysis, where the diffusion coefficient of material 2300 was set to 16.5 m 2 The time interval was set to / s. Furthermore, a Neumann boundary condition of flux = 0 (zero) was set for the sides 2302, 2303 and bottom surface 2304 of material 2300. The initial concentration of the diffusive substance within material 2300 was also set to 0 (zero). The transient change in the concentration distribution of the diffusive substance at each coordinate on the bottom surface 2304 of material 2300, assuming that at time = 0 (initial time), the diffusive substance with a concentration of 1.0 is in contact with all coordinates on the top surface 2301 of material 2300, was calculated by performing a numerical analysis using the transient three-dimensional diffusion equation. The finite difference method was used for the numerical analysis. In this case, the coordinates are those of the mesh (elements).

[0148] Note that in Figure 23(a), for the sake of notation, the dashed lines representing the top surface 2301, sides 2302 and 2303, and bottom surface 2304 of material 2300 are shown separated from the solid line representing the surface of material 2300; however, in reality, these dashed lines and the solid line coincide. In the following explanation, the concentration of the diffusing substance will be referred to as the diffusing substance concentration, as needed.

[0149] When the distribution of diffusive substance concentration in material 2300 is approximated by a 3D Gaussian distribution using Gaussian process regression, the coordinates x1~x of the base surface 2304 of material 2300 are... N , x * N+1~x * M Diffusing substance concentrations y1~y N , y * +1 ~y * M x can be represented as a random variable, for example, as shown in Figure 24. * N+1 ~x * M , y * N+1 ~y * M In Figure 24 and the equations described later, this corresponds to the symbols with an asterisk (*) above N+1 to M (this notation is the same for other symbols as well).

[0150] In this embodiment, the coordinates x1 to x within the material 2300 are N Diffusing substance concentrations y1~y N This example illustrates the case where is the explanatory variable. In this embodiment, the coordinate x within the material 2300 is also shown. * N+1 ~x * M Diffusing substance concentration y * N+1 ~y * M Let's take an example where the dependent variable is the concentration of the diffusing substance y, as will be explained later. * N+1 ~y * M mean μ(y * N+1 )~μ(y * M Let's take an example where ) is the dependent variable.

[0151] In the following explanation, the coordinates x1 to x within material 2300 are N Diffusing substance concentrations y1~y N (Explanatory variable: Diffused substance concentration) If necessary, set the reference diffused substance concentration y1~y N It is called that. Also, the coordinate x within material 2300 * N+1 ~x * M Diffusing substance concentration y* N+1 ~y * M (The target variable is the concentration of the diffusing substance) and, if necessary, the estimated diffusing substance concentration y * N+1 ~y * M It is called that.

[0152] Also, for the sake of explanation, in this embodiment, coordinates x1~x N , x * N+1 ~x * M The coordinates are not fixed. Specifically, coordinates x1 to x N Let x be the coordinate of the 1st to Nth reference diffusing substance concentrations. * N+1 ~x * M Let x be the coordinates of the estimated diffusive substance concentrations from the 1st to the MNth. Also, these coordinates x1 to x N , x * N+1 ~x * M The total number of elements is assumed to be constant at M. For example, in Figure 23, the reference diffusing substance concentrations y1 to y N For coordinates having , the elements are ranked from 1st to Nth in ascending order, starting with those closest to one vertex 2305 on the base surface 2304 of material 2300. If the distance from vertex 2305 is the same, the elements are ranked in ascending order, starting with those closest in the width direction. Estimated diffusive substance concentration y * N+1 ~y * M Similarly, for coordinates that have a specific value, we assign them a sequence from the 1st to the MNth value.

[0153] N(μ,σ 2 ) has a mean of μ and a variance of σ 2 This indicates that the probability density function follows a normal distribution. Note that in Figure 24, N(μ,σ) 2 μ in ) is an M x 1 zero matrix, and N(μ,σ 2 ) of σ 2However, let's take the case where it is an M x M matrix as an example. k(x,x') is a Gaussian kernel, which is one of the nonlinear kernel functions represented by equation (8) below. The nonlinear kernel function may also be an exponential kernel, a periodic kernel, a Matern kernel, or other kernel functions.

[0154]

number

[0155] Here, θ is a tuning parameter used to adjust the spread of the Gaussian function. Note that x and x' in equation (8) are x1 to x N , x * N+1 ~x * M One of the following is given. Also, in Figure 24, for the sake of notation, the variance of the normal distribution σ is shown. 2 To indicate which coordinates each row and column of the matrix represents store the kernel function values ​​for, the following are added above and to the left of the matrix: x1···x N , x * N+1 ···x * M (Note that in Figure 24, x1···x N , x * N+1 ···x * M (This does not appear in the actual mathematical formula). For example, the variance σ of the normal distribution. 2 The first row and first column of the matrix representing the coordinates x1, x1, store the value of the Gaussian kernel k(x1,x1) for each coordinate.

[0156] Reference diffusive substance concentrations y1~y N This is represented by an N x 1 matrix y as shown in equation (9) below, and the estimated diffusing substance concentration y * N+1 ~y * M This is represented by the following equation (10), which is a matrix y with MN rows and 1 column. * Expressed as follows, the estimated diffusive substance concentration y * N+1 ~y *M mean μ(y * N+1 )~μ(y * M A matrix μ(y) with MN rows and 1 column that stores ) * ) can be expressed, for example, by equation (11) below.

[0157]

number

[0158] Here, T represents the transpose matrix, and -1 represents the inverse matrix. Also, k(X,X * ), k(X,X) is expressed by equations (12) and (13) below.

[0159]

number

[0160] k(X,X * ) is the variance σ of the normal distribution in Figure 24. 2 This is a matrix that stores the elements from column N+1 to column M of the elements from row 1 to row N of the matrix representing (in Figure 24, the vertically arranged "x1···x N Among the area to the right of ", the horizontally arranged "x * N+1 ···x * M (See the lower area of ​​"). Also, k(X,X) is the variance of the normal distribution σ in Figure 24. 2 This is a matrix that stores the elements from column 1 to column N of the elements from row 1 to row N of the matrix representing (in Figure 24, the elements "x1···x" are arranged vertically). N Among the areas to the right of "x1···x N (See the area below ").

[0161] As mentioned above, in this embodiment, the reference diffusing substance concentrations y1~y N The explanatory variable is the mean μ(y) of the estimated diffusing substance concentration. * N+1 )~μ(y* M Let's take an example where the dependent variable is y1~y. N The fewer the number of coordinates (value of N), the fewer the coordinates x1...x N Based on the diffusing substance concentration at x, numerous coordinates x * N+1 ···x * M The concentration of diffusing substances can be estimated. However, the reference diffusing substance concentration y1~y N If the number of (value of N) is too small, the average μ(y) of the estimated diffusing substance concentration will be low. * N+1 )~μ(y * M There is a risk that the estimation accuracy of the average μ(y) of the diffusing substance concentration will decrease. * N+1 )~μ(y * M Within the range where the estimation accuracy does not decrease, the smallest possible number of reference diffusing substance concentrations y1~y N It is preferable to select this as an explanatory variable. In this embodiment, a learning model represented by the formula shown in Figure 24 is created (reference diffusing substance concentrations y1~y N This provides an example of a case where (deciding) this is determined.

[0162] [Creating a learning model] First, referring to the flowchart in Figure 22A, an example of the processing method for creating the learning model in this embodiment will be explained along with an example of the functions of the processing unit 2100 shown in Figure 21. Note that the flowchart in Figure 22A uses a different learning model in each step of the flowchart in Figure 10A described in the second embodiment. Therefore, this explanation will focus on the differences from each step of the flowchart in Figure 10A. In this embodiment, the method of the second embodiment is used as an example. However, the methods of the third and fourth embodiments may also be applied to this embodiment.

[0163] [First learning data acquisition process (S2201), first learning data acquisition unit 110] The first learning data acquisition unit 110 acquires multiple sets (Q sets) of first learning data. In this embodiment, the first learning data acquisition unit 110 acquires all coordinates x1 to x of the bottom surface 2304 of the material 2300. N , x * N+1 ~x * M As an example, we will show a case where the value of the diffusing substance concentration in is obtained by the numerical analysis described above. In this embodiment, we will show a case where Q is a variable that identifies the time. Therefore, the first learning data acquisition unit 110 acquires the first learning data at each of the Q time points.

[0164] In this case, the coordinates x1 to x calculated at the same time are N , x * N+1 ~x * M Diffusing substance concentrations y1~y N , y * N+1 ~y * M A set of first training data is formed using the values ​​of x1 to x1, which are calculated at the same time. N , x * N+1 ~x * M Among the values ​​of the diffusing substance concentration in the coordinate x1~x N Values ​​of diffusing substance concentrations in (reference diffusing substance concentrations y1~y) N The value of the explanatory variable is the value of the data, and the estimated diffusing substance concentration y * N+1 ~y * M The value of this variable is the ground truth value of the dependent variable that is paired with the data of the explanatory variable.

[0165] Note that the first training data includes data for explanatory variables and ground truth data for the objective variable, but the diffusion substance concentrations y1~y included in the first training data are not included. N , y * N+1 ~y * MWhether the value of is the data value of the explanatory variable or the ground truth data value (ground truth value) of the dependent variable is determined during the selection process described later.

[0166] In this embodiment, the first learning data acquisition unit 110 acquires coordinates x1 to x calculated at multiple time points. N , x * N+1 ~x * M This example illustrates how to obtain multiple sets (Q sets) of first learning data by acquiring the values ​​of the diffusing substance concentration at x1 to x N , x * N+1 ~x * M The values ​​for the diffusing substance concentration at 50 time points (50 sets of first training data) are obtained.

[0167] [Explanatory variable selection process (S2202), Learning model candidate calculation process (S2203), Second training data creation process (S2204), Regression equation calculation process for model evaluation (S2205), Evaluation index calculation process (S2206), Explanatory variable selection unit 911, Learning model candidate calculation unit 912, Second training data creation unit 913, Regression equation calculation unit for model evaluation 914, Evaluation index calculation unit 915]

[0168] <First selection process> The explanatory variable selection unit 911 selects one coordinate x1 from among the coordinates of the M diffusive substance concentrations included in the first training data as a candidate coordinate for the explanatory variable (reference diffusive substance concentration) in the learning model, and outputs it to the learning model candidate calculation unit 912, individually for each of the M diffusive substance concentrations included in the first training data.

[0169] Then, in Figure 24, the learning model candidate calculation unit 912 selects y1 as the reference diffusion substance concentration candidate at coordinate x1, which is one of the M diffusion substance concentrations included in the first learning data, and sets the other diffusion substance concentrations as the target variable (y * 2~y * MEquation (11), which is set as ), is calculated as a candidate for the learning model. This is done for each of the M-1 diffusion substance concentrations included in the first learning data, excluding the candidate for the reference diffusion substance concentration. This results in the calculation of M-1 candidate learning models.

[0170] The second training data creation unit 913 creates second training data for each of the M-1 candidate learning models calculated by the candidate learning model calculation unit 912.

[0171] Specifically, the second training data creation unit 913 substitutes the value of one candidate reference diffusing substance concentration y1 included in the first training data used when calculating the candidate training model into the candidate training model, thereby obtaining the estimated value of the target variable in the candidate training model (average μ(y) of the estimated diffusing substance concentration). * 2_e1_i )~μ(y * M_e1_i The value of )) is calculated. Candidate learning models in the first selection process are those with diffusion substance concentrations other than the reference diffusion substance concentration y1 as the target variable (y * 2~y * M Equation (11) is given by ).

[0172] The second learning data creation unit 913 generates the estimated value of the target variable (average μ(y) of the estimated diffusive substance concentration) in such a candidate learning model. * 2_e1_i )~μ(y * M_e1_i The calculation of (i is an integer from 1 to Q) is performed for each of the first training data in group Q (where i is an integer from 1 to Q). The estimated value of the target variable (mean μ(y) of the estimated diffusing substance concentration in one candidate learning model is calculated. * 2_e1_i )~μ(y * M_e1_i Q values ​​of )) are calculated.

[0173] The second training data creation unit 913 generates the estimated values ​​of Q target variables (average μ(y) of the estimated diffusive substance concentration) for each of the M-1 candidate training models as described above. * 2_e1_i )~μ(y* M_e1_i Calculate )).

[0174] Then, the second learning data creation unit 913 calculates the estimated value of the target variable (average μ(y) of the estimated diffusive substance concentration) in one candidate learning model calculated as described above. * 2_e1_i )~μ(y * M_e1_i )) and the diffusion substance concentration y paired with the candidate reference diffusion substance concentration y1 used to calculate the estimated value of the target variable in the first training data used when calculating the candidate learning model. * 2_t_i ~y * M_t_i The second set of Q learning data is created as learning data consisting of sets of data including and . The second learning data creation unit 913 creates such a set of Q second learning data for each of the M-1 learning model candidates calculated by the learning model candidate calculation unit 912. As a result, M-1 second learning data groups are created as second learning data groups consisting of the second set of Q second learning data. Note that in the first selection process, the diffusing substance concentration y * 2_t_i ~y * M_t_i The value of this parameter is the correct value of the target variable in the first training dataset.

[0175] The model evaluation regression equation calculation unit 914 calculates a model evaluation regression equation (regression coefficient) using each of the M-1 second training data sets individually. As a result, one model evaluation regression equation is calculated for each of the M-1 candidate learning models (M-1 model evaluation regression equations are calculated). The model evaluation regression equation calculation unit 914 stores the candidate learning models and the model evaluation regression equation for each candidate learning model in relation to each other.

[0176] In this embodiment, the regression equation for model evaluation is represented, for example, by equation (14) below. Therefore, the regression coefficients of the regression equation for model evaluation are the regression coefficient vector [c N+1 ···cM ] T This is represented as follows: N represents the Nth selection process. In the first selection process, N is 1.

[0177]

number

[0178] The evaluation index calculation unit 915 uses the second training data (second training data group) and the regression equation for model evaluation to calculate the variability evaluation index V m And, the gradient evaluation index grad m The evaluation index calculation unit 915 uses one second training data group out of M-1 second training data groups, the regression equation for model evaluation calculated using the second training data group, and to calculate the variability evaluation index V. m And, the gradient evaluation index grad m This process is performed for each of the M-1 second training data sets. This gives us a variability evaluation index V for the M-1 candidate learning models. m and gradient evaluation index grad m Each of these is calculated as one set (Variability evaluation index V) m and gradient evaluation index grad m (The M-1 group is calculated from this set). Variability evaluation index V for candidate learning models. m and gradient evaluation index grad m This refers to the variability evaluation index V calculated using the regression equation for model evaluation for the candidate learning model. m and gradient evaluation index grad m The evaluation index calculation unit 915 calculates the candidate learning model and the variability evaluation index V for the candidate learning model. m and gradient evaluation index grad m Remember them by associating them with each other.

[0179] As described above, the variability evaluation index V of the M-1 group mAnd once the gradient evaluation index grad is calculated, the explanatory variable selection unit 911 uses the gradient evaluation index grad, which indicates that the gradient of the regression equation for model evaluation is non-negative. m The variability evaluation index V is paired with this. m Among these, the variability evaluation index V represents the degree of variability of the second training data relative to the regression equation used for model evaluation, which is minimized. m The variability evaluation index V for the M-1 group m The system selects from the gradient evaluation index grad. The explanatory variable selection unit 911 then selects an explanatory variable (diffusive substance concentration (candidate explanatory variable)) that is included in the candidate learning model stored in association with the selected variability evaluation index V as one of the explanatory variables (reference diffuse substance concentration) of the learning model.

[0180] In this embodiment, the variation evaluation index V m , gradient evaluation index grad m This can be expressed, for example, by equations (15) and (16) below.

[0181]

number

[0182] Here, m is an integer from N+1 to M. Variability evaluation index V m and gradient evaluation index grad m is the coordinate x * N+1 ~x * M It is calculated one by one (MN times each) for each. Variability evaluation index V m , gradient evaluation index grad m These are, respectively, coordinate x * m Variability evaluation index V m , gradient evaluation index grad m This represents the following. Also, as mentioned above, N represents the Nth selection process (where N is the coordinate x1~x N (This is the same as N). In the first selection process, μ(y * m_e1_i ) is μ(y *2_e1_i )~μ(y * M_e1_i ) will be one of the following. In the Nth process, μ(y * m_e1_i ) is μ(y * N+1_e1_i )~μ(y * M_e1_i It will be one of the following.

[0183] In equation (15), μ(y * m_e2_i ) is the average μ(y) of the estimated diffusive substance concentration calculated by the second learning data creation unit 913. * m_e1_i ) and the average μ(y) of the estimated diffusing substance concentration. * m_e1_i In the first training data used when calculating the estimated diffusive substance concentration, the average μ(y * m_e1_i Candidate y of the reference diffusing substance concentration used in the calculation of ) N The concentration of the diffusing substance y is paired with this. * m_t_i This is the average estimated diffusive substance concentration calculated by substituting it into the regression equation for model evaluation.

[0184] Also, in equation (15), y- * m_t This is the average μ(y) of the estimated diffusive substance concentration calculated by the second learning data creation unit 913. * m_e1_i In the first training data used when calculating the estimated diffusive substance concentration, the average μ(y * m_e1_i Candidate y of the reference diffusing substance concentration used in the calculation of ) N The data is paired with the diffusion substance concentration y * m_t_i The mean value (y * m_t_1 ~y * m_t_Q It is the average value.

[0185] <Selection process for the second and subsequent times> In this embodiment, as in the second embodiment, we will illustrate the case where a stepwise forward selection method (variable increment method) is used as a method for narrowing down the explanatory variables used in the learning model.

[0186] The explanatory variable selection unit 911 selects one coordinate x from among the M coordinates of diffusive substance concentrations included in the first training data, excluding the already selected explanatory variable (reference diffusive substance). N The coordinate x of the candidate explanatory variable (reference diffusing substance concentration) in the learning model. N The process of selecting a candidate and outputting it to the learning model candidate calculation unit 912 is performed for each of the MN diffuse substance concentrations included in the first learning data, excluding the already selected reference diffuse substance concentration and the candidate reference diffuse substance concentration. This results in the calculation of MN learning model candidates.

[0187] Then, in Figure 24, the learning model candidate calculation unit 912 selects N-1 reference diffusion substance concentrations from the M diffusion substance concentrations included in the first learning data and the candidate reference diffusion substance concentrations selected as described above, and then calculates y1~y N Let the concentration of other diffusive substances be the objective variable (y * N+1 ~y * M Equation (11), which is set as ), is calculated as a candidate learning model for each of the explanatory variables (M-1 concentrations of diffusive substances) included in the first training data. This results in the calculation of MN candidate learning models.

[0188] The second training data creation unit 913 creates second training data for each of the MN candidate learning models calculated by the candidate learning model calculation unit 912.

[0189] Specifically, the second training data creation unit 913 takes the value of one candidate reference diffusion substance concentration y1 included in the first training data used when calculating candidate training models, and the already selected N-1 reference diffusion substance concentrations y2~y NBy substituting this into the candidate learning model, the estimated value of the target variable in the candidate learning model (average μ(y) of the estimated diffusive substance concentration) can be obtained. * N+1_e1_i )~μ(y * M_e1_i )) is calculated. The candidate learning model in the Nth selection process is the reference diffusive substance concentration y1 and the reference diffusive substance concentrations y2~y N The concentration of diffusive substances other than y is the dependent variable (y * N+1 ~y * M Equation (11) is given by ).

[0190] The second learning data creation unit 913 generates the estimated value of the target variable (average μ(y) of the estimated diffusive substance concentration) in such a candidate learning model. * N+1_e1_i )~μ(y * M_e1_i The calculation of the mean μ(y) of the first training data for group Q is performed. This allows us to obtain the estimated value of the target variable (mean μ(y) of the estimated diffusive substance concentration in one candidate learning model. * N+1_e1_i )~μ(y * M_e1_i Q values ​​of )) are calculated at a time.

[0191] The second training data creation unit 913 generates Q estimates of the target variable (y) for each of the MN candidate training models as described above. * N+1_e1_i )~μ(y * M_e1_i Calculate )).

[0192] Then, the second learning data creation unit 913 calculates the estimated value of the target variable (average μ(y) of the estimated diffusive substance concentration) in one candidate learning model calculated as described above. * N+1_e1_i )~μ(y * M_e1_i )) and the diffusion substance concentration y paired with the candidate reference diffusion substance concentration y1 used to calculate the estimated value of the target variable in the first training data used when calculating the candidate learning model. * N+1_t_i ~y* M_t_i Q sets of second training data are created as training data consisting of a set of data including and . The second training data creation unit 913 creates such Q sets of second training data for each of the MN candidate learning models calculated by the learning model candidate calculation unit 912. As a result, MN sets of second training data are created as second training data sets consisting of Q sets of second training data. Note that in the Nth selection process, the diffusing substance concentration y * N+1_t_i ~y * M_t_i The value of this parameter is the correct value of the target variable in the first training dataset.

[0193] The model evaluation regression equation calculation unit 914 calculates a model evaluation regression equation (regression coefficient) using each of the MN second training data sets individually. As a result, one model evaluation regression equation is calculated for each of the MN candidate learning models (MN model evaluation regression equations are calculated). The model evaluation regression equation calculation unit 914 stores the candidate learning models and the model evaluation regression equation for each candidate learning model in relation to each other.

[0194] The evaluation index calculation unit 915 uses the second training data (second training data group) and the regression equation for model evaluation to calculate the variability evaluation index V m and gradient evaluation index grad m The MN set is calculated from the set of the following. The evaluation index calculation unit 915 calculates the candidate learning model and the variation evaluation index V for the candidate learning model. m and gradient evaluation index grad m The and are stored in relation to each other. Note that the processing of the evaluation index calculation unit 915 in the Nth selection process is the same as the explanation of the evaluation index calculation unit 915 in the section <First Selection Process>, but with "M-1" replaced by "MN".

[0195] As described above, the variability evaluation index V of the MN group m and gradient evaluation index grad mOnce the set is calculated, the explanatory variable selection unit 911 uses the gradient evaluation index grad, which indicates that the gradient of the regression equation for model evaluation is non-negative. m The variability evaluation index V is paired with this. m Among these, the variability evaluation index V represents the degree of variability of the second training data relative to the regression equation used for model evaluation, which is minimized. m (The variability evaluation index V whose value is maximized in equation (15)) m ) and the variability evaluation index V and gradient evaluation index grad of the MN group. m The system selects from the following. The explanatory variable selection unit 911 then selects one of the explanatory variables (diffusive substance concentration (candidate explanatory variable)) from among the candidate explanatory variables (diffusive substance concentration) of the learning model stored in association with the selected variability evaluation index V, excluding the explanatory variable that has already been selected, as one of the explanatory variables (reference diffuse substance concentration) of the learning model.

[0196] <Conditions for ending the selection process> The explanatory variable selection unit 911, after selecting explanatory variables (reference diffusive substance concentrations) for the learning model as described above, determines whether or not the search termination condition is met. The search termination condition can be, for example, the same condition as the search termination condition described in the second embodiment. The process described in the section "Selection Process from the Second Time Onward" is executed repeatedly until the search termination condition is met. As described in the second embodiment, in this embodiment as well, the method for narrowing down the explanatory variables used in the learning model is not limited to the stepwise forward selection method (variable increment method). For example, the modified method exemplified in the second embodiment may be used as a method for narrowing down the explanatory variables used in the learning model.

[0197] [Explanatory variable determination process (S2207), explanatory variable determination unit 916] The explanatory variable determination unit 916 determines the explanatory variables (reference diffusive substance concentrations y1~y) selected by the explanatory variable selection unit 911 when the search termination condition is met. N ) is determined as an explanatory variable in the learning model.

[0198] [Storage process (S2208), storage section 920] The memory unit 920 stores the explanatory variables (reference diffusing substance concentrations y1~y) determined by the determination unit 910 (explanatory variable determination unit 916). N The explanatory variable is (y( * N+1 )~μ(y * M The storage unit 920 stores information about the learning model having )) as the target variable. The storage unit 920 also stores the gradient evaluation index grad stored in association with the learning model by the evaluation index calculation unit 915. m Remember this. As mentioned above, the gradient evaluation index grad m is the coordinate x * m This is a gradient evaluation index in which m is an integer from N+1 to M. In the following explanation, the coordinate x * m The dependent variable in this case (average μ(y) of the estimated diffusing substance concentration * N+1 )~μ(y * M The symbol ) can be changed to μ(y * m ) is written as .

[0199] [Estimation of the dependent variable] Next, referring to the flowchart in Figure 22B, an example of a processing method for calculating an estimated value of the target variable using the learning model of this embodiment will be explained, along with an example of the functions of the processing unit 2100 shown in Figure 21. The processing shown in the flowchart in Figure 22B starts after the learning model has been stored in the storage unit 920 in the processing shown in the flowchart in Figure 22A.

[0200] [Data acquisition process for estimation (S2211), Data acquisition unit for estimation 930] The estimation data acquisition unit 930 uses the explanatory variables (reference diffusing substance concentrations y1~y) determined by the determination unit 910 (explanatory variable determination unit 916). N Estimation data including the data of the target variable (average μ(y) of the estimated diffusive substance concentration) is obtained. In this embodiment, the estimation data includes the target variable (average μ(y) of the estimated diffusive substance concentration. * m This example shows a case where the data for )) is not included.

[0201] [Dependent variable calculation process (S2212), dependent variable calculation unit 940] The target variable calculation unit 940 calculates the estimated value of the target variable (average μ(y) of the estimated diffusive substance concentration) by providing estimation data to the learning model stored in the memory unit 920. * m Calculate )).

[0202] [Target variable correction process (S2213), target variable correction unit 2110] The objective variable correction unit 2110 calculates the estimated value of the objective variable (average μ(y) of the estimated diffusive substance concentration) calculated by the objective variable calculation unit 940. * m The gradient evaluation index grad calculated by the evaluation index calculation unit 915 is used as an evaluation index for the learning model stored in the memory unit 920 (memory process (S2208)). m Correction is performed using the following. Note that the learning model stored in the memory unit 920 (memory process (S2208)) is determined by the explanatory variable determination unit 916, and the explanatory variables (reference diffusing substance concentrations y1~y N This is a learning model that has ) as an explanatory variable.

[0203] Figure 25 shows all coordinates x * N+1 ~x * M The average μ(y) of the estimated diffusing substance concentration in the region * m_q ) and the correct data value (correct value) of the diffusion substance concentration y * m_t_q This figure shows an example of the relationship between and . As mentioned above, m is an integer from N+1 to M. Also, q is a variable representing time and is a value from 1 to Q. That is, the average μ(y) of the estimated diffusing substance concentration * m_q ), the correct data value of the diffusion substance concentration (correct value) y * m_t_q This is the value at time q.

[0204] The average μ(y) of the estimated diffusive substance concentration at time q *m_q ) and the correct data value (correct value) of the diffusion substance concentration at time q y * m_t_q The point determined by and is defined by all integers m from N+1 to M (i.e., coordinate x * N+1 ~x * M Figure 25(a) shows the results obtained and plotted for all of the above. Note that the variability evaluation index V shown above the graph in Figure 25(a) m (=0.64), gradient evaluation index grad m (=0.89) is the average value of the values ​​from m=N+1 to M. For reference, the coefficient of determination R calculated using the data used to obtain Figure 25(a) is also shown. 2 It was -0.0196.

[0205] Figure 26 shows the coordinate x * N+1 ~x * M One of the coordinates x * m' The average μ(y) of the estimated diffusing substance concentration in the region * m'_q ) and the correct data value (correct value) of the diffusion substance concentration y * m'_t_q This figure shows an example of the relationship between and . Figure 26(a) shows the relationship between the single coordinate x from Figure 25(a) * m' These are the points extracted from [the relevant section].

[0206] As shown in Figure 26(a), one coordinate x * m' Focusing on this, the average μ(y) of the estimated diffusing substance concentration * m'_q ) and the correct data value (correct value) of the diffusion substance concentration y * m'_t_q The relationship between and is generally linear. However, the gradient evaluation index grad m The value of (the slope of the graph in Figure 25(a)) corresponds to the coordinate x * m Because it differs depending on the coordinate x, as shown in Figure 25(a), all coordinates x * N+1~x * M Regarding the average μ(y) of the estimated diffusing substance concentration, * m_q ) and the correct data value (correct value) of the diffusion substance concentration y * m_t_q When the relationship between and is shown in conjunction, these relationships do not exhibit a linear relationship.

[0207] The inventors of this invention have determined the coordinate x * m The average μ(y) of the estimated diffusing substance concentration in the region * m_q ) to the said coordinate x * m Gradient evaluation index in grad m By correcting using this method, each coordinate x * N+1 ~x * M These relationships in (average μ(y) of estimated diffusive substance concentration * m_q ) and the correct data value (correct value) of the diffusion substance concentration y * m_t_q We found that this brings the relationship between and closer. Figure 25(b) shows the average μ(y) of the corrected estimated diffusive substance concentration. * m_q ) and the correct data value (correct value) of the diffusion substance concentration y * m_t_q Here is an example of the relationship between and . Here, the coordinate x * m The average μ(y) of the estimated diffusing substance concentration in the region * m_q ) to the said coordinate x * m Gradient evaluation index in grad m Dividing by all coordinates x * N+1 ~x * M By performing this operation, the average μ(y) of the estimated diffusing substance concentration can be calculated. * m_q An example of correcting ) is given.

[0208] As shown in Figure 25(b), the average μ(y) of the corrected estimated diffusive substance concentration *m_q ) and the correct data value (correct value) of the diffusion substance concentration y * m_t_q The relationship between and is that all coordinates x * N+1 ~x * M Even when the relationship between these factors is shown repeatedly, it generally remains a linear relationship, and the variability evaluation index V m (=0.999), gradient evaluation index grad m (=1.0) approaches 1 (or becomes 1).

[0209] Figure 26(b) shows one coordinate x stored in the memory unit 920. * m' Gradient evaluation index in grad m' This figure shows an example. Coordinate x * m' Gradient evaluation index in grad m' This is the slope of the regression equation 2611 for model evaluation shown in Figure 26(b). The mean μ(y) of the estimated diffusive substance concentration shown in Figure 26(a) * m'_q ) is used as the gradient evaluation index grad m' The average μ(y) of the estimated diffusive substance concentration corrected by dividing by... * m'_q ) and the correct data value (correct value) of the diffusion substance concentration y * m'_t_q The relationship between and is as shown in Figure 26(c).

[0210] [Output process (S2214), output section 950] The output unit 950 outputs the estimated value of the target variable corrected by the target variable correction unit 2110 (average μ(y) of the estimated diffusive substance concentration). * m_q Outputs information about )).

[0211] [summary] Figures 27 to 29 show the diffusive substance concentration (Figures 27(a) to 29(a)) and gradient evaluation index grad at the bottom surface 2304 of material 2300. m This is a diagram showing an example of (Figures 27(b) to 29(b)). Figure 27 shows the results of the method of this embodiment. Figure 28 shows the method of this embodiment without correction by the objective variable correction unit 2110 (the estimated value of the objective variable calculated by the objective variable calculation unit 940 (average μ(y) of the estimated diffusive substance concentration). * m_q This figure shows the results of the method using () as the solution. Figure 29 shows the explanatory variables (reference diffusing substance concentrations y1~y N This figure shows the results of a method that creates a learning model by randomly selecting ).

[0212] In Figures 27-29, the gray circles (●) represent the reference diffusing substance concentrations y1-y N Coordinates x1~x N This is shown. Note that in Figures 27 to 29, due to notation constraints, there are some places where the gray circles are not clearly visible, but in each figure (a) and (b), there are gray circles in the same position (for example, in Figure 27(a) and Figure 27(b), there are gray circles in the same position). Also, the total number of coordinates of the bottom surface 2304 of material 2300 is 1681. In any of the methods shown in Figures 27 to 29, the variation evaluation index V m The search termination condition was set to be when the result exceeds 0.999.

[0213] As shown in Figure 28, in the method of this embodiment in which no correction is made by the objective variable correction unit 2110, the gradient evaluation index grad shown in Figure 28(b) is obtained. m To ensure that the distribution is uniform and approximately 1.0, 62 coordinates are assigned, with reference diffusing substance concentrations y1~y N Coordinates x1~x N By doing so, the remaining 1619 coordinates x * N+1 ~x * M Diffusing substance concentration y * N+1 ~y * M mean μ(y * N+1 )~μ(y * M ) is used as the variability evaluation index V m We were able to calculate it so that it exceeds 0.999.

[0214] In contrast, as shown in Figure 29, the explanatory variables (reference diffusing substance concentrations y1~y) N In a method that uses a learning model created by randomly selecting ) 170 coordinates x1~x N Reference diffusing substance concentrations y1~y N If not used, the variability evaluation index V m The remaining 1511 coordinates x such that it exceeds 0.999 * N+1 ~x * M Diffusing substance concentration y * N+1 ~y * M mean μ(y * N+1 )~μ(y * M It was not possible to calculate ). Therefore, it can be seen that a method without correction by the target variable correction unit 2110 may be used for the method of this embodiment (i.e., the methods of the second to fourth embodiments may be used).

[0215] However, as shown in Figure 27, the method of this embodiment uses only 10 coordinates x1~x N Reference diffusing substance concentrations y1~y N By using this, the remaining 1671 coordinates x * N+1 ~x * M Diffusing substance concentration y * N+1 ~y * M mean μ(y * N+1 )~μ(y * M ) is used as the variability evaluation index V m We were able to calculate a value greater than 0.999. Therefore, it is clear that performing the correction by the target variable correction unit 2110 is more preferable (i.e., the method of this embodiment is more preferable).

[0216] As described above, in this embodiment, the processing apparatus 2100 estimates the target variable (average μ(y) of the estimated diffusing substance concentration). * m The gradient evaluation index grad was calculated using )) as an evaluation metric for the learning model. m This is used for correction. Therefore, the accuracy of the target variable can be improved with a learning model that uses fewer explanatory variables.

[0217] Method of this embodiment (Estimated value of the target variable (Average μ(y) of estimated diffusive substance concentration) * m The correction of (y)) is highly effective when applied to a learning model that has a nonlinear kernel function (see Figures 25(a) and 25(b)). However, the method of this embodiment may also be applied even if the learning model does not have a nonlinear kernel function. For example, the method of this embodiment may be applied to a nonlinear model that does not have a nonlinear kernel function. Note that the method of this embodiment may also be applied to a linear model, but whether or not the correction is applied, the estimated value of the target variable (average μ(y) of the estimated diffusive substance concentration) * m There will be no significant difference.

[0218] (Other embodiments) Furthermore, the embodiments of the present invention described above can be realized by a computer executing a program. A computer-readable recording medium on which the program is stored, and a computer program product such as the program itself, can also be applied as embodiments of the present invention. Examples of recording media include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, etc. Moreover, embodiments of the present invention may be realized by a PLC (Programmable Logic Controller) or by dedicated hardware such as an ASIC (Application Specific Integrated Circuit). Furthermore, the embodiments of the present invention described above are merely examples of how the invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0219] Furthermore, the disclosure of the above embodiments is as follows, for example. [Disclosure 1] A processing device that performs a process including evaluating a learning model that shows the relationship between one or more explanatory variables and the target variable explained by those explanatory variables, A first training data acquisition means for acquiring multiple sets of first training data, each set consisting of a data set containing the data of the explanatory variable and the ground truth data of the target variable explained by the explanatory variable, A second learning data creation means for creating multiple sets of second learning data as second learning data for calculating a model evaluation regression equation, which is a linear regression equation showing the relationship between the ground truth value, which is the value of the ground truth data of the target variable, and the estimated value, which is the value of the target variable calculated by the learning model, using the first learning data and the learning model. A model evaluation regression equation calculation means for calculating the model evaluation regression equation using the second training data, An evaluation index calculation means for calculating an evaluation index for the learning model, It has, The second training data creation means calculates an estimated value of the target variable in the learning model by providing the data values ​​of the explanatory variables included in the first training data to the learning model, and creates a set of data as the second training data that includes the estimated value of the target variable in the learning model and the ground truth value, which is the value of the ground truth data of the target variable that is paired with the data of the explanatory variables used to calculate the estimated value of the target variable in the first training data. The aforementioned evaluation index comprises a variability evaluation index and a gradient evaluation index. The aforementioned variability evaluation index is an index that represents the variability of the second training data from the regression equation for model evaluation, The aforementioned gradient evaluation index is an index representing the gradient of the regression equation for model evaluation, wherein the processing device is an index. [Disclosure 2] The evaluation index calculation means calculates the variability evaluation index using the first and second variations with respect to the regression equation for model evaluation. The first variation is expressed using the difference between the estimated value of the target variable included in the second training data and the mean value of the correct values ​​of the target variable. The aforementioned second variation is, The difference between the estimated value of the target variable included in the second training data and the estimated value of the target variable calculated by substituting the correct value of the target variable, which is paired with the estimated value of the target variable in the second training data, into the regression equation for model evaluation, The processing apparatus according to Disclosure 1, which is expressed using one of the following: the estimated value of the dependent variable calculated by substituting the correct values ​​of the dependent variable into the regression equation for model evaluation, and the difference between the mean value of the correct values ​​of the dependent variable. [Disclosure 3] The apparatus according to disclosure 1 or 2, wherein the intercept of the regression equation for model evaluation is 0 (zero). [Disclosure 4] The processing apparatus according to any one of disclosures 1 to 3, further comprising a determination means for calculating a plurality of candidates as candidates for a learning model that shows the relationship between a portion of the explanatory variables included in the first learning data and the target variable explained by the portion of the explanatory variables, using the first learning data; calculating an evaluation index for each of the plurality of candidates for the learning model; and determining the explanatory variables in the learning model based on the evaluation index. [Disclosure 5] The aforementioned determination means is An explanatory variable selection means for selecting explanatory variables from among the explanatory variables included in the first training data, A learning model candidate calculation means calculates a learning model having the explanatory variables selected by the explanatory variable selection means as explanatory variables using the first training data, as a candidate for the learning model, The second training data creation means and The regression equation calculation means for model evaluation, The evaluation index calculation means, When the conditions for ending the search for the explanatory variables are met, the explanatory variable determination means determines the explanatory variables selected by the explanatory variable selection means as the explanatory variables in the learning model. It further possesses, The second training data creation means uses the first training data and the candidate training model to create multiple sets of second training data as second training data for calculating the regression equation for model evaluation. The evaluation index calculation means calculates an evaluation index for the candidate learning model, The processing apparatus according to disclosure 4, wherein the explanatory variable selection means repeatedly selects the explanatory variables based on an evaluation index for the candidate learning model until the search termination condition is met. [Disclosure 6] The processing apparatus according to disclosure 5, wherein the explanatory variable selection means selects the explanatory variable in the candidate from among the plurality of candidate learning models in which the gradient of the regression equation for model evaluation, represented by the gradient evaluation index, is non-negative, and the variability, represented by the variability evaluation index, is minimized. [Disclosure 7] The processing apparatus according to disclosure 5 or 6, wherein the explanatory variable selection means modifies the explanatory variables in the plurality of candidate learning models by increasing or decreasing the number of explanatory variables by one if the search termination condition is not met. [Disclosure 8] The processing apparatus according to any one of disclosures 5 to 7, wherein the explanatory variable selection means, in the first selection of the explanatory variables, selects the explanatory variables from among the multiple candidate learning models for which the number of explanatory variables is an initial value, the candidate for which the gradient of the regression equation for model evaluation, represented by the gradient evaluation index, is non-negative, and the variability represented by the variability evaluation index is minimized. [Disclosure 9] The processing apparatus according to any one of disclosures 5 to 8, wherein the explanatory variable selection means randomly selects an explanatory variable from among the explanatory variables included in the first training data in the first selection of the explanatory variable. [Disclosure 10] The processing apparatus according to any one of disclosures 5 to 9, wherein the explanatory variable selection means, in the first selection of the explanatory variable, creates a sample by resampling using the first training data, calculates regression coefficients by performing a penalty regression analysis that can reduce the regression coefficients to 0 using the sample, and selects the explanatory variable based on the calculated regression coefficients. [Disclosure 11] Estimation data acquisition means for acquiring estimation data including data of the explanatory variables determined by the determination means of the processing apparatus described in any one of disclosures 4 to 10, A target variable calculation means for calculating the value of the target variable using the data of the explanatory variables included in the estimation data and the learning model having the explanatory variables determined by the determination means of the processing device described in disclosure 1 or 2 as explanatory variables, A processing device. [Disclosure 12] The apparatus according to disclosure 11, further comprising a target variable correction means for correcting the value of the target variable calculated by the target variable calculation means using the gradient evaluation index, which is the evaluation index for the learning model. [Disclosure 13] The processing apparatus according to disclosure 12, wherein the learning model is a model created by Gaussian process regression and has a nonlinear kernel function. [Disclosure 14] The aforementioned explanatory variables include factors used in the processing process, The apparatus according to any one of disclosures 1 to 13, wherein the objective variable includes a factor obtained in the processing process by performing the processing using a factor used in the processing process. [Disclosure 15] The aforementioned treatment process is a process that treats the water to be treated using a group of microorganisms present in the water to be treated, which contains a specific substance. The aforementioned explanatory variables are factors relating to the quantity of the microbial community, The apparatus according to disclosure 14, wherein the objective variable is a factor relating to the rate of change of the amount of the specific substance. [Disclosure 16] A processing method that includes performing a process that evaluates a learning model showing the relationship between one or more explanatory variables and the target variable explained by those explanatory variables, A first training data acquisition step involves acquiring multiple sets of first training data, each set consisting of a data set containing the data of the explanatory variable and the ground truth data of the target variable explained by the explanatory variable. A second learning data creation step involves creating multiple sets of second learning data as second learning data for calculating a model evaluation regression equation, which is a linear regression equation showing the relationship between the ground truth value, which is the value of the ground truth data of the target variable, and the estimated value, which is the value of the target variable calculated by the learning model, using the first learning data and the learning model. A model evaluation regression equation calculation step, which calculates the regression equation for model evaluation using the second training data, A step of calculating evaluation metrics for the learning model, It has, The second training data creation step involves providing the data values ​​of the explanatory variables included in the first training data to the training model to calculate an estimated value of the target variable in the training model, and creating a set of data as the second training data that includes the estimated value of the target variable in the training model and the ground truth value, which is the value of the ground truth data of the target variable that is paired with the data of the explanatory variables used to calculate the estimated value of the target variable in the first training data. The aforementioned evaluation index comprises a variability evaluation index and a gradient evaluation index. The aforementioned variability evaluation index is an index that represents the variability of the second training data from the regression equation for model evaluation, A processing method wherein the gradient evaluation index is an index that represents the gradient of the regression equation for model evaluation. [Disclosure 17] A program for causing a computer to function as one of the means of the processing apparatus described in any one of disclosures 1 to 15. [Explanation of Symbols]

[0220] 20. Biological treatment equipment 20a Biological Processing Area 20b Subsidence area 21 Sponge carrier 22 Air aeration 23 Bulkhead 24. Water to be treated 25. Treated water 100 Processing Units 110 First Learning Data Acquisition Unit 120 Second Training Data Creation Section 130 Model Evaluation Regression Equation Calculation Unit 140 Evaluation Index Calculation Unit 150 Output section 411 Regression equation for model evaluation 421 Learning Models 511, 521 Regression equations for model evaluation 611 Regression equation for model evaluation 900 Processing Unit 910 Decision Section 911 Explanatory Variable Selection Section 912 Learning Model Candidate Calculation Unit 913 Second Learning Model Creation Section 914 Model Evaluation Regression Equation Calculation Unit 915 Evaluation Index Calculation Unit 916 Explanatory Variable Determination Unit 920 Storage section 930 Data acquisition unit for estimation 940 Target Variable Calculation Unit 950 Output section 1111~1113 Regression equations for model evaluation 1211~1213 Regression equations for model evaluation 1311 Regression equation for model evaluation 1411~1412 Regression equation for model evaluation 1811 Regression equation for model evaluation 1911-1913 Regression equation for model evaluation 2100 Processing Unit 2110 Target Variable Correction Unit 2300 Object 2301 Top surface 2302, 2303 Side view 2304 Bottom 2611 Regression equation for model evaluation

Claims

1. A processing device that performs a process including evaluating a learning model that shows the relationship between one or more explanatory variables and the target variable explained by those explanatory variables, A first training data acquisition means for acquiring multiple sets of first training data, each set consisting of a data set containing the data of the explanatory variable and the ground truth data of the target variable explained by the explanatory variable, A second learning data creation means for creating multiple sets of second learning data as second learning data for calculating a model evaluation regression equation, which is a linear regression equation showing the relationship between the ground truth value, which is the value of the ground truth data of the target variable, and the estimated value, which is the value of the target variable calculated by the learning model, using the first learning data and the learning model. A model evaluation regression equation calculation means for calculating the model evaluation regression equation using the second training data, An evaluation index calculation means for calculating an evaluation index for the learning model, It has, The second training data creation means calculates an estimated value of the target variable in the learning model by providing the data values ​​of the explanatory variables included in the first training data to the learning model, and creates a set of data as the second training data that includes the estimated value of the target variable in the learning model and the ground truth value, which is the value of the ground truth data of the target variable that is paired with the data of the explanatory variables used to calculate the estimated value of the target variable in the first training data. The aforementioned evaluation index comprises a variability evaluation index and a gradient evaluation index. The aforementioned variability evaluation index is an index that represents the variability of the second training data from the regression equation for model evaluation, The aforementioned gradient evaluation index is an index representing the gradient of the regression equation for model evaluation, wherein the processing device is an index.

2. The evaluation index calculation means calculates the variability evaluation index using the first and second variations with respect to the regression equation for model evaluation. The first variation is expressed using the difference between the estimated value of the target variable included in the second training data and the mean value of the correct values ​​of the target variable. The aforementioned second variation is, The difference between the estimated value of the target variable included in the second training data and the estimated value of the target variable calculated by substituting the correct value of the target variable, which is paired with the estimated value of the target variable in the second training data, into the regression equation for model evaluation, The processing apparatus according to claim 1, which is expressed using one of the following: the estimated value of the objective variable calculated by substituting the correct values ​​of the objective variable into the regression equation for model evaluation, and the difference between the mean value of the correct values ​​of the objective variable.

3. The processing apparatus according to claim 1 or 2, wherein the intercept of the regression equation for model evaluation is 0 (zero).

4. The processing apparatus according to claim 1 or 2, further comprising: a determination means for calculating a plurality of candidates as candidates for a learning model that shows the relationship between a portion of the explanatory variables included in the first learning data and the target variable explained by the portion of the explanatory variables, using the first learning data; calculating an evaluation index for each of the plurality of candidates for the learning model; and determining the explanatory variables in the learning model based on the evaluation index.

5. The aforementioned determination means is An explanatory variable selection means for selecting explanatory variables from among the explanatory variables included in the first training data, A learning model candidate calculation means calculates a learning model having the explanatory variables selected by the explanatory variable selection means as explanatory variables using the first learning data, as a candidate for the learning model, The second learning data creation means and, The regression equation calculation means for model evaluation, The evaluation index calculation means, When the conditions for ending the search for the explanatory variables are met, the explanatory variable determination means determines the explanatory variables selected by the explanatory variable selection means as the explanatory variables in the learning model. It further possesses, The second training data creation means uses the first training data and the candidate training model to create multiple sets of second training data as second training data for calculating the regression equation for model evaluation. The evaluation index calculation means calculates an evaluation index for the candidate learning model, The processing apparatus according to claim 4, wherein the explanatory variable selection means repeatedly selects the explanatory variables based on an evaluation index for the candidate learning model until the search termination condition is met.

6. The processing apparatus according to claim 5, wherein the explanatory variable selection means selects the explanatory variable in the candidate from among the plurality of candidate learning models in which the gradient of the regression equation for model evaluation, represented by the gradient evaluation index, is non-negative, and the variability, represented by the variability evaluation index, is minimized.

7. The processing apparatus according to claim 5, wherein the explanatory variable selection means modifies the explanatory variables in the plurality of candidate learning models by increasing or decreasing the number of explanatory variables by one if the search termination condition is not met.

8. The processing apparatus according to claim 5, wherein the explanatory variable selection means, in the first selection of the explanatory variables, selects the explanatory variables from among the plurality of candidate learning models for which the number of explanatory variables is an initial value, the candidate for which the gradient of the regression equation for model evaluation, represented by the gradient evaluation index, is non-negative, and the variability represented by the variability evaluation index is minimized.

9. The processing apparatus according to claim 5, wherein the explanatory variable selection means randomly selects an explanatory variable from among the explanatory variables included in the first training data in the first selection of the explanatory variable.

10. The apparatus according to claim 5, wherein the explanatory variable selection means, in the first selection of the explanatory variable, creates a sample by resampling using the first training data, calculates regression coefficients by performing a penalty regression analysis that can reduce the regression coefficients to zero using the sample, and selects the explanatory variable based on the calculated regression coefficients.

11. Estimation data acquisition means for acquiring estimation data including the data of the explanatory variables determined by the determination means, A target variable calculation means that calculates the value of the target variable using the data of the explanatory variables included in the estimation data and the learning model having the explanatory variables determined by the determination means as explanatory variables, The apparatus according to claim 4, having the following features.

12. The processing apparatus according to claim 11, further comprising a target variable correction means for correcting the value of the target variable calculated by the target variable calculation means using the gradient evaluation index, which is the evaluation index for the learning model.

13. The processing apparatus according to claim 12, wherein the learning model is a model created by Gaussian process regression and has a nonlinear kernel function.

14. The aforementioned explanatory variables include factors used in the processing process, The apparatus according to claim 1 or 2, wherein the objective variable includes a factor obtained in the processing process by performing the processing using a factor used in the processing process.

15. The aforementioned treatment process is a process that treats the water to be treated using a group of microorganisms present in the water to be treated, which contains a specific substance. The aforementioned explanatory variables are factors relating to the quantity of the microbial community, The apparatus according to claim 14, wherein the objective variable is a factor relating to the rate of change of the amount of the specific substance.

16. A processing method that includes performing a process that evaluates a learning model showing the relationship between one or more explanatory variables and the target variable explained by those explanatory variables, A first training data acquisition step involves acquiring multiple sets of first training data, each set consisting of a data set containing the data of the explanatory variable and the ground truth data of the target variable explained by the explanatory variable. A second learning data creation step involves creating multiple sets of second learning data as second learning data for calculating a model evaluation regression equation, which is a linear regression equation showing the relationship between the ground truth value, which is the value of the ground truth data of the target variable, and the estimated value, which is the value of the target variable calculated by the learning model, using the first learning data and the learning model. A model evaluation regression equation calculation step, which calculates the model evaluation regression equation using the second training data, A step of calculating evaluation metrics for the learning model, It has, The second training data creation step involves providing the data values ​​of the explanatory variables included in the first training data to the training model to calculate an estimated value of the target variable in the training model, and creating a set of data as the second training data that includes the estimated value of the target variable in the training model and the ground truth value, which is the value of the ground truth data of the target variable that is paired with the data of the explanatory variables used to calculate the estimated value of the target variable in the first training data. The aforementioned evaluation index comprises a variability evaluation index and a gradient evaluation index. The aforementioned variability evaluation index is an index that represents the variability of the second training data from the regression equation for model evaluation, A processing method wherein the gradient evaluation index is an index that represents the gradient of the regression equation for model evaluation.

17. A program for causing a computer to function as each means of the processing apparatus according to claim 1 or 2.

Images