Data evaluation device and data evaluation method

Abstract

This data evaluation device includes: a data evaluation unit that, on the basis of test data, training data, or validation data prepared for training a model for predicting information to be predicted from information different from the information to be predicted, each data including a time series of the information to be predicted and a time series of the different information, can predict, on the basis of the test data, the training data, or the validation data, the information to be predicted with regard to the test data by using the model having been trained using the training data or the validation data, or performs evaluation in a delay coordinate system; and an evaluation value determination unit that determines whether the training data or the validation data is insufficient from the evaluation result of the data evaluation unit.

Description

Data evaluation device and data evaluation method

[0001] The disclosed technology relates to a data evaluation device and a data evaluation method.

[0002] The delayed coordinate system is a vector that combines any number of continuous variables in any time series data x(0), ..., x(t), x(t+1), x(t+2), ... (Figure 13). Figure 13 shows an image in which multiple continuous data points in the original time series are applied to each axis.

[0003] In the delayed coordinate system, for example, predictable time series data affecting river water levels form a trajectory with a certain pattern (Fig. 14). Fig. 14 shows an example in which data in the delayed coordinate system draws a trajectory with a certain pattern.

[0004] By analyzing this trajectory, it is possible to identify indicators such as insufficient data or differences in the trends of the data to be predicted compared to past data.

[0005] Furthermore, by making predictions in a delayed coordinate system, it is possible to train a prediction model with less hyperparameter tuning than with conventional machine learning techniques.

[0006] Okuno, Shunya, Koji Ikeuchi, and Kazuyuki Aihara. "Practical data‐driven flood forecasting based on dynamical systems theory." Water Resources Research 57.3 (2021): e2020WR028427.

[0007] Developing a prediction system using machine learning requires human costs for development, costs for the computer that performs the learning, and time costs.

[0008] It is only after the system development is complete that it can be determined whether the accuracy of the prediction system has reached the target.

[0009] If the accuracy of the prediction system does not meet the target due to insufficient or incomplete training data, there is a problem of having to go back and rework, which incurs further costs.

[0010] The disclosed technology has been made in consideration of the above points, and aims to provide a data evaluation device and a data evaluation method that can evaluate the lack of data for training a model even before the model is trained.

[0011] A first aspect of the present disclosure is a data evaluation device including: a data evaluation unit that performs an evaluation in a delay coordinate system based on the test data, the learning data, or the validation data, which is test data, learning data, or validation data prepared for training a model for predicting information to be predicted from information different from the information to be predicted, and each data includes a time series of the information to be predicted and a time series of the different information, to determine whether the information to be predicted can be predicted for the test data using the model trained using the learning data or the validation data; and an evaluation value determination unit that determines whether the learning data or the validation data is insufficient based on the evaluation result of the data evaluation unit.

[0012] A second aspect of the present disclosure is a data evaluation method in which a computer performs an evaluation in a delay coordinate system based on test data, learning data, or validation data prepared for learning a model for predicting information to be predicted from information different from the information to be predicted, each of which includes a time series of the information to be predicted and a time series of the different information, to determine whether the prediction target information can be predicted for the test data using the model trained using the learning data or the validation data, and determines whether the learning data or the validation data is insufficient based on the evaluation result of the data evaluation unit.

[0013] The disclosed technology makes it possible to assess the lack of data for training a model.

[0014] 1 is a block diagram showing the configuration of a data evaluation system of this embodiment. FIG. 1 is a schematic block diagram of an example of a computer that functions as the data evaluation device of this embodiment. FIG. 1 is a block diagram showing the configuration of the data evaluation device of this embodiment. FIG. 2 is a block diagram showing the configuration of a user terminal of this embodiment. FIG. 3 is a flowchart showing a data evaluation processing routine of the data evaluation device of this embodiment. FIG. 4 is a flowchart showing a processing routine illustrating the flow in which the data evaluation device of the first embodiment evaluates data. FIG. 5 is a diagram for explaining data with close distances in a delay coordinate system. FIG. 6 is a flowchart showing a processing routine illustrating the flow in which the data evaluation device of the second embodiment evaluates data. FIG. 7 is a diagram for explaining data with close changes in a delay coordinate system. FIG. 8 is a flowchart showing a processing routine illustrating the flow in which the data evaluation device of the third embodiment evaluates data. FIG. 9 is a graph showing the results predicted by a model in an example. FIG. 10 is a graph showing the prediction error of a model in an example. FIG. 11 is a diagram for explaining a delay coordinate system. FIG. 12 is a diagram showing an example of a data trajectory in a delay coordinate system.

[0015] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0016] [First embodiment] <Configuration of a data evaluation system according to a first embodiment> Fig. 1 is an explanatory diagram showing an example configuration of a data evaluation system 100. In this embodiment, a data evaluation system 100 that evaluates data for training a model that predicts river water levels as prediction target information will be described. The data evaluation system 100 includes a data evaluation device 10, a user terminal 20, and an information acquisition device 30.

[0017] The data evaluation device 10 is, for example, a server computer capable of various information processing and sending and receiving information. Note that the device equivalent to the data evaluation device 10 is not limited to a server computer, and may be, for example, a personal computer. In this embodiment, the data evaluation device 10 functions as a device for evaluating data for learning a model for predicting river water levels.

[0018] The user terminal 20 is a general-purpose computer such as a personal computer. In this embodiment, the user terminal 20 functions as a device that requests the data evaluation device 10 to evaluate data for learning a model for predicting river water levels.

[0019] The information acquisition device 30 is a general-purpose computer such as a personal computer. In this embodiment, the information acquisition device 30 functions as a device that acquires data for predicting the water level of a river (for example, water level, rainfall, etc.) from a group of sensors and transmits the data to the data evaluation device 10. Note that the information acquisition device 30 may also acquire data that is thought to affect the water level, such as tidal information, in addition to the water level and rainfall, and transmit the data to the data evaluation device 10.

[0020] <Configuration of Data Evaluation Apparatus According to First Embodiment> FIG. 2 is a block diagram showing the hardware configuration of a data evaluation apparatus 10 according to this embodiment.

[0021] 2, the data evaluation device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0022] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, a data evaluation program is stored in the ROM 12 or the storage 14. The data evaluation program may be a single program, or may be a group of programs composed of multiple programs or modules.

[0023] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0024] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0025] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0026] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0027] Next, a description will be given of the functional configuration of the data evaluation device 10. Fig. 3 is a block diagram showing an example of the functional configuration of the data evaluation device 10.

[0028] As shown in FIG. 3, the data evaluation device 10 functionally comprises a preprocessing unit 101, a database 102, a data evaluation unit 103, an evaluation value determination unit 104, a learning unit 105, a prediction result output unit 106, and a prediction result evaluation unit 107.

[0029] The preprocessing unit 101 performs preprocessing on information received from the user terminal 20 or the information acquisition device 30, and stores the information in the database 102. The preprocessing includes, for example, shaping the data, removing noise from the data, labeling the training data, validation data, and test data, and specifying the information to be predicted.

[0030] Here, the test data, learning data, and validation data are data for learning a model to predict information to be predicted (e.g., water level) from information different from the information to be predicted (e.g., rainfall), and each data includes a time series of the information to be predicted and a time series of the different information.

[0031] The database 102 stores the data related to rivers after the data formatting. An example of the format of the data stored in the database 102 is shown in Table 1.

[0032]

[0033] The data evaluation unit 103 evaluates the training data and validation data in response to an evaluation request from a user.

[0034] Specifically, the data evaluation unit 103 evaluates, based on the test data, the learning data, and the validation data, whether the prediction target information can be predicted for the test data using a model trained using the learning data and the validation data in a delay coordinate system. The delay coordinate system is a coordinate system that expresses, as vectors, a time series of the prediction target information (data for a predetermined time) and a time series of information different from the prediction target information (data for a predetermined time).

[0035] More specifically, the data evaluation unit 103 performs evaluation based on the distance between the test data and the learning data, and the distance between the test data and the validation data in the delay coordinate system.

[0036] The evaluation value determination unit 104 determines whether there is a shortage of learning data and validation data from the evaluation result of the data evaluation unit 103. Specifically, the evaluation value determination unit 104 compares the result of the data evaluation unit 103 with a standard and outputs an evaluation such as "pass" if the result conforms, or "fail" if the result does not conform.

[0037] The learning unit 105 uses the learning data and validation data in the database 102 to learn a model for predicting the prediction target information. Learning is performed using data in the database 102 under predetermined settings. As a learning method, any machine learning technique such as an artificial neural network (ANN) may be used.

[0038] The prediction result output unit 106 outputs the prediction result using the test data in the database 102. Specifically, the prediction result output unit 106 uses the model trained by the training unit 105 and the test data in the database 102 to output the numerical value of the prediction result.

[0039] The prediction result evaluation unit 107 compares the prediction result of the test data with the true value and calculates evaluation indices such as error. By comparing the numerical value of the prediction result with the true predicted value, the difference and RMSE (Root Mean Squared Error) are calculated, and the prediction result of the trained model is evaluated.

[0040] <Configuration of User Terminal According to This Embodiment> FIG. 2 is a block diagram showing the hardware configuration of the user terminal 20 according to this embodiment.

[0041] 2, the user terminal 20, like the data evaluation device 10, has a CPU 11, a ROM 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0042] The ROM 12 or the storage 14 stores a program for requesting data evaluation from the data evaluation device 10. This program may be a single program, or may be a group of programs made up of multiple programs or modules.

[0043] The input unit 15 receives input of data possessed by the user, and information such as data on prediction target information, test data, learning data, and validation data settings.

[0044] The display unit 16 displays the data evaluation results.

[0045] Next, a description will be given of the functional configuration of the user terminal 20. Fig. 4 is a block diagram showing an example of the functional configuration of the user terminal 20.

[0046] As shown in FIG. 4, the user terminal 20 functionally comprises a data input unit 201, an evaluation request unit 202, and an evaluation result display unit 203.

[0047] The data input unit 201 transmits information such as input of data relating to rivers held by the user, setting of data for prediction target information, setting of test data, learning data, and validation data to the data evaluation device 10.

[0048] The evaluation request unit 202 transmits a request for evaluation of data from a user to the data evaluation device 10 .

[0049] The evaluation result display unit 203 acquires the evaluation result determined together with the evaluation value obtained by the data evaluation device 10 and displays it on the display unit 16 .

[0050] <Operation of Data Evaluation Device According to This Embodiment> Next, the operation of the data evaluation system 100 will be described.

[0051] First, the information acquisition device 30 acquires data for predicting the water level of the river (for example, water level, rainfall, etc.) from the sensor group and transmits the data to the data evaluation device 10 .

[0052] The user terminal 20 accepts information such as input of data held by the user (e.g., water level, rainfall, etc.), setting of data for the information to be predicted, setting of test data, learning data, and validation data, and transmits it to the data evaluation device 10.

[0053] Furthermore, when the user terminal 20 receives a request for evaluation of data, it transmits the request for evaluation of data to the data evaluation device 10 .

[0054] 5 is a flowchart showing the flow of data evaluation by the data evaluation device 10. The data evaluation process is performed by the CPU 11 reading out a data evaluation program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. It is assumed that data for predicting the water level of a river, transmitted from the information acquisition device 30, has been input to the data evaluation device 10. It is also assumed that data transmitted from the user terminal 20 and a request for data evaluation have been input to the data evaluation device 10. The data evaluation process is an example of a data evaluation method.

[0055] First, in step S100, the CPU 11 functions as a preprocessing unit 101 to perform preprocessing on information received from the user terminal 20 and the information acquisition device 30, and stores the information in the database 102. For example, the CPU 11 aggregates information from the information acquisition device 30 and the user terminal 20, formats the information into a form that can be input into the database 102, and stores the information in the database 102.

[0056] In step S102, the CPU 11, as the data evaluation unit 103, evaluates the training data and validation data in response to an evaluation request from the user.

[0057] In step S104, the CPU 11, as the evaluation value determination unit 104, determines whether there is a shortage of learning data and validation data from the evaluation result of the data evaluation unit 103. For example, the evaluation value obtained by the data evaluation unit 103 is determined to be acceptable or unacceptable.

[0058] In step S106, the CPU 11 functions as the evaluation value determination unit 104 to transmit the evaluation value obtained by the data evaluation unit 103 and the determination result of step S104 to the user terminal 20, and then ends the data evaluation process.

[0059] Then, the learning unit 105 of the data evaluation device 10 uses the information in the database 102 to learn a model for predicting the prediction target information by any machine learning method.

[0060] The prediction result output unit 106 outputs the prediction of the model trained by the training unit 105 for the test data in the database 102 .

[0061] The prediction result evaluation unit 107 evaluates the predicted result using RMSE or the like.

[0062] The above step S102 is realized by the processing routine shown in FIG.

[0063] In step S110, the CPU 11 sets an arbitrary time delay width τ and a time interval Δt, and sets a delay coordinate system as the data evaluation unit 103. An arbitrary point at time t on the delay coordinate system is a vector expressed by the following equation.

[0064] However, L ｔ is the information to be predicted, such as the water level at the prediction point at time t, and P ｔ is information different from the information to be predicted, such as the amount of rain at time t.

[0065] In step S112, the CPU 11, functioning as the data evaluation unit 103, extracts all test data X from the database 102. Ｔｅｓｔ and all the training data and validation data X Ｌｅａｒｎ and converts it into data that is a vector in the delayed coordinate system. When converting it into data in the delayed coordinate system, data acquired from the database 102, such as the amount of rise in water level, may be further processed.

[0066] In step S114, the CPU 11, as the data evaluation unit 103, calculates a point x of the test data in the delay coordinate system. ｔ ∈x Ｔｅｓｔ For each of the training data and validation data x ｌ ∈x Ｌｅａｒｎ Distance d from ｔ，ｌ is calculated according to the following formula:

[0067] In step S116, the CPU 11, as the data evaluation unit 103, calculates one point x of the test data. ｔ ∈x Ｔｅｓｔ For each of the distances d ｔ，ｌ The number of training data and validation data that satisfy the following conditions is calculated as the evaluation value.

[0068] where ε is a preset neighborhood distance criterion (see FIG. 7). In FIG. 7, a point x of the test data is ｔ ∈x Ｔｅｓｔ 10 shows an example in which the number of training data and validation data that are close to each other is three.

[0069] By calculating the number of data items that are close in distance as an evaluation value, it is possible to determine whether or not there is a shortage of data from the perspective of "sufficiency of the time amount of data."

[0070] As described above, the data evaluation device according to this embodiment uses a model trained using training data and validation data to evaluate whether target information for prediction can be predicted for test data in a delayed coordinate system, and determines from the evaluation result whether there is a shortage of training data or validation data. This makes it possible to evaluate whether there is a shortage of data for training the model.

[0071] Furthermore, by detecting insufficient training data and validation data in advance, there will be no need to go back and forth due to insufficient accuracy after the development of a system to predict river water levels, reducing additional costs.

[0072] Furthermore, in the development of a real-world prediction system using machine learning, a model is trained using training data and validation data collected from observation data. However, in addition to the time required to collect the training dataset, training the model requires significant costs in terms of both financial and time. Whether the prediction system has achieved sufficient accuracy is confirmed after the model has been trained. However, if the accuracy is found to be insufficient due to deficiencies or insufficiencies in the training data and validation data, new training data and validation data must be collected and the model must be trained, resulting in additional costs. The data evaluation system according to this embodiment evaluates, in a delayed coordinate system, whether the information to be predicted among the information stored in the database can be predicted from other information, and determines whether there is a shortage of data stored in the database. As a result, before training the model, it is possible to evaluate whether the training data and validation data stored in the database are sufficient for learning the information to be predicted, and to add or modify data as necessary. Furthermore, by training the model using training data and validation data that meet data evaluation standards, it is possible to efficiently build a highly accurate prediction model.

[0073] If the amount of training data and validation data is small, the evaluation value may be calculated by the following method. That is, in step S114, the data evaluation unit 103 calculates a point x of the test data in the delayed coordinate system.ｔ ∈x Ｔｅｓｔ For each of the training data and validation data x ｔ’ ∈x Ｌｅａｒｎ Distance d from ｔ，ｔ’ is calculated according to the following formula:

[0074] In step S116, the CPU 11, as the data evaluation unit 103, evaluates one point x of the test data. ｔ ∈x Ｔｅｓｔ For each of the distances d ｔ，ｔ’ Let t be the set of time indexes t that satisfy the following conditions: ｉｎｓｉｄｅ Let's say.

[0075] Then, the CPU 11 functions as the data evaluation unit 103 and calculates the evaluation value according to the following formula.

[0076] However, N is t ｉｎｓｉｄｅ is the number of time indices included in

[0077] Second Embodiment Next, a data evaluation system according to a second embodiment will be described. The data evaluation system according to the second embodiment has the same configuration as that of the first embodiment, and therefore the same reference numerals are used and the description thereof will be omitted.

[0078] The second embodiment differs from the first embodiment in that the data evaluation unit performs evaluation based on a comparison between the time changes of the test data and the time changes of the learning data in the delay coordinate system, and a comparison between the time changes of the test data and the time changes of the validation data.

[0079] In the second embodiment, step S102 of the data evaluation process is implemented by a processing routine shown in Fig. 8. Note that the same processes as those in the first embodiment are denoted by the same reference numerals and detailed description thereof will be omitted.

[0080] In step S110, the CPU 11, functioning as the data evaluation unit 103, sets an arbitrary time delay width τ and a time interval Δt, and sets a delay coordinate system.

[0081] In step S112, the CPU 11, functioning as the data evaluation unit 103, extracts all test data X from the database 102. Ｔｅｓｔ and all the training data and validation data X Ｌｅａｒｎ and convert it into data that is a vector in the delayed coordinate system.

[0082] In step S114, the CPU 11, as the data evaluation unit 103, calculates a point x of the test data in the delay coordinate system. ｔ ∈x Ｔｅｓｔ For each of the training data and validation data x ｔ’ ∈x Ｌｅａｒｎ For the distance d ｔ，ｔ’ Calculate.

[0083] In step S200, the CPU 11, as the data evaluation unit 103, evaluates one point x of the test data. ｔ ∈x Ｔｅｓｔ For each of the above, the distance d between the test data and the training data and the validation data is ｔ，ｔ’ The subset X of the training data and validation data that satisfies the following condition: ｉｎｓｉｄｅ ⊂X Ｌｅａｒｎ Ask for.

[0084] Then, the CPU 11, as the data evaluation unit 103, calculates the subset X ｉｎｓｉｄｅ At some time t, a point x is included in ｔ ∈x ｉｎｓｉｄｅ and the next point x ｔ＋Δｔ Therefore, the change amount v ｔ Calculate (Figure 9).

[0085] This V ｔ The entire set is V ｔ FIG. 9 shows an image of a method for calculating v.

[0086] Then, the CPU 11, as the data evaluation unit 103, calculates the calculated V ｔ Then, the average value of the amount of change ^v is calculated according to the following formula (FIG. 9).

[0087] Here, w(v, z) is an arbitrary weighting function that satisfies the following equation:

[0088] Note that z is a variable for reflecting information not included in v. Figure 9 shows an image of a method for calculating ^v.

[0089] In step S202, the CPU 11, as the data evaluation unit 103, evaluates one point x of the test data. ｔ ∈x Ｔｅｓｔ For each of the points x ｔ＋Δｔ Then, the amount of change is calculated according to the following formula.

[0090] In step S204, the CPU 11 functions as the data evaluation unit 103 to compare ^v with v for each time t and calculate an evaluation value. The following two evaluation values ​​are calculated here.

[0091] The other configuration of the data evaluation system is the same as that of the first embodiment, and therefore description thereof will be omitted.

[0092] Third Embodiment Next, a data evaluation system according to a third embodiment will be described. The data evaluation system according to the third embodiment has the same configuration as that of the first embodiment, and therefore the same reference numerals are used and the description thereof will be omitted.

[0093] The third embodiment differs from the first embodiment in that the data evaluation unit performs evaluation based on the prediction accuracy when predicting test data from the time changes of training data and validation in the delay coordinate system.

[0094] In the third embodiment, step S102 of the data evaluation process is implemented by a processing routine shown in Fig. 10. Note that the same processes as those in the first and second embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0095] In step S110, the CPU 11, functioning as the data evaluation unit 103, sets an arbitrary time delay width τ and a time interval Δt, and sets a delay coordinate system.

[0096] In step S112, the CPU 11, functioning as the data evaluation unit 103, extracts all test data X from the database 102. Ｔｅｓｔ and all the training data and validation data X Ｌｅａｒｎ and convert it into data that is a vector in the delayed coordinate system.

[0097] In step S114, the CPU 11, as the data evaluation unit 103, calculates a point x of the test data in the delay coordinate system. ｔ ∈x Ｔｅｓｔ For the training data and validation data x ｔ’ ∈x Ｌｅａｒｎ For the distance d ｔ，ｔ’ Calculate.

[0098] In step S200, the CPU 11, as the data evaluation unit 103, evaluates one point x of the test data. ｔ ∈x Ｔｅｓｔ , the distance d between the test data and the training and validation data ｔ，ｔ’ The subset X of the training data and validation data that satisfies the following condition: ｉｎｓｉｄｅ ⊂X Ｌｅａｒｎ Ask for.

[0099] Then, the CPU 11, as the data evaluation unit 103, calculates the subset X ｉｎｓｉｄｅ At some time t, a point x is included in ｔ ∈x ｉｎｓｉｄｅ and the next point x ｔ＋Δｔ Therefore, the change amount v ｔ Calculate (Figure 9).

[0100] This V ｔ The entire set is V ｔ FIG. 9 shows an image of a method for calculating v.

[0101] Then, the CPU 11, as the data evaluation unit 103, calculates the calculated V ｔ Then, the average value of the amount of change ^v is calculated according to the following formula (FIG. 9).

[0102] Here, w(v, z) is an arbitrary weighting function that satisfies the following equation:

[0103] Note that z is a variable for reflecting information not included in v. Figure 9 shows an image of a method for calculating ^v.

[0104] In step S302, the CPU 11, as the data evaluation unit 103, evaluates one point x of the test data. ｔ ∈x Ｔｅｓｔ Using the change ^v calculated in step S200, the information x Δt ahead is calculated according to the following formula: ｔ＋Δｔ Predict.

[0105] In step S304, the CPU 11 determines whether or not the iteration is to be completed. If the predicted value for N×Δt time ahead is obtained in step S302, it is determined that the iteration is to be completed, and the process proceeds to step S306. If not, the process returns to step S114, and the processes of steps S114 to S302 are repeated, and the information x for the next time is further calculated. ｔ＋Δｔ Predict.

[0106] In step S306, the CPU 11, as the data evaluation unit 103, evaluates one point x of the test data. ｔ ∈x Ｔｅｓｔ For each of the above, the information x predicted in step S302 is ｔ＋Δｔ The error between the prediction target information included in the metric and the true prediction target information obtained from the test data is calculated using RMSE or the like, and used as an evaluation value.

[0107] The other configuration of the data evaluation system is the same as that of the first embodiment, and therefore description thereof will be omitted.

[0108] [Fourth Embodiment] Next, a data evaluation system according to a fourth embodiment will be described. The data evaluation system according to the fourth embodiment has the same configuration as that of the first embodiment, and therefore the same reference numerals are used and the description thereof will be omitted.

[0109] The fourth embodiment differs from the first embodiment in that data is evaluated even during the period when predictions are being made using a trained model.

[0110] The data evaluation device 10 according to the fourth embodiment learns a model using the learning unit 105, and performs the processing of the preprocessing unit 101, the data evaluation unit 103, and the evaluation value determination unit 104 even during the period when the prediction result output unit 106 is predicting the information to be predicted.

[0111] As in the third embodiment, the data evaluation unit 103 performs evaluation based on the prediction accuracy when predicting test data from the time changes of the training data and validation in the delay coordinate system.

[0112] This allows the user to check the evaluation results of the data acquired in real time from the user terminal 20. If the evaluation result is displayed as "unacceptable," the user can know that an unexpected situation may have occurred, such as a river being blocked by a debris flow, and that the reliability of the prediction results has decreased.

[0113] Furthermore, the prediction in the delay coordinate system performed by the data evaluation unit 103 can be used as an actual prediction result.

[0114] <Example> An example of data evaluation, model learning, and data prediction according to the above embodiment will be described.

[0115] Figure 11 shows the prediction results of a model using data with a rating of "Fail" and a model using data with a rating of "Acceptable." Figure 11 shows that the prediction results of the model using data with a rating of "Fail" are significantly different from the true value.

[0116] Figure 12 shows the error in the prediction results of a model using data with a rating of "Fail" and the error in the prediction results of a model using data with a rating of "Acceptable." Figure 12 shows that the prediction results of the model using data with a rating of "Fail" deviate from the true value, resulting in a larger error.

[0117] As shown above, when a model is trained using data with a rating of "Fail," the accuracy is low, and when a model is trained using data with a rating of "Acceptable," the accuracy is high. In other words, the data rating (pass, fail, etc.) is linked to the predictive performance.

[0118] <Modifications> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

[0119] For example, various processes executed by the CPU after reading software (programs) in the above embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors having circuit configurations specifically designed to execute specific processes. Furthermore, the data evaluation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0120] In addition, in each of the above embodiments, the data evaluation program is described as being pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0121] The following additional notes are provided regarding the above-described embodiments.

[0122] (Supplementary Item 1) A data evaluation device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: perform an evaluation in a delay coordinate system based on the test data, the learning data, or the validation data, which is test data, learning data, or validation data prepared for learning a model for predicting information to be predicted from information different from the information to be predicted, and each data includes a time series of the information to be predicted and a time series of the different information, to determine whether the information to be predicted can be predicted for the test data using the model trained using the learning data or the validation data; and determine whether the learning data or the validation data is insufficient from a result of the evaluation.

[0123] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute a data evaluation process, wherein the data evaluation process comprises: performing an evaluation in a delay coordinate system based on the test data, the learning data, or the validation data, which is test data, learning data, or validation data prepared for learning a model for predicting information to be predicted from information different from the information to be predicted, and each data includes a time series of the information to be predicted and a time series of the different information, to determine whether the information to be predicted can be predicted for the test data using the model trained using the learning data or the validation data; and determining whether the learning data or the validation data is insufficient from the results of the evaluation.

[0124] REFERENCE SIGNS LIST 10 Data evaluation device 11 CPU 14 Storage 15 Input unit 16 Display unit 20 User terminal 30 Information acquisition device 100 Data evaluation system 101 Preprocessing unit 102 Database 103 Data evaluation unit 104 Evaluation value determination unit

Claims

1. A data evaluation device including: a data evaluation unit that evaluates, in a delay coordinate system, whether the prediction target information can be predicted for test data, learning data, or validation data prepared for learning a model for predicting the prediction target information from information different from the prediction target information, each data including a time series of the prediction target information and a time series of the different information, by using the model learned using the learning data or the validation data based on the test data, the learning data, or the validation data; and an evaluation value determination unit that determines the presence or absence of insufficiency of the learning data or the validation data from the evaluation result of the data evaluation unit.

2. The data evaluation device according to claim 1, wherein the data evaluation unit performs the evaluation based on a distance between the test data and the learning data or the validation data in the delay coordinate system.

3. The data evaluation device according to claim 1, wherein the data evaluation unit performs the evaluation based on a comparison between a time change of the test data and a time change of the learning data or the validation data in the delay coordinate system.

4. A data evaluation method, wherein a computer evaluates, in a delay coordinate system, whether the prediction target information can be predicted for test data, learning data, or validation data prepared for learning a model for predicting the prediction target information from information different from the prediction target information, each data including a time series of the prediction target information and a time series of the different information, by using the model learned using the learning data or the validation data based on the test data, the learning data, or the validation data, and determines the presence or absence of insufficiency of the learning data or the validation data from the result of the evaluation.

Images