Data processing method and device

Abstract

To provide a data processing method and device which can suppress narrowing of a predictable numerical range, while improving prediction accuracy.SOLUTION: A data processing method includes: an outlier detection step of detecting an outlier from a plurality of data included in training data 31; a predicted value calculation step of creating a first regression model 36 using first training data 35 excluding the outlier from the training data 31 as teaching data, and determining a first predicted value which is a predicted value of an objective variable corresponding to a value of an explanatory variable of each of the excluded outliers, using the first regression model 36; and a data substitution step of substituting the excluded outlier with a value based on the first predicted value, to create substituted outlier training data 38 from the training data 31.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a data processing method and device for machine learning. [Background technology]

[0002] There are known methods for making various predictions using machine learning. For example, when predicting the physical properties of a material with an unknown composition, machine learning is performed using data already obtained through test manufacturing or the like as training data (teacher data), and the correlation between the composition of the material and its physical properties is learned, and predictions are made using the regression model obtained as a result of the learning.

[0003] Here, if the training data contains outliers, which are erroneous data or data with large errors, the prediction accuracy of the regression model obtained using the training data will decrease. For this reason, prior to machine learning, outliers are removed from the training data (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-33544 Summary of the Invention [Problem to be solved by the invention]

[0005] However, removing outliers from the training data reduces the number of data points in the training data, which poses a problem of narrowing the range of values ​​that can be accurately predicted by a regression model created using the training model.

[0006] Therefore, an object of the present invention is to provide a data processing method that can improve prediction accuracy while preventing the range of predictable values ​​from narrowing. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention provides a data processing method for processing training data including a plurality of data items each consisting of an explanatory variable and a dependent variable, the data processing method including: an outlier detection step of detecting outliers from the training data; a predicted value calculation step of creating first training data from the training data by excluding the outliers detected in the outlier detection step, creating a first regression model using the first training data as training data, and using the first regression model to determine a first predicted value, which is a predicted value of the dependent variable corresponding to the value of the explanatory variable of the excluded outlier; and a data replacement step of replacing the excluded outliers from the training data with a value based on the first predicted value, to create outlier-replaced training data.

[0008] In order to solve the above-mentioned problems, the present invention provides a data processing device that processes training data including a plurality of data items each consisting of an explanatory variable and a dependent variable, the data processing device comprising: an outlier detection processing unit that detects outliers from the training data; a predicted value calculation processing unit that creates first training data from the training data by excluding the outliers detected by the outlier detection processing unit, creates a first regression model using the first training data as training data, and calculates, using the first regression model, a first predicted value that is a predicted value of the dependent variable corresponding to the value of the explanatory variable for the excluded outlier; and a data replacement processing unit that replaces the excluded outliers from the training data with a value based on the first predicted value to create outlier-replaced training data. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a data processing method and device that can improve prediction accuracy while preventing the range of predictable values ​​from narrowing. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic configuration diagram of a data processing device according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram illustrating an example of learning data. [Figure 3] 10(a) is a diagram showing an example of the calculation results of the coefficient of variation, and FIG. 10(b) is a diagram showing the calculation results of FIG. 10(a) as a histogram. [Figure 4] FIG. 10 is an explanatory diagram showing training data used in the outlier determination process. [Figure 5] FIG. 4 is a diagram showing the calculation results of the error rate of each of the outlier candidate data obtained in FIG. [Figure 6] FIG. 2 is an explanatory diagram illustrating first learning data. [Figure 7] FIG. 1A is a flowchart of a data processing method according to an embodiment of the present invention, and FIG. 1B is a flowchart of an outlier detection process thereof. [Figure 8] FIG. 10 is a flow diagram of a data classification process. [Figure 9] FIG. 10 is a flowchart of an outlier determination process. [Figure 10] FIG. 10 is a flowchart of a predicted value calculation process. [Figure 11] FIG. 10 is a flow diagram of a data replacement process. [Figure 12] FIG. 10 is a flow diagram of a prediction process. [Figure 13] FIG. 10 is an explanatory diagram illustrating first learning data in a modified example of the present invention. [Figure 14] FIG. 10 is a flowchart of a predicted value calculation process according to a modified example of the present invention. [Figure 15] FIG. 10 is a graph showing the calculation results of MAPE for the examples and comparative examples. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Embodiment Mode] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0012] 1 is a schematic diagram of a data processing device 1 according to this embodiment. The data processing device 1 has a function of detecting outliers from training data 31 used in machine learning and replacing the values ​​of the response variables of the detected outliers with predicted values. The data processing device 1 also has a function of making predictions using a regression model 39 created using training data obtained by replacing the outliers with predicted values ​​(outlier-replaced training data 38, described below). Note that an outlier refers to a value that is significantly different from other data due to, for example, a measurement error, a human error such as an incorrect reading of an instrument or an input error, or the influence of noise.

[0013] The data processing device 1 has a control unit 2 and a storage unit 3. The data processing device 1 is, for example, a computer such as a personal computer or a server device, and includes a processing element such as a CPU, memories such as RAM and ROM, a storage device such as a hard disk, and a communication interface which is a communication device such as a LAN card.

[0014] The control unit 2 has a data acquisition processing unit 21, an outlier detection processing unit 22, a predicted value calculation processing unit 23, a data replacement processing unit 24, a prediction processing unit 25, and a prediction result presentation processing unit 26. Details of each unit will be described later. The storage unit 3 is realized by a predetermined storage area of ​​a memory or a storage device.

[0015] The data processing device 1 also has a display 4 and an input device 5. The display 4 is, for example, a liquid crystal display, and the input device 5 is, for example, a keyboard or a mouse. The display 4 may be configured as a touch panel, and may also serve as the input device 5. The display 4 and the input device 5 may also be configured separately from the data processing device 1 and configured to be able to communicate with the data processing device 1 via wireless communication or the like. In this case, the display 4 or the input device 5 may be configured as a mobile terminal such as a tablet or a smartphone.

[0016] (Data acquisition processing unit 21) The data acquisition processing unit 21 performs a data acquisition process to acquire learning data 31 from an external device. In the data acquisition process, data is acquired via a network from, for example, a server device that stores data on manufacturing results, and the acquired data is stored in the storage unit 3 as learning data 31. Note that the learning data 31 may be input to the data processing device 1 via a medium such as a USB memory, and the method for acquiring the learning data 31 is not particularly limited.

[0017] (Training data 31) Here, the training data 31 will be explained. FIG. 2 is a diagram showing an example of the training data 31. The training data 31 is a database used as training data during machine learning, and includes multiple pieces of data consisting of explanatory variables and response variables used in machine learning. The example in FIG. 2 shows a case in which the blend amounts of materials such as polymers and fillers are used as explanatory variables, and the physical properties (here, tensile strength) of a composite material manufactured using these materials are used as the response variable. By performing machine learning using this training data 31 and creating a regression model that shows the correlation between the explanatory variables (the blend amounts of each material) and the response variable (physical properties), it becomes possible to predict the physical properties that will be obtained when a composite material is manufactured with an unknown blend of materials.

[0018] In this embodiment, each data item included in the training data 31 includes multiple values ​​of the objective variable. Here, each data item includes five objective variable values ​​(tensile strength values ​​in the illustrated example). These objective variable values ​​are obtained, for example, by manufacturing a composite material with the same composition, forming multiple samples (here, five samples No. 1 to No. 5), and measuring the physical properties such as tensile strength of each sample. Here, a case will be described in which the training data 31 initially includes 1,316 data items.

[0019] (Outlier detection processing unit 22) The outlier detection processing unit 22 performs an outlier detection process to detect outliers from a plurality of data included in the learning data 31. The outlier detection process corresponds to an outlier detection step of the present invention. The outlier detection processing unit 22 has a data classification processing unit 221, an outlier determination processing unit 222, and a detection result presentation processing unit 223. Note that the specific processing content of the outlier detection process described below is merely an example, and outliers may be detected by other methods. In other words, the specific method for detecting outliers can be selected as appropriate and is not limited to the method described below.

[0020] (Data classification processing unit 221) The data classification processing unit 221 performs a data classification process to pick out data that are candidates for outliers from the learning data 31. The data classification process corresponds to the data classification step of the present invention.

[0021] In the data classification process, first, the coefficient of variation of the value of the objective variable is calculated for each piece of data included in the learning data 31. The coefficient of variation can be calculated by the following formula (1). (coefficient of variation) = {(standard deviation) / (mean value)} × 100 (1) Then, among the data included in the learning data 31, data having the calculated coefficient of variation greater than a predetermined reference value is classified as outlier candidate data 32, and the remaining data is classified as normal data 33 and stored in the storage unit 3. A large coefficient of variation means that the value of the objective variable varies greatly, and it is considered that there is a high possibility that an outlier is included. Note that here, the outlier candidate data 32 and the normal data 33 are stored in the storage unit 3 as data separate from the learning data 31. However, this is not limiting, and for example, the outlier candidate data 32 and the normal data 33 may be distinguished from each other by attaching a flag or marker to each data item in the learning data 31. In other words, part of the learning data 31 may be classified as outlier candidate data 32 or normal data 33.

[0022] The reference value of the coefficient of variation for determining outlier candidates may be set as appropriate. For example, this reference value can be set taking into consideration the target objective variable, the overall variability of the data, etc., and can be the mean value, median, mode, mean value ±σ (σ is standard deviation), mean value ±2σ, mean value ±3σ, etc., of the coefficients of variation of all data included in the training data 31. Here, the coefficient of variation of representative formulation data, taking into consideration manufacturing performance, etc., among the data included in the training data 31, is used as the reference value.

[0023] FIG. 3(a) is a diagram showing an example of the calculation results of the coefficient of variation. In FIG. 3(a), the coefficient of variation was calculated for 1,316 data points and plotted in ascending order. In the illustrated example, the standard value was set to 17.7 using the coefficient of variation for a representative composition. As shown in FIG. 3(a), in the illustrated example, there were 87 pieces of data that exceeded the standard value. In the data classification process, these 87 pieces of data are designated as outlier candidate data 32, and the remaining 1,229 pieces of data are designated as normal data 33.

[0024] Figure 3(b) is a histogram showing the calculation results of the coefficient of variation in Figure 3(a). Figure 3(b) also shows the mean, median, mean + σ, mean + 2σ, and mean + 3σ values ​​of the coefficient of variation of all data included in the training data 31. As shown in Figure 3(b), the value of 17.7 used as the reference value in this study was greater than mean + σ and less than mean + 2σ.

[0025] (Outlier determination processing unit 222) The outlier determination processing unit 222 performs an outlier determination process to determine whether each data included in the outlier candidate data 32 picked up in the data classification process is an outlier. The outlier determination process corresponds to the outlier determination step of the present invention.

[0026] As shown in FIG. 4, in this embodiment, normal data 33 (1229 pieces of data) is used as training data in the outlier determination process to create a second regression model (regression model for outlier determination), which is a regression model that indicates the correlation between explanatory variables and dependent variables. In this embodiment, each data contains multiple dependent variable values, but here, the median of the multiple dependent variable values ​​is used for learning. Then, for each piece of outlier candidate data 32 (87 pieces of data), a second predicted value, which is a predicted value, is calculated using the created second regression model, and the error rate between the obtained second predicted value and the actual dependent variable value (actual value) is calculated. In this embodiment, each data contains multiple dependent variable values, but here, the median of the multiple dependent variable values ​​is used as the actual value. The error rate is calculated using the following formula: (Error rate) = 100 × {(Predicted value) - (Actual value)} / (Predicted value) The outlier determination processing unit 222 determines whether each data item in the outlier candidate data 32 is an outlier based on the error rate calculated by the calculation. In this embodiment, when the absolute value of the calculated error rate is equal to or greater than a preset threshold value, the data item is determined to be an outlier.

[0027] FIG. 5 is a diagram showing the calculation results of the error rate for each of the 87 pieces of outlier candidate data 32 obtained in FIG. 3(a). In the illustrated example, the threshold is set to 20%, but the threshold can be set as appropriate. In this case, if the calculated error rate is +20% or more or -20% or less, the data is determined to be an outlier. In the example of FIG. 5, 33 pieces of data out of the 87 pieces of outlier candidate data 32 are determined to be outliers. Data determined to be an outlier is stored in the storage unit 3 as outlier data 34.

[0028] (Detection result presentation processing unit 223) The detection result presentation processing unit 223 performs a detection result presentation processing for presenting the determination result of the outlier determination processing, i.e., the detection result of the outlier. In the detection result presentation processing, the data detected as an outlier (outlier data 34) is presented to the user by, for example, displaying the data detected as an outlier on the display device 4. Note that the detection result presentation processing unit 223 is not essential and can be omitted.

[0029] (Prediction value calculation processing unit 23) The predicted value calculation processing unit 23 performs a predicted value calculation process to obtain a predicted value (first predicted value) of the value of the dependent variable for each outlier detected in the outlier detection process. The predicted value calculation process corresponds to a predicted value calculation step of the present invention. More specifically, as shown in FIG. 6 , the predicted value calculation processing unit 23 first generates first learning data 35 (learning data for predicted value calculation) by excluding outlier data 34, which are data of the outliers detected in the outlier detection process, from the learning data 31. In this embodiment, the first learning data 35 (the number of data items is 1283 (=1229+(87−33))) is generated by excluding all of the outlier data 34 from the learning data 31. Then, the first learning data 35 is used as training data to generate a first regression model 36, which is a regression model showing the correlation between the explanatory variables and the dependent variable. The generated first regression model 36 is stored in the storage unit 3.

[0030] Thereafter, the predicted value calculation processing unit 23 calculates, for each piece of outlier data 34 (33 pieces of data), a first predicted value, which is a predicted value of the objective variable corresponding to the value of the explanatory variable included in the data, using the created first regression model 36. The first predicted value corresponding to each outlier calculated by the predicted value calculation processing is stored in the storage unit 3 as predicted value data 37 (33 pieces of data).

[0031] (Data replacement processing unit 24) The data replacement processing unit 24 performs a data replacement process in which the value of the objective variable for each outlier in the learning data 31 (the outliers excluded from the learning data 31 in the predicted value calculation process) is replaced with a value based on the first predicted value obtained in the predicted value calculation process, thereby creating outlier-replaced learning data 38. In this embodiment, the objective variable value of the outlier is replaced with the first predicted value. The data replacement process corresponds to the data replacement process of the present invention. The data replacement processing unit 24 stores the created outlier-replaced learning data 38 (1316 data items, see FIG. 6 ) in the storage unit 3.

[0032] (Prediction processing unit 25) The prediction processing unit 25 performs a prediction process to predict the value of the objective variable (third predicted value) using the outlier-replaced training data 38 obtained by the data replacement processing unit 24. The prediction process corresponds to the prediction step of the present invention. In the prediction process, first, the outlier-replaced training data 38 is used as training data to create a third regression model 39 (a regression model for predicting physical properties, etc.), which is a regression model that shows the correlation between the explanatory variables and the objective variable. Note that for data containing multiple objective variable values, the median of the multiple objective variable values ​​is used for learning. The created third regression model 39 is stored in the storage unit 3. Thereafter, the created third regression model 39 is used to predict a third predicted value, which is a predicted value of the objective variable corresponding to the value of the explanatory variable to be predicted. For example, in the example of FIG. 2, the objective variable is tensile strength (a physical property).

[0033] More specifically, prediction source data 40, which is the value of each explanatory variable input via the input device 5 or the like, is applied to the regression model 39 to obtain a third predicted value, which is the value of the corresponding objective variable. The obtained third predicted value is stored in the storage unit 3 as predicted data 41.

[0034] (Prediction result presentation processing unit 26) The prediction result presentation processing unit 26 performs a prediction result presentation process for presenting the prediction result of the prediction process. In the prediction result presentation process, for example, the display device 4 displays the prediction data 41 obtained in the prediction process.

[0035] (Data processing method) 7(a) is a flow diagram of a data processing method according to this embodiment. As shown in FIG. 7(a), first, in step S1, a data acquisition process is performed. In the data acquisition process, a data acquisition processing unit 21 acquires learning data 31 (1316 pieces of data in the case of FIG. 6) from an external device or the like. The acquired learning data 31 is stored in a storage unit 3.

[0036] Thereafter, in step S2, an outlier detection process is performed. In the outlier detection process, first, a data classification process is performed in step S7, as shown in FIG. 7(b). In the data classification process, first, in step S71, a coefficient of variation for each piece of data included in the learning data 31 is calculated, as shown in FIG. 8. Thereafter, in step S72, data whose coefficient of variation is greater than a preset reference value is stored in the storage unit 3 as outlier candidate data 32 (87 pieces of data in the case of FIG. 6). Thereafter, in step S73, data whose coefficient of variation is equal to or less than the reference value is stored in the storage unit 3 as normal data 33 (1229 pieces of data in the case of FIG. 6). Thereafter, the process returns, and proceeds to step S8 in FIG. 7(b).

[0037] In step S8, an outlier determination process is performed. In the outlier determination process, as shown in FIG. 9, first, in step S81, 1 is assigned as an initial value to n, a variable representing a data number, and the number of pieces of outlier candidate data 32 is assigned to n_max. Then, in step S82, a second regression model is created using normal data 33 as training data. Then, in step S83, the values ​​of the explanatory variables of the nth piece of data among the outlier candidate data 32 are applied to the second regression model to obtain a second predicted value (the value of the objective variable). Then, in step S84, the error rate between the second predicted value and the actual value (the value of the objective variable of the nth piece of data) is obtained. Then, in step S85, it is determined whether the absolute value of the obtained error rate is equal to or greater than a preset threshold. If step S85 returns YES (Y), in step S86, the nth piece of data is determined to be an outlier and stored in the storage unit 3 as outlier data 34 (33 pieces of data in the case of FIG. 6), and the process proceeds to step S88. If the determination in step S85 is NO (N), the nth data is determined not to be an outlier in step S87, and the process proceeds to step S88. In step S88, it is determined whether the variable n is equal to or greater than n_max. If the determination in step S88 is NO (N), the process returns to step S82 after incrementing the variable n in step S89. If the determination in step S88 is YES (Y), the process returns and proceeds to step S9 in FIG. 7(b).

[0038] In step S9, a detection result presentation process is performed. In the detection result presentation process, the detected outlier is presented by, for example, displaying the data determined to be an outlier in step S8, i.e., the outlier data 34, on the display 4. Thereafter, the process returns and proceeds to step S3 in FIG. 7(a).

[0039] In step S3, a predicted value calculation process is performed. In the predicted value calculation process, as shown in FIG. 10, first, in step S31, 1 is assigned as an initial value to m, a variable representing a data number, and the number of outlier data 34 is assigned to m_max. Thereafter, in step S32, the predicted value calculation processing unit 23 creates first learning data 35 (in the case of FIG. 6, the number of data is 1283 (=1229+(87−33))) by excluding outliers (outlier data 34) from the learning data 31. In step S33, a first regression model 36 is created using the first learning data 35 as training data. Thereafter, in step S34, the values ​​of the explanatory variables of the m-th data among the outlier data 34 are applied to the first regression model 36 to obtain a first predicted value (the value of the objective variable). In step S35, the obtained first predicted value is stored in the storage unit 3 as predicted value data 37 (in the case of FIG. 6, the number of data is 33). Thereafter, in step S36, it is determined whether the variable m is equal to or greater than m_max. If the determination in step S36 is NO (N), the variable m is incremented in step S37, and then the process returns to step S34. If the determination in step S36 is YES (Y), the process returns and proceeds to step S4 in FIG. 7(a).

[0040] In step S4, a data replacement process is performed. In the data replacement process, as shown in Fig. 11, in step S41, the values ​​of the objective variables of the outliers (outlier data 34) in the learning data 31 are replaced with first predicted values ​​(predicted value data 37) to create outlier-replaced learning data 38 (1316 data items in the case of Fig. 6), which is stored in the storage unit 3. Thereafter, the process returns and proceeds to step S5 in Fig. 7(a).

[0041] In step S5, a prediction process is performed. In the prediction process, as shown in FIG. 12, first, in step S51, prediction source data 40 is input using the input device 5 or the like. The input prediction source data 40 is stored in the storage unit 3. Thereafter, in step S52, a third regression model 39 is created using the outlier-replaced learning data 38 as training data, and stored in the storage unit 3. Thereafter, in step S53, the prediction source data 40 is applied to the regression model 39 to obtain a third predicted value (the value of the dependent variable), and in step S54, the obtained third predicted value is stored in the storage unit 3 as predicted data 41. Thereafter, the process returns, and proceeds to step S6 in FIG. 7(a).

[0042] In step S6, a prediction result presentation process is performed. In the prediction result presentation process, the prediction result of the prediction process (prediction data 41) is presented by, for example, displaying it on the display 4. The prediction source data 40 corresponding to the prediction data 41 may also be displayed on the display 4. Thereafter, the process ends.

[0043] (Variation) In the present embodiment, in the predicted value calculation process, data obtained by removing all outlier data 34 from the learning data 31 is used as the first learning data 35 to calculate the first predicted value. However, this is not limiting, and as shown in FIG. 13 , when calculating the first predicted value of a certain outlier, outlier data 34 other than the certain outlier may be included in the first learning data 35. That is, all data other than the outlier for which the first predicted value is to be calculated may be used as the first learning data 35 (1,315 pieces of data in the case of FIG. 13 ). In this case, a first regression model 36 is created individually for each outlier (33 regression models 36 for calculating predicted values ​​are created in the case of FIG. 13 ).

[0044] That is, in the predicted value calculation process, for each outlier detected in the outlier detection step, first training data 35 obtained by excluding the data of the target outlier (in this modification, only the data of the target outlier) from the training data 31 may be used as training data to create a first regression model 36. Then, the created first regression model 36 may be used to find a first predicted value, which is a predicted value of the objective variable corresponding to the value of the explanatory variable of the data of the target outlier.

[0045] The control flow of the predicted value calculation process in this case is shown in Fig. 14. The control flow shown in Fig. 14 is the same as that shown in Fig. 10, except that step S32 in Fig. 10 is replaced with step S32a and the return destination from step S37 is changed to step S32a. As shown in Fig. 14, in step S32a, first learning data 35 is created by excluding only the m-th data of the outlier data 34 from the learning data 31.

[0046] In the present embodiment, the value of the objective variable of the outlier is simply replaced with the first predicted value in the data replacement process, but this is not limiting and the value may be replaced with a value based on the first predicted value. For example, the value may be replaced with the average value of the objective variable of the outlier (e.g., the median if there are multiple values) and the first predicted value.

[0047] (Improvement of prediction accuracy by this embodiment) The prediction accuracy was calculated using the data processing method according to the present embodiment. Outliers were detected using 1,197 pieces of training data 31. In Example 1, a first predicted value was calculated using first training data 35, from which all detected outlier data 34 were removed. In Example 2, as described with reference to FIGS. 13 and 14 , all training data 31 other than the outliers for which the first predicted value was calculated was used as the first training data 35 to calculate the first predicted value. In Examples 1 and 2, a third regression model was created using outlier-replaced training data 38, in which the values ​​of the objective variables of the outliers were replaced with the first predicted values, and 81 pieces of data prepared separately from the training data 31 were used as test data to calculate the mean absolute percentage error (MAPE).

[0048] For comparison, the MAPE was calculated for Comparative Example 1 (no processing), in which a third regression model was created using the training data 31 containing outliers as is, and Comparative Example 2 (all outliers removed), in which a third regression model was created using training data from which all outliers had been removed, in the same manner as in Examples 1 and 2. The results are summarized in Figure 15.

[0049] As shown in FIG. 15 , the MAPE of Comparative Example 2 is lower than that of Comparative Example 1, and the prediction accuracy is slightly improved by removing outliers. However, in Comparative Example 2, the number of data used for learning is reduced by removing outliers, and the range of values ​​that can be predicted with high accuracy is narrowed. In contrast, in Examples 1 and 2 of the present invention, the MAPE is lower than that of Comparative Examples 1 and 2, and it was confirmed that the prediction accuracy is improved. Furthermore, in Examples 1 and 2, the value of the dependent variable is replaced with the first predicted value without removing outliers, so the number of data is not reduced, and the range of values ​​of the explanatory variables that can be predicted with high accuracy is wider than in Comparative Example 2.

[0050] (Actions and Effects of the Embodiments) As described above, the data processing method according to this embodiment includes a process of creating a first regression model 36 using, as training data, the first training data 35 from which outliers detected in the outlier detection process have been removed, and a process of calculating a predicted value using the first regression model 36 to obtain a first predicted value, which is a predicted value of a dependent variable corresponding to the value of the explanatory variable for each outlier, and replacing the outliers in the training data 31 with values ​​based on the first predicted value to create outlier-replaced training data 38.

[0051] By using the first regression model 36, from which outliers have been removed in the predicted value calculation process, as training data, it is possible to accurately determine the first predicted value, which is the predicted value of the dependent variable for the value of the explanatory variable that is an outlier. As a result, it is possible to improve the prediction accuracy in the prediction process using the outlier-replaced training data 38, in which the outliers have been replaced with the first predicted value, as training data (see FIG. 15 ). Furthermore, in this embodiment, the outliers are replaced with the first predicted value without being removed, so the number of data points in the training data does not decrease. Therefore, this embodiment makes it possible to prevent the numerical range of the explanatory variable that can be predicted with high accuracy from becoming narrow.

[0052] (Summary of the embodiment) Next, the technical ideas grasped from the above-described embodiments will be described by using the reference numerals and the like in the embodiments. However, the reference numerals and the like in the following description do not limit the components in the claims to the members and the like specifically shown in the embodiments.

[0053] [1] A data processing method for processing training data (31) including a plurality of data consisting of explanatory variables and dependent variables, the data processing method comprising: an outlier detection step of detecting outliers from the training data (31); a predicted value calculation step of creating first training data (35) from the training data (31) by excluding the outliers detected in the outlier detection step, creating a first regression model (36) using the first training data (35) as teacher data, and using the first regression model to determine a first predicted value, which is a predicted value of the dependent variable corresponding to the value of the explanatory variable of the excluded outlier; and a data replacement step of replacing the excluded outliers from the training data (31) with a value based on the first predicted value, to create outlier-replaced training data (38).

[0054] [2] The data processing method according to [1], wherein in the predicted value calculation step, the first learning data (35) is created by excluding all of the outliers detected in the outlier detection step from the learning data (31), the first learning data (35) is used as training data to create the first regression model (36), and the first predicted value corresponding to each of the excluded outliers is obtained using the first regression model (36).

[0055] [3] The data processing method according to [1], wherein in the predicted value calculation step, for each of the outliers detected in the outlier detection step, the first learning data (35) is created by excluding each of the outliers from the learning data (31), the first learning data (35) is used as training data to create the first regression model (36), and the first predicted value corresponding to each of the data is obtained using the first regression model (36).

[0056] [4] The data processing method according to [1], wherein each data included in the learning data (31) includes a plurality of values ​​of the dependent variable, and the outlier detection step includes: a data classification step of calculating a coefficient of variation of the value of the dependent variable for each of the data, and classifying each of the data into outlier candidate data (32) and normal data (33) based on the calculated coefficient of variation and a reference value; and an outlier determination step of creating a second regression model using the normal data (33) as teacher data, using the second regression model to calculate a second predicted value that is a predicted value of the dependent variable corresponding to a value of an explanatory variable of each data of the outlier candidate data (32), and determining whether each of the data of the outlier candidate data is an outlier based on the second predicted value and the value of the dependent variable of each data of the outlier candidate data.

[0057] [5] The data processing method according to [1], further comprising a prediction step of creating a third regression model (39) using the outlier-replaced learning data (38) as training data, and predicting a third predicted value, which is a predicted value of the dependent variable corresponding to a value of an explanatory variable to be predicted, using the third regression model (39).

[0058] [6] A data processing device (1) for performing data processing on learning data (31) including a plurality of data items each consisting of an explanatory variable and a dependent variable, the data processing device (1) comprising: an outlier detection processing unit (22) for detecting an outlier from the learning data (31); a predicted value calculation processing unit (23) for creating first learning data (35) from the learning data (31) by excluding the outliers detected by the outlier detection processing unit (22), creating a first regression model (36) using the first learning data (35) as teacher data, and using the first regression model (36) for calculating a first predicted value, which is a predicted value of the dependent variable corresponding to the value of the explanatory variable of the excluded outlier; and a data replacement processing unit (24) for replacing the excluded outlier from the learning data (31) with a value based on the first predicted value, to create outlier-replaced learning data (38).

[0059] (Addendum) Although the embodiments of the present invention have been described above, the invention according to the claims is not limited to the above-described embodiments. It should be noted that not all of the combinations of features described in the embodiments are necessarily essential to the means for solving the problems of the invention. Furthermore, the present invention can be appropriately modified and implemented within the scope of its spirit. [Explanation of symbols]

[0060] 1...Data processing device 2...Control unit 21...Data acquisition processing unit 22...Outlier detection processing unit 221...Data classification processing unit 222...outlier determination processing unit 23...Prediction value calculation processing unit 24...Data replacement processing section 25...Prediction processing unit 3...Storage section 31...Learning data 32...Outlier candidate data 33...Normal data 34...Outlier data 35…First training data 36…First regression model 37...Prediction data 38…Training data with outlier replacement 39…Third regression model

Claims

1. A data processing method for processing learning data including a plurality of data items each consisting of an explanatory variable and a target variable, an outlier detection step of detecting an outlier from the learning data; a predicted value calculation step of creating first learning data from the learning data by excluding the outliers detected in the outlier detection step, creating a first regression model using the first learning data as teacher data, and using the first regression model to determine a first predicted value, which is a predicted value of the objective variable corresponding to the value of the explanatory variable of the excluded outlier; a data replacement step of replacing the excluded outliers from the learning data with values ​​based on the first predicted values ​​to create outlier-replaced learning data, Data processing methods.

2. In the predicted value calculation step, the first learning data is created by excluding all of the outliers detected in the outlier detection step from the learning data, the first learning data is used as teacher data to create the first regression model, and the first predicted value corresponding to each of the excluded outliers is obtained using the first regression model. The data processing method according to claim 1 .

3. In the predicted value calculation step, for each of the outliers detected in the outlier detection step, the first learning data is created by excluding the respective data from the learning data, the first learning data is used as teacher data to create the first regression model, and the first predicted value corresponding to each of the data is obtained using the first regression model. The data processing method according to claim 1 .

4. each data included in the learning data includes a plurality of values ​​of the objective variable; The outlier detection step includes: a data classification step of calculating a coefficient of variation of the value of the objective variable for each of the data, and classifying each of the data into outlier candidate data and normal data based on the calculated coefficient of variation and a reference value; an outlier determination step of creating a second regression model using the normal data as teacher data, using the second regression model to determine a second predicted value which is a predicted value of the dependent variable corresponding to a value of an explanatory variable of each data item of the outlier candidate data, and determining whether each data item of the outlier candidate data is an outlier based on the second predicted value and the value of the dependent variable of each data item of the outlier candidate data. The data processing method according to claim 1 .

5. The method further comprises a prediction step of creating a third regression model by using the outlier-replaced learning data as teacher data, and predicting a third predicted value, which is a predicted value of the dependent variable corresponding to a value of an explanatory variable to be predicted, by using the third regression model. The data processing method according to claim 1 .

6. A data processing device that processes learning data including a plurality of data consisting of explanatory variables and target variables, an outlier detection processing unit that detects outliers from the learning data; a predicted value calculation processing unit that creates first learning data from the learning data by excluding the outliers detected by the outlier detection processing unit, creates a first regression model using the first learning data as teacher data, and calculates a first predicted value, which is a predicted value of the objective variable corresponding to the value of the explanatory variable of the excluded outlier, using the first regression model; a data replacement processing unit that replaces the excluded outliers from the learning data with values ​​based on the first predicted value to create outlier-replaced learning data, Data processing device.

Images