Data selection system and data selection method

Abstract

To achieve both improvement of prediction accuracy of a model and acceleration of estimation processing of an influence of input learning data on the prediction accuracy of the model.SOLUTION: A data selection system comprises: table data which includes an explanatory variable and an objective variable of a prediction model used for machine learning as a column name in a header and manages the data of the explanatory variable and the data of the objective variable input to the prediction model as respective learning data in a table format; a data selection unit which selects partial data as respective learning data excluding a plurality of pieces of verification data selected from the respective learning data of the table data; a model learning unit which, after the partial data is input to the prediction model, estimates the prediction accuracy; and a data exclusion unit that evaluates the impact of inputting partial data on the prediction accuracy of the prediction model, and excludes harmful data estimated to reduce the prediction accuracy among the data for validation from each training data to be input into the prediction model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a data selection system and a data selection method, and is suitable for application to a data selection system relating to a technology that achieves both improved prediction accuracy and increased speed through machine learning, for example. [Background technology]

[0002] In recent years, in various industries, the efficiency of business operations has been improved by using predictions based on machine learning. Classification problems and regression problems in supervised learning are typical problem settings in machine learning. Classification problems and regression problems are problems in which the label value of data whose correct label is unknown is predicted based on training data, which is a pair of a correct label and input data. In general, the more training data there is, the higher the prediction accuracy tends to be, but there are cases in which the prediction accuracy is improved by excluding certain data from the training data due to various reasons such as input errors and noise.

[0003] Patent Document 1 proposes a method for identifying data that is harmful in the sense that it reduces prediction accuracy. In the method disclosed in Patent Document 1, each piece of learning data d is used to estimate whether the prediction accuracy will be improved by excluding a specific piece of learning data d. In the method disclosed in Patent Document 1, the influence of each piece of learning data d on the prediction accuracy is estimated. The influence here refers to the difference between the prediction accuracy of a model M trained with all learning data and the prediction accuracy of a model M' trained excluding the learning data d.

[0004] However, in the method disclosed in Patent Document 1, as described above, the prediction accuracy of the model M' trained excluding the training data d may be insufficient. There are two reasons for this. The first is an error caused by estimating the difference from information such as gradient, rather than actually training the model M' and calculating the difference in prediction accuracy. The second is because the interaction between the specific training data and other training data is not taken into consideration. Specifically, the prediction accuracy of the model does not decrease even if only the training data d is removed, but for example, if another training data d' is removed at the same time as the training data d, the prediction accuracy of the model decreases. This is because a special situation is not taken into consideration.

[0005] Therefore, as a natural method for improving the prediction accuracy of a model, Non-Patent Document 1 discloses a method for considering the interactions between multiple learning data and examining the prediction accuracy of a model trained by inputting various learning data. In this method, a large number of models are trained for all the learning data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-33544 [Non-patent literature]

[0007] [Non-Patent Document 1] Ghorbani, Amirata, and James Zou. "Data shapley: Equitable valuation of data for machine learning." International conference on machine learning. PMLR, 2019. Summary of the Invention [Problem to be solved by the invention]

[0008] However, the method disclosed in Non-Patent Document 1 requires learning various models for each training data, which requires an enormous amount of calculation time, and it is not possible to quickly estimate the effect of each training data on the prediction accuracy of the model.

[0009] The present invention has been made in consideration of the above points, and aims to propose a data selection system and a data selection method that can achieve both improved prediction accuracy of a model and faster estimation processing of the effect of input learning data on the prediction accuracy of the model. [Means for solving the problem]

[0010] In order to solve such problems, the present invention provides table data that includes explanatory variables and objective variables of a predictive model used in machine learning as column names in a header, and manages data of the explanatory variables and data of the objective variables to be input to the predictive model in a tabular format as each piece of learning data; a data selection unit that selects partial data as each piece of learning data excluding a plurality of selected pieces of verification data from each piece of the learning data in the table data; a model learning unit that inputs the partial data into the predictive model to estimate the prediction accuracy; and a data exclusion unit that evaluates the effect on the prediction accuracy of the predictive model when the partial data is input, and excludes harmful data among the verification data that is estimated to lower the prediction accuracy from each piece of learning data to be input to the predictive model.

[0011] In addition, the present invention includes a table data management step of managing explanatory variables and objective variables of a predictive model used in machine learning as table data in which a header includes column names, and data of the explanatory variables and data of the objective variables to be input to the predictive model are managed in a tabular format as each learning data; a data selection step of a data selection unit selecting partial data as each learning data of the table data excluding a plurality of selected verification data from each learning data of the table data; a model learning step of a model learning unit inputting the partial data into the predictive model to estimate the prediction accuracy; and a data exclusion step of a data exclusion unit evaluating the effect on the prediction accuracy of the predictive model when the partial data is input, and excluding harmful data among the verification data that is estimated to lower the prediction accuracy from each learning data to be input to the predictive model. Effect of the Invention

[0012] According to the present invention, it is possible to provide a data selection system and a data selection method that can achieve both improved prediction accuracy of a model and faster estimation processing of the effect of input learning data on the prediction accuracy of the model. [Brief description of the drawings]

[0013] [Figure 1] 2 is a block diagram showing an example of the software configuration of the data selection system according to the first embodiment. FIG. [Diagram 2] FIG. 2 is a diagram illustrating an example of table data illustrated in FIG. [Diagram 3] FIG. 2 is a diagram showing an example of analysis parameters shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of data harmfulness information corresponding to the three types of learning algorithms shown in FIG. 3. [Diagram 5] FIG. 13 is a diagram showing an example of an analysis result screen showing an analysis result including data harmfulness information. [Figure 6] 13 is a flowchart showing an example of a procedure for a harmfulness counting process. [Figure 7A]13A and 13B are diagrams illustrating an example of an evaluation count array and an evaluation result array immediately after initialization and during an update. [Figure 7B] 13A and 13B are diagrams illustrating an example of an evaluation count array and an evaluation result array immediately after initialization and during an update. [Figure 8] FIG. 11 is a diagram illustrating an example of processing related to verification data. [Figure 9] FIG. 13 is a diagram showing an example of data harmfulness information. [Figure 10] FIG. 11 is a block diagram showing an example of the software configuration of a data selection system according to a second embodiment. [Figure 11] 13 is a flowchart showing an example of a procedure of a harmfulness counting process according to the second embodiment. [Figure 12] FIG. 13 is a diagram showing an example of an analysis result screen showing analysis results regarding data harmfulness information, individual attributes, and common attributes. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0015] (1) First embodiment 1 is a block diagram showing an example of the software configuration of a data selection system 1 according to a first embodiment. The data selection system 1 is configured with at least one computer, and executes a data selection method by running a data selection program on the computer. The computer is configured with a CPU (Central Processing Unit), cache, memory, storage devices such as a hard disk drive, and a network card etc., all connected to a bus, and is equipped with a screen display unit (not shown). The data selection program executes processing using these hardware resources.

[0016] The data selection system 1 comprises analysis data 100, an analysis unit 300, and a display control unit 202, and preferably comprises a database (not shown).

[0017] The analysis data 100 includes table data 101 and analysis parameters 102, and is stored in the storage device described above. The table data 101 includes explanatory variables and objective variables of a prediction model as column names in a header, and manages the explanatory variable data and objective variable data input to the prediction model in a tabular format as each piece of learning data. The table data 101 includes a data body in which R symbols (R rows) representing missing values ​​or symbols such as text or real values ​​are arranged vertically and C symbols horizontally, and a header indicating the name and attribute of each column.

[0018] The analysis parameters 102 are parameters used by the analysis unit 300, and include a parameter name and its value. The parameter name indicates the name of the parameter managed by the analysis parameters 102, and the value indicates the parameter value itself. The analysis parameters 102 are set in advance. The parameters managed in the analysis parameters 102 will be described in detail later.

[0019] The analysis unit 300 includes a control unit 301, a partial data generation unit 302, a model learning unit 303, and a data harmfulness update unit 304. An overview of the processing of the analysis unit 300 will be described.

[0020] The control unit 301 acquires the table data 101 and the analysis parameters 102, and initializes an evaluation count array and an evaluation result array, which will be described later. The control unit 301 prepares or acquires verification data used to evaluate the prediction accuracy of the prediction model, as will be described later. The verification data has exactly the same column structure as the table data 101.

[0021] The partial data generation unit 302 selects at least one, and preferably a plurality of pieces of learning data as a subset from a row set constituted by a group of learning data included in the table data 101 .

[0022] The partial data generation unit 302 is an example of a data selection unit, and selects at least one, preferably multiple, of the learning data in the table data 101, excluding each of the selected learning data (hereinafter also referred to as "partial data").

[0023] The partial data generation unit 302 is an example of a data selection unit, and selects each piece of learning data excluding verification data selected based on a predetermined evaluation priority from among the pieces of learning data, as partial data.

[0024] The partial data generating unit 302 is an example of a data selecting unit, and creates a harmfulness calculation data list including partial data as each learning data excluding multiple pieces of verification data selected from each learning data, and the model learning unit 303 inputs the partial data based on the harmfulness calculation data list to a prediction model to estimate prediction accuracy. Here, the harmfulness indicates an index regarding the degree of influence that each learning data may have on the prediction accuracy of the model when it is input.

[0025] As described later, the partial data generation unit 302 uses an algorithm corresponding to the multi-armed bandit problem as a method for calculating the evaluation priority.

[0026] The model learning unit 303 inputs partial data as each piece of learning data excluding the validation data into a prediction model and estimates prediction accuracy.

[0027] The model learning unit 303 uses each of the subsets as learning data, and learns and creates a prediction model (for example, a prediction model M(d) described below) using a learning algorithm with the learning data as input. The input learning data is partial data that is partly extracted and excluded at random from each learning data, which is a set of rows in the table data 101. The method of exclusion will be described later.

[0028] The data harmfulness updating unit 304 is an example of a data exclusion unit, which evaluates the impact on the prediction accuracy of the prediction model when partial data as each learning data excluding the verification data is input to the prediction model, and excludes harmful data among the verification data that is estimated to have a negative impact on the prediction accuracy from each learning data to be input to the prediction model.

[0029] The data harmfulness update unit 304 evaluates the prediction accuracy of each prediction model generated by the model learning unit 303. Specifically, the data harmfulness update unit 304 calculates the harmfulness of each row from the evaluation result based on the evaluation index of each prediction model. When the evaluation index is lowered by adding data of a certain row in the table data 101 to the learning data, the data harmfulness update unit 304 regards the data of the row as data with a high harmfulness (corresponding to the above-mentioned "harmful data"). A specific example will be described later.

[0030] In this embodiment, the control unit 301 controls the partial data generating unit 302, the model learning unit 303, and the data harmfulness updating unit 304 to repeatedly execute the above-mentioned processes until a termination condition, which will be described later, is satisfied.

[0031] The analysis unit 300 generates data harmfulness information representing information on learning data that may reduce the prediction accuracy of a prediction model based on the analysis data 100, and transmits the information to the display control unit 202. The display control unit 202 causes the screen display unit to display the data harmfulness information, as described below. This data harmfulness information is, for example, R rows of data, similar to the table data 101, and each column corresponds to each viewpoint of harmfulness. Each element corresponding to each viewpoint is made up of a numerical value representing whether each data of the corresponding row ID in the table data 101 is harmful data or not. This numerical value may be a continuous value, or may be, for example, a binary value of "0" or "1" representing whether the data is harmful or not, or may be a list of numerical values.

[0032] The data selection system 1 according to this embodiment includes analysis parameters 102 that manage the proportion of each piece of learning data in table data 101 that should be selected as verification data, and a partial data generation unit 302, which is an example of a data selection unit, selects each piece of learning data excluding the verification data selected at the proportion specified by the analysis parameters 102 from each piece of learning data, and sets it as partial data.

[0033] Display control unit 202 controls the display on a screen display unit (not shown) when the analyst sets and stores analysis data 100 and when the analyst checks the results of analysis by analysis unit 300.

[0034] For example, a database (not shown) stores partial data consisting of a part of the table data 101. The partial data may be stored in any one of the storage devices in the data selection system 1 instead of in a database (not shown).

[0035] Fig. 2 shows an example of the table data 101 shown in Fig. 1. The table data 101 has a header including the names of each column and R rows of data. The header indicates the name of each column, and includes, for example, a row ID (IDentifier), a first explanatory variable, a second explanatory variable, a third explanatory variable, and a response variable that is a prediction target.

[0036] Each piece of learning data constituting the table data 101 includes, for example, three explanatory variables, namely, "date" as a first explanatory variable, "time" as a second explanatory variable, and "temperature" as a third explanatory variable, and a response variable, namely, "power consumption (Wh)". Here, a case where the response variable takes a discrete value is called a "classification problem", and a case where the response variable takes a continuous value is called a "regression problem". In this embodiment, a regression problem is exemplified, but the present invention can be similarly applied to a classification problem. Note that image data and language data can also be expressed as a set of numerical values, and can therefore be expressed as the table data 101.

[0037] In the illustrated example, in row ID2, the second explanatory variable (time) and the third explanatory variable (temperature) are missing values ​​(NULL).

[0038] Fig. 3 shows an example of the analysis parameters 102 shown in Fig. 1. The analysis parameters 102 manage parameter names, which are names of parameters, and their values. As described above, the analysis parameters 102 include the proportion of each piece of learning data in the table data 101 that should be selected as verification data, and further include the following parameters.

[0039] The analysis parameters 102, for example, manage a value of "100" as the total number s of stages, manage "100" as the number of data to be evaluated in one stage, and manage three types of learning algorithms: "decision tree," "SVR (support vector regression)," and "ridge regression." In addition, the analysis parameters 102 include a threshold parameter τ and a tolerance parameter ε, which will be described later.

[0040] FIG. 4 shows an example of data harmfulness information corresponding to the three types of learning algorithms shown in FIG. 3. In the example shown, the extent to which each piece of data is harmful (hereinafter also referred to as "harmfulness") is expressed as a continuous value from three perspectives: harmfulness in decision tree, harmfulness in SVR, and harmfulness in ridge regression. Here, the harmfulness indicates that a larger positive value is more harmful, and a larger negative value is less harmful. The meaning of each harmfulness and an example of a calculation method will be described later.

[0041] 5 shows an example of an analysis result screen showing an analysis result including data harmfulness information. In the example shown, in addition to the data harmfulness information calculated by the analysis unit 300, table data 101 is combined and displayed as a component of the analysis result, and some of the display is omitted. Each row indicated by a row ID includes a check box 202X that is operated to determine whether or not the data is a target for determining whether or not it is harmful data.

[0042] This analysis result screen includes a data reflection button 202Z. The data reflection button 202Z is ​​a button for storing setting information on whether each row indicated by the row ID is harmful data or not in a storage device in response to an operation on the check box 202X. Note that, when the data harmfulness information and the number of columns of the table data 101 are large, this analysis result screen may display only a part of them. In that case, it is preferable that the part to be displayed can be selected from a list of column names. Note that, instead of being set in response to an operation on the check box 202X, the setting information may be set in advance based on a predetermined criterion.

[0043] For example, if the setting information is set by an analyst, the analyst checks the checkbox 202X of data that is considered to be harmful while looking at the data harmfulness information and the feature quantities (explanatory variables and objective variables) of each learning data, and presses the data reflection button 202Z, and the result is reflected in the above-mentioned database (not shown). Specifically, partial data consisting of a part of the table data 101 from which the checked data has been deleted is generated in the database. Alternatively, a model trained using the partial data is generated and stored in a storage device (not shown).

[0044] Fig. 6 is a flowchart showing an example of the procedure of harmfulness counting processing executed by the analysis unit 300. Fig. 7 shows an example of the evaluation count array and the evaluation result array immediately after initialization and during update, Fig. 8 shows an example of processing related to verification data, and Fig. 9 shows an example of data harmfulness information.

[0045] In step S101, the control unit 301 acquires the table data 101 and the analysis parameters 102. In addition, the control unit 301 prepares verification data. Furthermore, in step S101, the control unit 301 initializes an evaluation count array and an evaluation result array. The evaluation count array and the evaluation result array are each arrays capable of storing R elements (200 elements are exemplified in this embodiment) corresponding to the number of rows "R rows" of the table data 100 shown in FIG. 2. Variables r=1 to 200 are associated with the R arrays, respectively.

[0046] The evaluation count array is an array having R elements each representing the number of times that the harmfulness has been evaluated up to the present time for each row r (in this embodiment, examples are "1" corresponding to row ID 1 to "200" corresponding to row ID 200) of the table data 101 shown in Fig. 2 (see the upper side of Fig. 7A and the upper side of Fig. 7B, which will be described later). Immediately after initialization, all elements of the evaluation count array are set to 0, as shown in Fig. 7A.

[0047] On the other hand, the evaluation result array is an array having R lists [r_1, r_2, . . . , r_k] of the harmfulness evaluation results up to the present time for data in each row r (in this embodiment, "1" corresponding to row ID 1 to "200" corresponding to row ID 200 are exemplified) of table data 101 shown in Fig. 2 as the r-th element (see the lower side of Fig. 7A and the lower side of Fig. 7B described later). The evaluation result array immediately after initialization is a list with all elements empty, as shown in Fig. 7A.

[0048] Furthermore, in step S101 of Fig. 6, the control unit 301 prepares or acquires verification data. The verification data is used when evaluating the prediction accuracy of the prediction model M(d) in step S105 described later, and has exactly the same column structure as the table data 101. The verification data may be prepared in advance by an analyst separately from the table data 101, or may be data consisting of one of two rows of the table data 101 divided based on a method specified by the analyst (for example, one of data A and data B described later). Details of the latter will be described later.

[0049] In this embodiment, for example, a proportion of verification data is specified in the analysis parameters 102, and at least one piece of data, preferably multiple pieces of data, are randomly extracted from the table data 101 for the specified proportion of rows, and used as the verification data.

[0050] In this embodiment, first, the data harmfulness update unit 304 starts a first loop process in which harmful data is repeatedly removed from each piece of learning data to be input to the prediction model T times for the total number of stages t=1 to T.

[0051] In step S102, the model learning unit 303 calculates an evaluation priority P(r) for each row r of the table data 101. This evaluation priority P(r) is an array having R real numbers representing the degree to which the harmfulness should be evaluated next as the r-th element for data in each row r of the table data 101 shown in FIG. 2 (in this embodiment, "1" corresponding to row ID 1 to "200" corresponding to row ID 200 are exemplified). The evaluation priority P(r) is calculated based on the evaluation count array and the evaluation result array described above. The evaluation priority P(r) is calculated by the following calculation formula (1). P(r)=(-1)*sqrt(N(r))×(|M(r)-τ|+ε)···(1)

[0052] Here, N(r) is the r-th element of the evaluation count array. M(r) is the average value of the list that is the r-th element of the evaluation result array. Also, sqrt represents a non-negative square root. Note that non-negative refers to a non-negative value. The threshold parameter τ and the allowable error parameter ε are set in advance in the analysis parameters 102 (see FIG. 3). The allowable error parameter τ is a parameter that represents the threshold value for determining whether something is harmful. The allowable error parameter ε is a parameter that represents the allowable error for which a judgment can be made incorrectly.

[0053] In this embodiment, the smaller the number of evaluations N(r) up to now, the higher the evaluation target. Also, in this embodiment, the closer the average value M(r) of past evaluation results is to the threshold parameter τ, the more difficult it is to determine whether it is harmful at the present time, so the higher the evaluation target is. The above-mentioned calculation method of the evaluation priority P(r) is based on an algorithm corresponding to a problem of determining whether the expected value of the evaluation priority P(r) is greater than the threshold parameter τ, for example, a multi-armed bandit problem, but other calculation methods may be used. In this embodiment, for example, when it is desired to efficiently search for the top K data in terms of harmfulness, a method using a calculation formula based on an algorithm corresponding to a multi-armed bandit problem that identifies the top K data can be adopted. In this case, K is set in advance through the analysis parameter 102. Here, in this embodiment, learning data that may adversely affect the prediction accuracy of the prediction model M(d) is also referred to as "harmful data". Note that the number of harmful data to be extracted is K, and the threshold parameter of the harmfulness is τ.

[0054] In step S103, the partial data generating unit 302 calculates a data list for calculating harmfulness. The data list for calculating harmfulness is a list of 100 pieces of data with high priorities for evaluating the harmfulness of the table data 101. Specifically, the partial data generating unit 302 sorts the evaluation priorities, for example, in descending order, and extracts the top D pieces, r1, r2, . . . , rD. D is the number of pieces of data for evaluating the harmfulness in one stage, and is set in advance to 100, for example, in the "number of pieces of data for evaluation in one stage" in the analysis parameters 102 shown in FIG. 3.

[0055] Thereafter, steps S104 to S106 are repeated to perform the following second loop process from d=0 to d=D. First, in step S104, the model learning unit 303 inputs learning data and learns a prediction model M(d) using a learning algorithm. Here, the learning data at d=d0 is the learning data of d0+d' rows, which will be described later, extracted from the learning data of each row constituting the table data 101. First, the d0 row is rows r1, r2, . . . , rd0 corresponding to the first d0 pieces of the harmfulness calculation data list. Note that in the process at d0=0, nothing is extracted from the harmfulness calculation data list. In addition, the remaining d' row learning data is the learning data of d' row randomly selected from the R-d0 rows that are not included in the first d0 pieces of learning data. The same d' row learning data is used for d=0 to d=D.

[0056] In this embodiment, the learning algorithm used is an algorithm previously built into the data selection system 1, or one or more algorithms previously specified in the analysis parameters 102. In this embodiment, for example, three types of algorithms are used: decision tree, SVR (support vector regression), and ridge regression. A list of available learning algorithms is pre-implemented in the data selection system 1, and one or more algorithms can be selected from the settings of the analysis parameters 102, for example by checking a checkbox.

[0057] The learning algorithm may be not only a commonly used one, but also a unique learning algorithm. For example, when using a deep learning model, the calculation time becomes enormous, and the following can be devised. First, the model learning unit 303 creates a prediction model learned from the entire table data 101 in advance. Then, the model learning unit 303 performs additional learning for only one epoch with the learning data in this process. In this way, since the difference between the learning data at d=d0 and the learning data at d=-d0-1 is only rd0, additional learning of one piece of data is only required once, and can be realized at high speed.

[0058] In step S105, the data harmfulness update unit 304 evaluates the prediction accuracy of the prediction model M(d). Specifically, an evaluation index V(d) is calculated based on the result of inferring the validation data with the prediction model M(d) and the correct label of the validation data M(d). For example, the evaluation index may be squared error for a regression problem, or AUC (Area Under the Curve) for a classification problem.

[0059] In step S106, the data harmfulness update unit 304 updates the evaluation count array and the evaluation result array based on the evaluation index V(d) obtained in step S105. First, the data harmfulness update unit 304 increments the rd-th element of the evaluation count array by "1" since the d-th data rd has been evaluated. Next, the data harmfulness update unit 304 adds V(d-1)-V(d) to the array that is the rd-th element of the evaluation result array. Here, V(d-1)-V(d) represents the amount by which the evaluation index has changed as a result of inference with the data rd excluded. The data harmfulness update unit 304 determines that the larger this amount V(d-1)-V(d), the higher the evaluation index of the learning model that does not use the d-th data rd, i.e., the d-th data rd is highly harmful data.

[0060] As described above, in this embodiment, the second loop process of steps S104 to S106 is executed D times, and then the second loop process ends.

[0061] In step S107, the control unit 103 determines whether or not a termination condition is satisfied, and if the termination condition is satisfied, ends the first loop process and executes step S108, which will be described later. Here, an example of the termination condition is that the maximum calculation time set in advance in the analysis parameters 102 is exceeded.

[0062] In step S108, the control unit 103 calculates the harmfulness of each row r by tallying up the elements of each array in the evaluation result array shown in the upper part of Fig. 9 illustrating an example of a decision tree. In step S108, the control unit 103 calculates the average value for the elements of each array in the evaluation result array as shown in the center of Fig. 9 (for example, "0.3" for row ID 1 and "0.2" for row ID 2 in the illustrated example), and outputs this as data harmfulness information as shown in the lower part of Fig. 9.

[0063] 9, a decision tree is used as the algorithm, and the above-mentioned steps S102 to S108 are performed for each algorithm previously specified in the analysis parameters 102, and the results are combined using the row ID as a key. For example, when the algorithm is a decision tree, SVR (support vector regression), and ridge regression, data harmfulness information as shown in FIG. 4 is obtained.

[0064] Finally, the display control unit 202 displays an analysis result screen, for example for a decision tree, as shown in Fig. 5 based on the data harmfulness degree information and a part of the table data 101. The data harmfulness degree update unit 304 excludes harmful data that is estimated to lower the prediction accuracy among the verification data from each learning data to be input to the prediction model.

[0065] Incidentally, in this embodiment, when the verification data is a part of the table data 101 as described above, it is not possible to calculate the harmfulness of the verification data part. In this case, in steps S102 to S108, data (partial data) obtained by excluding the verification data part from the original table data 101 is used as the table data 101. As a result of completing step S108, the data harmfulness update unit 304 calculates the harmfulness of the part excluding the verification data part, but for example, the harmfulness of all rows of the original table data 101 can be calculated by the following procedure.

[0066] That is, the partial data generation unit 302 divides the original table data 101, for example, as shown in FIG. 8, into data A consisting of learning data of 100×2 (200) rows, which is the number of data to be evaluated in one stage shown in FIG. 3, and data B consisting of learning data of 100×2 (200) rows, which is the number of data to be evaluated in one stage shown in FIG. 3.

[0067] Next, the partial data generation unit 302 verifies data B as verification data for the first 100 pieces of data A, and then verifies data B as verification data for the last 100 pieces of data A. Furthermore, the data harmfulness update unit 304 verifies data A as verification data for the first 100 pieces of data B, and then verifies data A as verification data for the last 100 pieces of data B.

[0068] Then, first, the harmfulness update unit 304 calculates the harmfulness of data A by performing steps S102 to S108 using data B as verification data, and then calculates the harmfulness of data B by performing steps S102 to S108 using data A as verification data, and combines the results.

[0069] Hereinafter, for the above-mentioned steps S102 to S107, the above-mentioned loop processing is repeated until the termination condition is satisfied, and is performed T times from the execution count t=1 to t=T (total number of stages) in one stage. In this embodiment, these T times of loop processing are called one stage. The total number of stages T is set in advance in the analysis parameters 102 shown in FIG. 3. If the termination condition is satisfied in step S107, the loop processing is terminated midway even if the total number of stages T has not been completed. The termination condition is considered to be satisfied when it is determined that the current verification data does not affect the prediction accuracy of the prediction model. This is because, in consideration of speeding up the processing, it is considered pointless to further verify the current verification data. If such a termination condition is appropriately set, the data harmfulness update unit 304 can find harmful data that may affect the prediction accuracy of the prediction model in a shorter time.

[0070] In the above embodiment, the regression problem is taken as an example, but the present embodiment can also be applied to the classification problem. Specifically, the present embodiment can also be applied to the case where the data harmfulness information is calculated not only for the harmfulness of the entire verification data to the prediction accuracy, but also for the harmfulness of each class, which is a part of the data. In the following explanation of the classification problem, the same configuration and procedure as those in the regression problem will not be described.

[0071] In this embodiment, the data harmfulness update unit 304 calculates one or more harmfulness indicators for each class of each piece of learning data, and outputs one or more harmfulness indicators for each pair of a row and a class in the table data 101.

[0072] First, the number of classes in a classification problem is set to C. For example, in a problem of distinguishing between defective and non-defective products, C=2. In step S105, when the data harmfulness update unit 304 evaluates the prediction accuracy of the prediction model M(d), it does not evaluate the inference results of all the verification data at once, but evaluates the prediction accuracy for each class. Next, in step S106, the data harmfulness update unit 304 calculates the harmfulness for each class using the evaluation results. In this case, the number of columns of data harmfulness information becomes C times. For example, the data harmfulness update unit 304 calculates the harmfulness for the defective class in the decision tree, and also calculates the harmfulness for the non-defective class in the decision tree.

[0073] The data selection system according to the present embodiment includes table data 101, which includes explanatory variables and objective variables of a predictive model used for machine learning as column names in a header, and which manages data of the explanatory variables and data of the objective variables to be input to the predictive model in a tabular format as each piece of learning data, a partial data generation unit 302 as an example of a data selection unit that selects partial data as each piece of learning data excluding a plurality of selected pieces of verification data from each piece of learning data in the table data 101, a model learning unit 303 that inputs the partial data into the predictive model to estimate prediction accuracy, and a data harmfulness update unit 304 as an example of a data exclusion unit that evaluates the effect on the prediction accuracy of the predictive model when partial data is input, and excludes harmful data among the verification data that is estimated to lower the prediction accuracy from each piece of learning data to be input to the predictive model.

[0074] In order to achieve high estimation accuracy, the data selection system 1 according to this embodiment considers interactions between input learning data and estimates based on the prediction accuracy of models trained from various subsets of input learning data. In addition, in order to estimate quickly, the data selection system 1 excludes from the evaluation targets those that are obviously harmful and those that are not obviously harmful at the time of evaluation up to a certain point, and focuses on evaluating only those that are difficult to determine whether they are harmful or not. The data selection system 1 excludes from the evaluation targets those learning data that are obviously harmful and those that are not obviously harmful at the time of evaluation up to a certain point, and focuses on evaluating only those learning data that are difficult to determine whether they are harmful or not. For example, as a result of learning models M(d,1) and M(d,2), if the prediction accuracy of the model trained with learning data including d is sufficiently high, d is quickly removed from the candidates for learning data to be excluded.

[0075] In this way, by narrowing down the data (learning data) that should be evaluated over time as explanatory variables and objective variables, which are input data for the predictive model used in machine learning, it is possible to improve the predictive accuracy of the predictive model while also speeding up the process of estimating the impact that the input learning data has on the predictive accuracy of the model.

[0076] In this embodiment, the table data includes analysis parameters 102 that manage the proportion of each piece of learning data to be selected as validation data, and the partial data generation unit 302 selects each piece of learning data, excluding validation data selected at the proportion specified by the analysis parameters 102, from among the pieces of learning data, as partial data. In this way, data that should be evaluated over a long period of time can be narrowed down to the proportion specified by the analysis parameters 102, thereby making it possible to speed up the process.

[0077] In this embodiment, the partial data generation unit 302 selects each piece of learning data excluding the verification data selected based on a predetermined evaluation priority from among the learning data, and sets it as partial data. In this way, data to be evaluated can be narrowed down over time, thereby speeding up the process.

[0078] In this embodiment, the partial data generating unit 302 creates a harmfulness calculation data list including partial data as each learning data except for a plurality of selected verification data from each learning data, and the model learning unit 303 inputs the partial data based on this harmfulness calculation data list into the prediction model to estimate the prediction accuracy. In this way, data that should be evaluated over a long period of time can be narrowed down based on the harmfulness calculation data list, thereby speeding up the process.

[0079] In this embodiment, the partial data generation unit 302 uses an algorithm corresponding to the multi-armed bandit problem as a method for calculating the evaluation priority. In this way, since there is no need to use a complex and non-versatile algorithm, it is possible to speed up the process of verifying the effect that the input learning data may have on the prediction accuracy of the prediction model.

[0080] In this embodiment, the data harmfulness update unit 304 calculates one or more harmfulness indices for each class of each piece of learning data, and outputs one or more harmfulness indices for each pair of a row and a class in the table data 101. In this way, data that should be evaluated over a long period of time can be narrowed down using the harmfulness indices, thereby speeding up the process.

[0081] In this embodiment, the data harmfulness update unit 304 repeats the process of removing harmful data from each piece of learning data to be input to the prediction model until the end condition is satisfied. In this way, by setting an appropriate end condition, the process of removing harmful data from each piece of learning data to be input to the prediction model can be ended at an appropriate timing, so that the process can be ended in a short time to increase the speed.

[0082] (2) Second embodiment In the second embodiment, the description of the configuration and operation similar to those of the first embodiment will be omitted, and the following description will focus on the differences.

[0083] The second embodiment differs from the first embodiment in that after the data harmfulness information is calculated in the analysis unit 300, the data harmfulness information is analyzed to obtain complementary harmfulness information, and an analysis result screen including the complementary harmfulness information is displayed on the screen display unit as described below. The complementary harmfulness information includes individual attributes characteristic of one harmful data and common attributes common to multiple harmful data.

[0084] 10 is a block diagram showing an example of the software configuration of the data selection system 1a according to the second embodiment. The data selection system 1a according to the second embodiment differs from the data selection system 1 according to the first embodiment in that a harmful data characteristic analysis unit 305 is added to the analysis unit 300.

[0085] The harmful data characteristic analysis unit 305 analyzes at least one of an individual attribute characteristic of one harmful data and a common attribute common to a plurality of harmful data. Furthermore, the harmful data characteristic analysis unit 305 displays an analysis result screen including at least one of the individual attribute and the common attribute on a screen display unit (not shown). Details of the harmful data characteristic analysis unit 305 will be described later.

[0086] Fig. 11 is a flow chart showing an example of a procedure of a harmfulness tallying process in the second embodiment. The procedure of the harmfulness tallying process in the second embodiment is similar to the procedure of the harmfulness tallying process shown in Fig. 6 in the first embodiment described above, except that step S109, which will be described later, is executed after step S108 is executed.

[0087] In step S109, the harmful data characteristic analysis unit 305 calculates individual attributes characteristic of one harmful data and common attributes common to a plurality of harmful data, and obtains analysis results regarding the individual attributes and common attributes.

[0088] 12 shows an example of an analysis result screen showing the analysis results of data harmfulness information, individual attributes, and common attributes. In the example shown in the figure, in addition to the data harmfulness information, which is the output in the first embodiment, the original table data 101, and the harmfulness judgment check box 202X, for example, as the analysis results of the above-mentioned individual attributes, a total of three individual attributes, namely, outlier information, the ratio of missing values, and the uncertainty of prediction, and as the analysis results of the above-mentioned common attributes, a total of two common attributes, namely, the correlation coefficient with the harmfulness and information on characteristic explanatory variables, are displayed, with some of the display omitted. Below, an example of a calculation method for each item of the individual attributes and the common attributes is shown.

[0089] First, we will discuss individual attributes. The first example, outlier information, will be described. For each row r, we calculate an index showing which explanatory variables are outliers. For example, we calculate what percentage of each explanatory variable row r falls into, and display those whose values ​​are close to 0% or 100%.

[0090] In Figure 12, in row ID 1, the explanatory variable "temperature" is in the top 2% of values, which indicates that the temperature is a very high value. In other words, it can be seen that the high temperature is the reason that row ID 1 is harmful data. Note that there is also a method to calculate and display not only the percentile of the explanatory variable values, but also the percentile of the feature contribution of the prediction model. An example of the feature contribution is the SHAP (SHapley Additive exPlanation) value.

[0091] The second example, the proportion of missing values, will be described. For each row r, the number of missing values ​​or the proportion of missing values ​​contained in each row r is calculated, and rows with a proportion larger or smaller than a threshold are displayed. The threshold may be set in advance in the data selection system, or may be set in the analysis parameters 102.

[0092] In Figure 12, row ID 2 has 67.7% missing values. In other words, it can be seen that the reason row ID 2 is harmful data is because it has a large percentage of missing values. In other words, in row ID 2 shown in Figure 2, two explanatory variables, the second explanatory variable and the third explanatory variable, out of the three explanatory variables, the first to third explanatory variables, are missing values.

[0093] The third example is prediction uncertainty. For each row r, we calculate and display the uncertainty in the prediction of the prediction model. In the case of a classification problem, an example of uncertainty is the second largest value among the predicted probabilities of each class. When the prediction is certain, the predicted probability is biased toward one class, but when there is a dilemma between two classes, the second largest value will also be large. In Figure 12, the prediction uncertainty for row ID 1 is 0.3, which is larger than the other data.

[0094] Next, common attributes will be described. The first example is the correlation coefficient with harmfulness. The correlation coefficient between each feature and harmfulness is calculated, and the top explanatory variables are displayed along with their correlation coefficient values. The calculation does not have to be performed on all data, and the top X items with the highest harmfulness can be selected. This X is specified in the analysis parameters 102. Figure 12 shows that for the top 100 items with the highest harmfulness, the correlation coefficient between harmfulness and temperature is high. In other words, it can be seen that the higher the temperature, the more likely it is that the data will be harmful.

[0095] The second example, information on characteristic explanatory variables, will be described. The calculation target does not have to be all data, and the top X items with the highest degree of harmfulness may be selected. This X is specified in advance in the analysis parameters 102. From the calculation target data, the number of items with the same explanatory variable value is tallied for each explanatory variable, and the items with the largest number are displayed.

[0096] FIG. 12 shows that, among the top 100 harmful data, there are 20 pieces of data in which the explanatory variable time has a value of 10 (10:00). Also, among the top 100 harmful data, there are 10 pieces of data in which the explanatory variable date has a value of 20230803 (August 3, 2023). In other words, this shows that the data acquired at 10:00 and the data acquired on August 3, 2023 contain a lot of harmful data.

[0097] The data selection system 1a according to this embodiment includes a harmful data characteristic analysis unit 305 that analyzes at least one of individual attributes characteristic of harmful data and common attributes common to multiple harmful data. In this way, it is possible to easily understand the reason why certain learning data corresponds to harmful data.

[0098] In this embodiment, the harmful data characteristics analysis unit 305 displays at least one of the individual attributes and the common attributes on the screen display unit. In this way, it becomes easier to visually understand the reason why certain learning data corresponds to harmful data.

[0099] (3) Variations In the above-described embodiment, an example is given of the data selection system 1 being configured as a single computer, but this is not limited thereto, and the computer having the storage device for storing the analysis data 100 and the database unit 400, the analysis unit 300, and the display control unit 202 may each be configured separately and communicate data, etc. with each other. [Industrial Applicability]

[0100] The present invention can be applied to a data selection system related to a technology that achieves both improved prediction accuracy and increased speed through machine learning. [Explanation of symbols]

[0101] 1: Computer, 100: Analysis data, 101: Table data, 102: Analysis parameters, 201: Startup unit, 202: Screen display unit, 300: Analysis unit, 301: Control unit, 302: Partial data generation unit, 303: Model learning unit, 304: Data harmfulness update unit, 305: Harmful data feature analysis unit

Claims

1. table data that includes explanatory variables and objective variables of a prediction model used in machine learning as column names in a header, and manages data of the explanatory variables and data of the objective variables input to the prediction model in a table format as each learning data; a data selection unit that selects partial data as each of the learning data of the table data excluding a plurality of selected verification data; a model learning unit that inputs the partial data into the prediction model to estimate the prediction accuracy; a data exclusion unit that evaluates an influence on a prediction accuracy of the prediction model when the partial data is input, and excludes harmful data of the verification data that is estimated to lower the prediction accuracy from each of the learning data to be input to the prediction model; A data selection system comprising:

2. an analysis parameter for managing a ratio of each of the learning data to be selected as the verification data in the table data; The data selection unit is The partial data is obtained by selecting each of the learning data excluding the verification data selected at a ratio specified by the analysis parameter from among the learning data.

2. The data selection system according to claim 1.

3. The data selection unit is The partial data is obtained by selecting each of the learning data except for the verification data selected based on a predetermined evaluation priority from the learning data.

2. The data selection system according to claim 1.

4. The data selection unit is A harmfulness degree calculation data list is created, which includes the partial data as each of the learning data except for a plurality of pieces of verification data selected from each of the learning data, The model learning unit includes: The partial data based on the harmfulness calculation data list is input to the prediction model to estimate the prediction accuracy.

2. The data selection system according to claim 1.

5. The data selection unit is As a method for calculating the evaluation priority, an algorithm corresponding to the multi-armed bandit problem is used.

4. The data selection system according to claim 3.

6. The data selection unit is Calculate one or more harmfulness indices for each class of each training data, and output one or more harmfulness indices for each pair of row and class of the table data.

3. The data selection system according to claim 2.

7. The data removal unit is Repeating the process of excluding the harmful data from each of the learning data to be input to the prediction model until a predetermined termination condition is met.

2. The data selection system according to claim 1.

8. a harmful data characteristic analysis unit that analyzes at least one of individual attributes characteristic of the harmful data and common attributes common to a plurality of the harmful data; 2. The system of claim 1 .

9. The harmful data characteristic analysis unit is At least one of the individual attribute and the common attribute is displayed on a screen display unit.

9. The system of claim 8.

10. a table data management step of managing explanatory variables and objective variables of a prediction model used in machine learning as table data, the explanatory variables and objective variables being input to the prediction model being included in a header as column names, and managing the explanatory variables and objective variables in a table format as each piece of learning data; a data selection step in which a data selection unit selects partial data as each of the learning data of the table data excluding a plurality of selected verification data from each of the learning data of the table data; a model learning step in which a model learning unit inputs the partial data into the prediction model to estimate the prediction accuracy; a data exclusion step in which a data exclusion unit evaluates an influence on the prediction accuracy of the prediction model when the partial data is input, and excludes harmful data in the verification data that is estimated to lower the prediction accuracy from each of the learning data to be input to the prediction model; 13. A data selection method comprising:

Images