Learning data selection method and computer system
The method addresses the inconsistency in prediction accuracy by calculating an influence score independent of the predictor, allowing for the selection of learning data that enhances model performance across varying conditions.
Patent Information
- Application Number
- JP2021117000
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-15
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2041-07-15
AI Technical Summary
Existing methods for selecting learning data in supervised learning rely on influence scores that are dependent on the characteristics of the predictor, leading to inconsistent prediction accuracy across different learning conditions.
A method and system for selecting learning data that calculates an influence score independent of the predictor, by associating learning data with test data under specific learning conditions, and using this score to generate a learning data set that improves prediction accuracy.
Enables the selection of learning data that enhances prediction accuracy by reducing dependence on specific learning conditions, resulting in improved model performance.
Smart Images

Figure 0007596228000001 
Figure 0007596228000002 
Figure 0007596228000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for selecting learning data to be used in machine learning. [Background technology]
[0002] In recent years, various services that utilize AI have been provided. Machine learning is known as a method for generating models that realize AI that can predict any event. Supervised learning is one of the machine learning algorithms. Supervised learning is learning that uses training data that consists of input and correct answer labels.
[0003] When generating a model using supervised learning, overlearning can be suppressed and prediction accuracy can be improved by performing learning using high-quality learning data. Here, high-quality learning data refers to learning data that is highly effective in improving the prediction accuracy of the model. In addition, in order to tune a model to suit a specific situation or application, it is necessary to perform learning using learning data that takes the situation or application into consideration.
[0004] Therefore, in supervised learning, it is important to appropriately select the learning data to be used. A technique described in Patent Document 1 is known for addressing this issue.
[0005] Patent Document 1 describes a method for analyzing learning data executed by a computer system that performs analysis on learning data used to construct a predictor, the computer system having a computer, the computer being connected to a database storing a plurality of learning data, the method including the steps of: selecting target learning data from the database by the computer; calculating an influence score representing the strength of influence that the target learning data has on the prediction accuracy of the predictor for a plurality of test data; and selecting, from the plurality of learning data, learning data to be included in a learning dataset to be input to construct a predictor, based on the influence score of each of the plurality of learning data. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2020-030738 A [Non-patent literature]
[0007] [Non-Patent Document 1] Pang Wei Koh, Percy Liang, “Understanding Black-box Predictions via Influence Functions,” arXiv preprint arXiv:1703.04730 (2017) Summary of the Invention [Problem to be solved by the invention]
[0008] When learning conditions such as a learning data set and hyperparameters are different, the behavior of the predictor for the input data is different. In the technology described in Patent Document 1, a predictor trained under arbitrary learning conditions is used, so the influence score may depend on the characteristics (learning conditions) of the predictor. Therefore, even if the influence score is used, the prediction accuracy may not be improved.
[0009] The present invention has been made in consideration of the above-mentioned problems. That is, the present invention provides a method and system for selecting learning data that contributes to improving the prediction accuracy of a target event by using an index that is not dependent on a predictor generated under specific learning conditions. [Means for solving the problem]
[0010] A representative example of the invention disclosed in the present application is as follows: That is, a method for selecting learning data used to generate a predictor, executed by a computer system, the computer system includes a computer having a processor and a memory connected to the processor, and is connected to a database storing a plurality of learning data and information for managing a plurality of predictors generated under different learning conditions, the method for selecting learning data includes a first step in which the computer selects a target predictor from among the plurality of predictors, and calculates, for each of a plurality of test data, an influence degree representing the strength of the influence of the learning data on the prediction accuracy of the target predictor for the test data; The method includes a second step of recording the influence of the training data for each of the plurality of test data in association with each other; a third step of the computer calculating an influence score of the training data for each of the plurality of predictors based on the influences of the training data associated with the predictor; a fourth step of the computer recording the influence scores of the training data for each of the plurality of predictors in association with each other; and a fifth step of the computer generating a training dataset by selecting the training data to be used from the plurality of training data based on the influence scores of each of the plurality of training data. Effect of the Invention
[0011] According to one aspect of the present invention, it is possible to select learning data that contributes to improving prediction accuracy. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a computer system of a first embodiment. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a computer according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a data structure of learning history information according to the first embodiment. [Figure 4A] FIG. 11 is a diagram illustrating an example of a data structure of impact information according to the first embodiment. [Figure 4B] FIG. 11 is a diagram illustrating an example of a data structure of impact information according to the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a data structure of influence score information according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a UI provided by a data receiving unit according to the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of a UI provided by a data receiving unit according to the first embodiment. [Figure 8] 1 is a flowchart illustrating a process executed by a computer according to a first embodiment. [Figure 9] 11 is a flowchart illustrating a learning continuation determination process executed by the computer according to the first embodiment. [Figure 10] 1 is a flowchart illustrating a process executed by a computer according to a first embodiment. [Figure 11] 11 is a flowchart illustrating an influence score calculation process executed by a computer according to the first embodiment. [Figure 12A] 11 is a flowchart illustrating an output information generating process executed by a computer according to the first embodiment. [Figure 12B] 11 is a flowchart illustrating an output information generating process executed by a computer according to the first embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of information output by the computer of the first embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of information output by the computer of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not to be interpreted as being limited to the description of the embodiment shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or purpose of the present invention.
[0014] In the configurations of the invention described below, the same or similar configurations or functions are given the same reference numerals, and duplicated explanations are omitted.
[0015] In this specification, the terms "first," "second," "third," and the like are used to identify components and do not necessarily limit the number or order.
[0016] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings, etc. EXAMPLES
[0017] FIG. 1 is a diagram illustrating an example of the configuration of a computer system according to a first embodiment.
[0018] The computer system is composed of computers 100-1, 100-2, 100-3, and a terminal 101. The computers 100-1, 100-2, 100-3, and the terminal 101 are connected to each other via a network 105 such as a wide area network (WAN) or a local area network (LAN). The network 105 may be connected by either a wired or wireless method.
[0019] In the following description, when there is no need to distinguish between the computers 100-1, 100-2, and 100-3, they will be referred to as the computer 100.
[0020] The terminal 101 is a computer operated by a user, and includes a processor, a main memory device, a sub-memory device, a network interface, an input device, and an output device, all of which are not shown. The terminal 101 manages a test data database 140. The test data database 140 stores test data for verifying the prediction accuracy of the predictor 122. The test data is composed of feature quantities of multiple items.
[0021] In this specification, a plurality of test data used to verify prediction accuracy is referred to as a test data set.
[0022] It is to be noted that the terminal 101 does not need to manage the test data database 140. In this case, a part or all of the training data stored in the training data database 124 is treated as test data.
[0023] The computer 100-1 is a computer that provides a UI (User Interface) that accepts operations from the terminal 101. The computer 100-1 includes a data accepting unit 110 and a result output unit 111.
[0024] Data accepting unit 110 accepts input from terminal 101 via the UI. Result output unit 111 outputs the results of the processing executed by computers 100-2 and 100-3 via the UI.
[0025] The computer 100-2 executes a learning process to generate a model (predictor 122) and predicts an arbitrary event using the model. The computer 100-2 has a learning condition setting unit 120, a learning unit 121, and a predictor 122, and also manages learning history information 123 and a learning data database 124.
[0026] The predictor 122 predicts an arbitrary event using input data. The learning condition setting unit 120 sets learning conditions in a learning process for generating the predictor 122. Here, the learning conditions are a concept including (1) a plurality of learning data used in the learning process and an input order of the plurality of learning data (learning order), and (2) hyperparameters. The learning conditions in this embodiment include at least a plurality of learning data used in the learning process and an input order of the plurality of learning data. The learning unit 121 executes the learning process.
[0027] The learning history information 123 is information for managing a learning history composed of learning conditions and learning results. The data structure of the learning history information 123 will be described with reference to FIG. 3. The learning data database 124 stores the learning data used in the learning process. The learning data is composed of feature amounts and correct answer labels for multiple items.
[0028] In this specification, a plurality of pieces of training data used in the training process are referred to as a training data set. As will be described later, training data to be included in the training data set is selected from the training data stored in the training data database 124.
[0029] The computer 100-3 selects learning data to be used for generating the predictor 122. Here, "generating the predictor 122" is a concept that includes generating a new predictor 122 and regenerating the predictor 122 by re-learning. The computer 100-3 has an influence score calculation unit 130 and a learning data selection unit 131, and also manages influence degree information 132 and influence score information 133.
[0030] The influence score calculation unit 130 calculates an influence score representing the strength of the influence of the learning data on the prediction accuracy of the predictor 122 for the test data set. The influence score in this embodiment is calculated using the influence representing the strength of the influence of the learning data on the prediction accuracy of the predictor 122 when making a prediction for any test data. The influence can be calculated using the calculation method described in Non-Patent Document 1. Note that this embodiment is not limited to the calculation method of the influence score and the influence.
[0031] Here, the calculation method of the influence degree described in Non-Patent Document 1 will be described. Using a plurality of learning data, test data, and predictor 122, influence score calculation unit 130 calculates a value indicating the fluctuation tendency of the parameter of predictor 122 when the learning data to be evaluated is included in the learning data set, and also calculates a value indicating the fluctuation tendency of the prediction result for any test data when the parameter of predictor 122 is varied. Influence score calculation unit 130 calculates the influence degree using two values. For one learning data to be evaluated, influence score calculation unit 130 calculates influence degrees for the number of test data. When the influence degree is positive, it indicates that the prediction accuracy of predictor 122 is improved, and when the influence degree is negative, it indicates that the prediction accuracy of predictor 122 is reduced.
[0032] The influence score calculated using the influence degree indicates the strength of the influence that the learning data has on the prediction accuracy of the predictor 122 in prediction for the test data group.
[0033] The learning data selection unit 131 selects learning data to be included in the learning data set based on the influence score of the learning data.
[0034] The influence information 132 is information for managing the influence of the learning data. The data structure of the influence information 132 will be described with reference to Figs. 4A and 4B. The influence score information 133 is information for managing the influence score of the learning data. The data structure of the influence score information 133 will be described with reference to Fig. 5.
[0035] Here, a description will be given of the hardware configuration of the computer 100. Fig. 2 is a diagram illustrating an example of the hardware configuration of the computer 100 of the first embodiment.
[0036] The computer 100 has a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. Each hardware configuration is connected to each other via an internal bus. Note that the computer 100 does not necessarily have to have the secondary memory device 203. The computer 100 may also have an input device and an output device.
[0037] The processor 201 executes a program stored in the main memory device 202. The processor 201 executes processing according to the program, thereby operating as a functional unit (module) that realizes a specific function, such as the influence score calculation unit 130. In the following explanation, when a process is explained using a functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.
[0038] The main memory 202 stores the programs executed by the processor 201 and information used by the programs. The main memory 202 also includes a work area that is temporarily used by the programs.
[0039] The main memory device 202 of the computer 100-1 stores programs for implementing the data receiving unit 110 and the result output unit 111. The main memory device 202 of the computer 100-2 stores programs for implementing the learning condition setting unit 120, the learning unit 121, and the predictor 122. The values of parameters defining the predictor 122 are stored in the learning history information 123. The main memory device 202 of the computer 100-3 stores programs for implementing the influence score calculation unit 130 and the learning data selection unit 131.
[0040] The secondary storage device 203 is a device such as a hard disk drive (HDD) or a solid state drive (SSD) that permanently stores data.
[0041] The secondary storage device 203 of the computer 100-2 stores the learning history information 123 and the learning data database 124. The learning data database 124 may be stored in a storage system accessible by the computer 100-2. The secondary storage device 203 of the computer 100-3 stores the influence information 132 and the influence score information 133.
[0042] In addition, regarding each functional unit of each computer 100, multiple functional units may be integrated into one functional unit, or one functional unit may be divided into multiple functional units for each function. Also, the functional units of each computer 100 may be integrated into one computer 100.
[0043] FIG. 3 is a diagram illustrating an example of a data structure of the learning history information 123 according to the first embodiment.
[0044] The learning history information 123 is data in a table format, and stores entries including a learning condition ID 301, a learning order 302, hyper parameters 303, and a learning result 304. There is one entry for one learning condition. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.
[0045] The learning condition ID 301 is a field for storing the identification information of the learning condition. In this embodiment, the identification information of the learning condition is a combination of the symbol "P" and an identification number. The identification information of the learning data is a combination of the symbol "L" and an identification number, and the identification information of the test data is a combination of the symbol "T" and an identification number.
[0046] The learning order 302 is a group of fields that stores information about the learning data included in the learning data set and the input order of the learning data. In this embodiment, a learning process is executed to generate a plurality of subsets (batches) from a plurality of learning data stored in the learning data database 124, and update parameters that define the predictor 122 for each batch. The learning order 302 includes fields equal to the number of batches. Identification information of the learning data arranged in the input order is stored in one field.
[0047] The hyperparameters 303 are a group of fields that store hyperparameters used in the learning process. The hyperparameters 303 include fields that store a learning rate, the number of epochs, and the like.
[0048] The learning result 304 is a group of fields that stores the results of the learning process. The learning result 304 includes fields equal to the number of batches. In one field, the parameter values and prediction accuracy that define the predictor 122 are stored as learning results.
[0049] 4A and 4B are diagrams illustrating an example of a data structure of the impact information 132 according to the first embodiment.
[0050] 4A is a diagram showing an image of the data structure of the impact information 132. The impact information 132 is three-dimensional array data with learning data, test data, and learning conditions as indexes.
[0051] FIG. 4B shows a two-dimensional array 400 where the learning condition ID 301 is "P_1".
[0052] L_i represents identification information of the learning data in the i-th row, and T_h represents identification information of the test data in the h-th column. The cells store the degree of influence. For example, in the i-th row and h-th column, an influence representing the strength of the influence that the learning data (L_i) has on the prediction accuracy of the predictor 122 when making a prediction for the test data (T_h) is stored. One row can be treated as a vector having the influence of one learning data as a component. In the following description, the vector having the influence as a component will be referred to as the learning data influence vector.
[0053] Therefore, the influence information 132 is information for managing the influence matrix (two-dimensional array 400) generated by enumerating the influence vectors for each learning condition.
[0054] FIG. 5 is a diagram illustrating an example of the data structure of the influence score information 133 according to the first embodiment.
[0055] The influence score information 133 includes multiple entries each consisting of a learning data ID 501, an influence score 502, and an integrated influence score 503. There is one entry for one learning data. Note that the fields included in the entry are not limited to those described above. Any of the above-mentioned fields may not be included, and other fields may be included.
[0056] Learning data ID 401 is a field that stores identification information of learning data. Influence score 402 is a field group that stores influence scores of learning data. Influence score 402 includes fields equal to the number of learning conditions. In each field, an influence score calculated using a predictor 122 corresponding to the learning condition is stored. Integrated influence score 503 is a field that stores an integrated influence score calculated from the influence scores of learning data calculated using predictors 122 for each learning condition.
[0057] 6 and 7 are diagrams illustrating an example of a UI provided by the data receiving unit 110 according to the first embodiment.
[0058] 6 is a UI provided by the data accepting unit 110, and is displayed on the output device of the terminal 101. The GUI 600 includes a data set setting field 610, a learning condition setting field 620, an output selection field 630, and an execute button 640.
[0059] The dataset setting field 610 is a field for setting information (dataset setting information) related to the training dataset and the test dataset. The dataset setting field 610 includes a training dataset setting field 611, a radio button 612, a radio button 613, and a test dataset setting field 614.
[0060] The learning data set setting field 611 is a field for setting a default learning data set. In the learning data set setting field 611, a path indicating the storage location of the learning data database 124 and the like are set.
[0061] The radio button 612 is selected when a test dataset is to be input. The radio button 613 is selected when a test dataset is not to be input. The test dataset setting field 614 is a field for setting the test dataset to be input. In the test dataset setting field 614, a path indicating the storage location of the test dataset or a file corresponding to the test dataset, etc. is set. When the radio button 612 is operated, input to the test dataset setting field 614 is enabled.
[0062] The learning condition setting field 620 is a field for setting information related to the learning conditions (learning condition setting information). The learning condition setting field 620 includes a learning condition number setting field 621, a batch size setting field 622, a learning rate search range 623, and a learning rate decay field 624.
[0063] The learning condition number setting field 621 is a field for setting the maximum number of learning conditions to be generated. The batch size setting field 622 is a field for setting the number of learning data to be included in a batch, i.e., the batch size. The learning rate search range 623 is a field for setting the range of the learning rate. The learning rate decay 624 is a field for setting the decay width of the learning rate.
[0064] The output selection field 630 is a field for selecting information to be output. The output selection field 630 includes check boxes 631, 632, 633, 634, and 635, and a learning condition ID setting field 636.
[0065] Check box 631 is selected when outputting information related to the influence score and the influence degree. Check box 632 is selected when outputting information related to the influence degree of a specific learning condition. The learning condition ID setting field 636 is a field for setting identification information of the learning condition. When check box 631 is operated, input to check box 632 and learning condition ID setting field 636 is enabled.
[0066] Check box 633 is selected when a learning data set consisting of learning data selected based on the influence score is to be output. Check box 634 is selected when a predictor 122 generated using a learning data set consisting of learning data selected based on the influence score is to be output. Check box 635 is selected when a learning effect indicating the relationship between a threshold value used when selecting learning data based on the influence score and prediction accuracy is to be output.
[0067] In the following description, a request corresponding to check box 631 will be referred to as an influence score output request, a request corresponding to check box 633 will be referred to as a learning dataset output request, a request corresponding to check box 634 will be referred to as a predictor output request, and a request corresponding to check box 635 will be referred to as a learning effect output request. Note that when check box 632 is operated, the influence score output request includes identification information of the learning conditions.
[0068] The execute button 640 is an operation button for instructing execution of processing. When the user operates the execute button 640, an execution request including the information set in the data set setting field 610, the learning condition setting field 620, and the output selection field 630 is transmitted to the data receiving unit 110. When the data receiving unit 110 receives the execution request, it transmits a learning instruction to the computer 100-2, and after receiving a response from the computer 100-2, transmits a learning data selection instruction to the computer 100-3.
[0069] The learning instruction includes data set setting information and learning condition setting information. The learning data selection instruction includes a request according to an operation of the output selection field 630.
[0070] The API 700 shown in FIG.
[0071] A user inputs commands to the API 700 to configure settings related to learning data, test data, learning conditions, and information to be output.
[0072] FIG. 8 is a flowchart illustrating the process executed by the computer 100-2 of the first embodiment.
[0073] When the computer 100-2 receives a learning instruction from the computer 100-1 that has accepted the execution request, the computer 100-2 executes the process described below.
[0074] The learning condition setting unit 120 acquires the data set setting information and the learning condition setting information included in the learning instruction (step S101). At this time, the learning condition setting unit 120 sets the maximum number Imax of learning conditions to be generated and the number of batches Smax. The number of batches may be given as an external input, or may be automatically calculated from the batch size and the number of learning data stored in the learning data database 124.
[0075] The learning condition setting unit 120 initializes a variable I (step S102). The variable I is a variable that represents an identification number of a learning condition.
[0076] The learning condition setting unit 120 generates a plurality of batches using the learning data stored in the learning data database 124 (step S103).
[0077] Specifically, the learning condition setting unit 120 randomly selects a predetermined number of learning data from the learning data database 124 based on the batch size and dataset setting information included in the learning condition setting information, and generates Smax batches by determining the input order of the multiple learning data.
[0078] At this time, the learning condition setting unit 120 adds an entry to the learning history information 123, and sets the learning condition ID 301 of the entry to identification information corresponding to the variable I. For example, when the variable I is 3, the identification information "P_3" is set to the learning condition ID 301. The learning condition setting unit 120 sets the learning data included in the batch in the input order in each field of the learning order 302 of the added entry.
[0079] The learning condition setting unit 120 determines hyperparameters such as a learning rate based on the learning condition setting information (step S104).
[0080] In this embodiment, the learning condition setting unit 120 determines a learning rate and a decay width. At this time, the learning condition setting unit 120 sets the determined hyperparameters to the hyperparameters 303 of the entry added to the learning history information 123. In addition, the learning condition setting unit 120 outputs a learning instruction including identification information of the learning conditions to the learning unit 121.
[0081] The learning unit 121 initializes a variable S (step S105). The variable S is a variable that represents an identification number of a batch.
[0082] The learning unit 121 learns parameters of the predictor 122 using the learning data included in the batch corresponding to the variable S (step S106).
[0083] Specifically, the learning unit 121 refers to the learning history information 123, searches for an entry corresponding to the identification information of the learning condition included in the learning instruction, and executes a learning process based on the batch corresponding to the variable S of the learning order 302 of the entry and the hyperparameters 303 of the entry. In this embodiment, learning is performed according to the input order. Furthermore, the learning unit 121 sets the parameter value and prediction accuracy of the predictor 122 in the field corresponding to the variable S of the learning result 304 of the entry corresponding to the identification information of the learning condition.
[0084] The learning unit 121 executes a learning continuation determination process (step S107), and determines whether or not to continue the learning based on the result of the process (step S108).
[0085] If it is determined that the learning is not to be continued, the learning unit 121 outputs a response indicating the stop of the learning to the learning condition setting unit 120. If the learning condition setting unit 120 receives the response, the process proceeds to step S111.
[0086] In this embodiment, when a state in which calculation of the influence degree is not expected is detected, the computer 100-2 ends learning of the current learning conditions.
[0087] If it is determined that learning is to be continued, the learning unit 121 outputs a response indicating the continuation of learning to the learning condition setting unit 120. When the learning condition setting unit 120 receives the response, it determines whether the value of the variable S matches Smax (step S109). That is, it determines whether processing has been completed for all batches.
[0088] If it is determined that the value of the variable S does not match Smax, the learning condition setting unit 120 adds 1 to the variable S and sets the value to the variable S (step S110), and then returns to step S106. In this case, the same process is performed using another batch.
[0089] If it is determined that the value of the variable S matches Smax, the learning condition setting unit 120 proceeds to step S111.
[0090] In step S111, the learning condition setting unit 120 determines whether or not the value of the variable I coincides with Imax (step S111). That is, it is determined whether or not the maximum number of learning conditions has been reached.
[0091] If it is determined that the value of the variable I does not match Imax, the learning condition setting unit 120 adds 1 to the variable I and sets the value to the variable I (step S112), and then returns to step S103. In this case, new learning conditions are set, and the same processing is executed under the new learning conditions.
[0092] If it is determined that the value of the variable I matches Imax, the learning condition setting unit 120 ends the process. At this time, the learning condition setting unit 120 transmits a response to the data receiving unit 110 indicating the completion of the process.
[0093] FIG. 9 is a flowchart illustrating the learning continuation determination process executed by the computer 100-2 according to the first embodiment.
[0094] The learning unit 121 calculates the gradient of the parameters of the predictor 122 (step S201). A method for calculating the gradient of the parameters is a known technique, and therefore a detailed description thereof will be omitted.
[0095] The learning unit 121 determines whether or not the continuation condition is satisfied based on the gradient of the parameter (step S202).
[0096] For example, the learning unit 121 determines whether the gradient of the parameter is 0 or is greater than a threshold value. If the gradient of the parameter is 0 or is greater than a threshold value, it is determined that the continuation condition is not satisfied.
[0097] If it is determined that the continuation condition is satisfied, the learning unit 121 determines to continue the learning (step S203), and then ends the learning continuation determination process.
[0098] If it is determined that the continuation condition is not satisfied, the learning unit 121 determines to stop learning (step S204), and then ends the learning continuation determination process.
[0099] FIG. 10 is a flowchart illustrating the process executed by the computer 100-3 of the first embodiment.
[0100] When the computer 100-3 receives a learning data selection instruction from the computer 100-1 that has accepted the execution request, the computer 100-3 executes the process described below.
[0101] The influence score calculation unit 130 executes an initialization process (Step S301). Specifically, the following process is executed.
[0102] (S301-1) The influence score calculation unit 130 acquires a plurality of pieces of learning data from the learning data database 124.
[0103] When the training data selection instruction includes operation information of radio button 612 in dataset setting field 610, influence score calculation unit 130 generates a training dataset from the acquired multiple training data, and acquires a test dataset included in the training data selection instruction. Influence score calculation unit 130 assigns an identification number to data included in each dataset.
[0104] When the learning data selection instruction includes operation information of the radio button 613 in the data set setting field 610, the influence score calculation unit 130 selects a predetermined number of learning data from the acquired multiple learning data as test data. The influence score calculation unit 130 generates a learning data set from the multiple learning data excluding the learning data selected as the test data, and also generates a test data set from the predetermined number of test data. The influence score calculation unit 130 assigns an identification number to the data included in each data set.
[0105] In addition, this embodiment is not limited to the method of selecting test data from the learning data database 124. For example, the influence score calculation unit 130 randomly selects test data. In addition, the number of test data can be set arbitrarily. The number can be updated at any timing.
[0106] (S301-2) The influence score calculation unit 130 initializes the influence degree information 132 and the influence score information 133.
[0107] This concludes the description of the process in step S301.
[0108] Next, influence score calculation unit 130 initializes variable I (step S302), initializes variable J (step S303), and also initializes variable K (step S304). Variable I is a variable representing the identification number of a learning condition, variable J is a variable representing the identification number of test data, and variable K is a variable representing the identification number of learning data.
[0109] Specifically, the influence score calculation section 130 sets the variable I, the variable J, and the variable K to 1, respectively.
[0110] In step S302, influence score calculation unit 130 sets the maximum number of learning conditions to be generated to Imax, and adds two-dimensional array 400 corresponding to variable I to influence information 132. In addition, influence score calculation unit 130 accesses learning history information 123, and acquires parameter values defining predictor 122 from learning result 304 of the entry corresponding to variable I. Here, it is assumed that the parameter value with the highest prediction accuracy is acquired from learning result 304 of the entry corresponding to learning condition I. In step S303, influence score calculation unit 130 sets the number of test data included in the test dataset to Jmax, and in step S304, sets the number of learning data included in the learning dataset to Kmax.
[0111] Next, influence score calculation section 130 acquires test data corresponding to variable J from the test data set, and acquires learning data corresponding to variable K from the learning data set (Step S305).
[0112] Next, the influence score calculation unit 130 calculates the influence using the predictor 122 corresponding to the test data (J), the learning data (K), and the learning condition (I) (step S306), and sets the calculated influence in the cell corresponding to the Kth row and Jth column of the two-dimensional array 400 corresponding to the variable I. Note that the method of calculating the influence is described in Non-Patent Document 1, and therefore a detailed description thereof will be omitted.
[0113] Next, the influence score calculation unit 130 determines whether or not the value of the variable K coincides with Kmax (step S307). That is, it is determined whether or not the processing has been completed for all the learning data.
[0114] If it is determined that the value of the variable K does not match Kmax, influence score calculation section 130 adds 1 to the variable K and sets the value of the variable K (step S308), and then returns to step S305.
[0115] If it is determined that the value of the variable K matches Kmax, the influence score calculation unit 130 determines whether the value of the variable J matches Jmax (step S309). That is, it is determined whether the processing is completed for all the test data.
[0116] If it is determined that the value of variable J does not match Jmax, influence score calculation section 130 sets a value obtained by adding 1 to variable J (step S310), and then returns to step S304.
[0117] If it is determined that the value of the variable J matches Jmax, the influence score calculation unit 130 determines whether or not the value of the variable I matches Imax (step S311). That is, it is determined whether or not to generate a new learning condition.
[0118] If it is determined that the value of variable I does not match Imax, influence score calculation section 130 adds 1 to variable I and sets the value of variable I (step S312), and then returns to step S303.
[0119] If it is determined that the value of the variable I matches Imax, the influence score calculation unit 130 executes an influence score calculation process (step S313). Details of the influence score calculation process will be described with reference to FIG.
[0120] Next, the influence score calculation unit 130 executes the output information generation process (step S314). After that, the influence score calculation unit 130 ends the process. Details of the output information generation process will be described with reference to Figs. 12A and 12B.
[0121] Note that multiple test data may be selected for one learning data to calculate the influence degree, i.e., step S303 and step S304 may be interchanged, and step S307 and step S308 and step S309 and step S310 may be interchanged.
[0122] FIG. 11 is a flowchart illustrating the influence score calculation process executed by the computer 100-3 of the first embodiment.
[0123] The influence score calculation unit 130 initializes a variable K (Step S401).
[0124] Specifically, the influence score calculation unit 130 sets the variable K to 1, and sets the number of pieces of learning data included in the learning dataset to Kmax. In addition, the influence score calculation unit 130 adds an entry to the influence score information 133, and sets the learning data ID 501 to the identification information of the learning data.
[0125] The influence score calculation unit 130 initializes a variable I (Step S402).
[0126] Specifically, the influence score calculation unit 130 sets the variable I to 1.
[0127] Next, the influence score calculation unit 130 acquires the Kth row of the two-dimensional array 400 corresponding to the variable I of the influence information 132 as the influence vector of the learning data (step S403).
[0128] Next, the influence score calculation unit 130 calculates an influence score of the learning data using the influence vector of the learning data, and registers it in the influence score information 133 (step S404).
[0129] This embodiment is not limited to the calculation method of the influence score. For example, the influence score calculation unit 130 calculates the total value or average value of each component of the influence vector as the influence score. In addition, the influence score calculation unit 130 calculates the inner product of an arbitrary coefficient vector and the influence vector as the influence score.
[0130] The influence score calculation section 130 refers to the influence score information 133, searches for an entry corresponding to the variable K, and sets the calculated influence score in the field corresponding to the variable I of the influence score 502 of the entry.
[0131] Next, the influence score calculation unit 130 determines whether or not the value of the variable I coincides with Imax (step S405). That is, it is determined whether or not the processing has been completed for all learning conditions.
[0132] If it is determined that the value of variable I does not match Imax, influence score calculation section 130 adds 1 to variable I and sets the value of variable I (step S406), and then returns to step S403.
[0133] If it is determined that the value of the variable I matches Imax, the influence score calculation unit 130 calculates an integrated influence score (step S407) and registers it in the influence score information 133.
[0134] Specifically, the influence score calculation unit 130 refers to the influence score information 133 and searches for an entry corresponding to the variable K. The influence score calculation unit 130 calculates an integrated influence score using a plurality of influence scores stored in the influence score 502 of the searched entry. For example, a method of calculating an average value of the influence scores, a weighted average of the influence scores, or the like as the integrated influence score can be considered. A method of determining the weight based on the prediction accuracy of the predictor 122 can be considered.
[0135] The influence score calculation section 130 sets the calculated integrated influence score in the integrated influence score 503 of the entry.
[0136] Next, the influence score calculation unit 130 determines whether or not the value of the variable K coincides with Kmax (step S408). That is, it is determined whether or not the processing has been completed for all the learning data.
[0137] If it is determined that the value of variable K does not match Kmax, influence score calculation unit 130 adds 1 to variable K and sets the value to variable K (step S409), and then returns to step S402. At this time, influence score calculation unit 130 adds an entry to influence score information 133, and sets identification information of the learning data to learning data ID 501.
[0138] If it is determined that the value of the variable K matches Kmax, the influence score calculation section 130 ends the influence score calculation process.
[0139] Fig. 12A and Fig. 12B are flowcharts explaining the output information generating process executed by the computer 100-3 of the embodiment 1. Fig. 13 and Fig. 14 are diagrams showing an example of information output by the computer 100-3 of the embodiment 1.
[0140] The influence score calculation unit 130 determines whether or not the learning data selection instruction includes an influence score output request (Step S501).
[0141] If it is determined that the learning data selection instructions do not include an influence score output request, the influence score calculation unit 130 proceeds to step S505.
[0142] If it is determined that the learning data selection instruction includes an influence score output request, the influence score calculation unit 130 determines whether or not a learning condition is specified (Step S502).
[0143] Specifically, the influence score calculation unit 130 determines whether or not the influence score output request includes identification information of the learning condition. If the influence score output request includes identification information of the learning condition, the influence score calculation unit 130 determines that the learning condition is specified.
[0144] If it is determined that the learning condition is not specified, the influence score calculation unit 130 outputs the influence score information 133 via the computer 100-1 (step S503). After that, the influence score calculation unit 130 proceeds to step S505.
[0145] In this case, the result output unit 111 of the computer 100-1 generates display information for displaying the influence score information 133, and transmits the display information to the terminal 101.
[0146] If it is determined that the learning condition is specified, the influence score calculation unit 130 outputs the influence data and the influence score information 133 via the computer 100-1 (step S504). After that, the influence score calculation unit 130 proceeds to step S505.
[0147] Here, the influence data is a two-dimensional array 400 corresponding to the specified learning conditions.
[0148] In this case, the result output unit 111 of the computer 100-1 generates display information for displaying the impact data and the impact score information 133, and transmits the display information to the terminal 101. An output screen 1300 as shown in FIG.
[0149] 13 to the user, the user can easily select learning data in accordance with the characteristics of the data to be input. Also, the user can easily determine whether or not it is necessary to add learning data that improves the prediction accuracy of the predictor 122 for the test data.
[0150] In step S505, the influence score calculation unit 130 determines whether or not the learning data selection instruction includes a learning data set output request (step S505).
[0151] If it is determined that the learning data selection instruction does not include a learning data set output request, the influence score calculation unit 130 proceeds to step S508.
[0152] If it is determined that the learning data selection instruction includes a learning data set output request, the influence score calculation unit 130 instructs the learning data selection unit 131 to generate a learning data set (step S506).
[0153] When the learning data selection unit 131 receives an instruction from the influence score calculation unit 130, it selects learning data based on the influence score information 133. The selection method may be, for example, the following method.
[0154] (Selection method 1) The learning data selection unit 131 selects a learning condition, and refers to the column of the influence score 502 corresponding to the learning condition, and selects n pieces of learning data in ascending order of influence score. The learning data selection unit 131 executes the same process for all learning conditions. The learning data selection unit 131 generates a learning data set by removing the selected learning data from the learning data database 124. Here, n is assumed to be set in advance. However, n can be updated at any time.
[0155] (Selection Method 2) The learning data selection unit 131 selects a learning condition, and refers to the column of the influence score 502 corresponding to the learning condition to select learning data whose influence score is smaller than the first threshold. The learning data selection unit 131 executes the same process for all learning conditions. Based on the above-mentioned process result, the learning data selection unit 131 counts the number of times that the influence score becomes smaller than the first threshold for each learning data. The learning data selection unit 131 generates a learning data set by excluding learning data whose number of times is larger than the second threshold from the learning data database 124. It is assumed that the first threshold and the second threshold are set in advance. However, the first threshold and the second threshold can be updated at any time.
[0156] (Selection Method 3) The training data selection unit 131 generates a training data set by removing training data whose integrated influence score is smaller than a threshold from the training data database 124. Note that the threshold is assumed to be set in advance. However, the threshold can be updated at any time.
[0157] In addition, in selection method 1 and selection method 2, the influence score may be corrected using a weight set based on the prediction accuracy. In selection method 1 and selection method 2, the absolute value of the influence score may be used as the criterion. In selection method 3, the absolute value of the integrated influence score may be used as the criterion.
[0158] Next, the influence score calculation unit 130 outputs the generated learning data set via the computer 100-1 (step S507). After that, the influence score calculation unit 130 proceeds to step S508.
[0159] In step S508, the influence score calculation unit 130 determines whether or not the learning data selection instruction includes a predictor output request (step S508).
[0160] If it is determined that the learning data selection instruction does not include a predictor output request, the influence score calculation unit 130 proceeds to step S511.
[0161] If it is determined that the learning data selection instruction includes a predictor output request, the influence score calculation unit 130 instructs the learning data selection unit 131 to generate a learning data set (step S509). The process executed by the learning data selection unit 131 is the same as the process described in step S506.
[0162] Next, the influence score calculation unit 130 transmits an execution request for the learning process to the computer 100-2 together with the generated learning data set (step S510). After that, the influence score calculation unit 130 proceeds to step S511. Note that the execution request may include identification information of the selected learning data.
[0163] When the learning unit 121 of the computer 100-2 receives a request to execute a learning process, the learning unit 121 executes a learning process of the predictor 122 using the learning data set generated by the learning data selection unit 131. The learning unit 121 outputs the predictor 122 generated by the learning process using the learning data set via the computer 100-1.
[0164] In step S511, the influence score calculation unit 130 determines whether or not the learning data selection instruction includes a learning effect output request (step S511).
[0165] If it is determined that the learning data selection instruction does not include a learning effect output request, the influence score calculation section 130 ends the output information generation process.
[0166] If it is determined that the learning data selection instruction includes a learning effect output request, the influence score calculation unit 130 initializes a threshold value used to generate a learning data set (Step S512).
[0167] For example, in the case of selection method 1, the influence score calculation section 130 sets the variable n to 1.
[0168] Next, the influence score calculation unit 130 instructs the learning data selection unit 131 to generate a learning data set (step S513). The instruction includes threshold information. The process executed by the learning data selection unit 131 is the same as the process described in step S506.
[0169] Next, the influence score calculation unit 130 transmits an execution request for the learning process to the computer 100-2 together with the generated learning data set (step S514). Note that the execution request may include identification information of the selected learning data.
[0170] When the learning unit 121 of the computer 100-2 receives a request to execute a learning process, the learning unit 121 executes a learning process of the predictor 122 using a learning data set generated by the learning data selection unit 131. The learning unit 121 transmits the prediction accuracy of the predictor 122 generated by the learning process using the learning data set to the computer 100-3. The influence score calculation unit 130 records data in which the threshold value and the prediction accuracy are associated with each other.
[0171] Next, the influence score calculation unit 130 determines whether or not to end the verification (step S515). For example, the influence score calculation unit 130 determines whether or not the verification count is greater than a threshold. If the verification count is greater than the threshold, the influence score calculation unit 130 determines to end the verification. It is assumed that the verification count is set in advance. However, the verification count can be updated at any timing.
[0172] If it is determined not to end the verification, the influence score calculation unit 130 updates the threshold value (step S516), and then returns to step S513.
[0173] When it is determined that the verification is to be ended, the influence score calculation unit 130 generates learning effect information indicating the relationship between the threshold value and the prediction accuracy based on the data correlating the threshold value and the prediction accuracy, and outputs the learning effect information via the computer 100-1 (step S517). After that, the influence score calculation unit 130 ends the output information generation process.
[0174] For example, in the case of selection method 1, as shown in FIG. 14, an output screen 1400 is presented that displays a graph showing the relationship between the number n of data to be removed and prediction accuracy.
[0175] The computer 100-3 may cooperate with the computer 100-2 to output the learning history information 123. The user transmits a request to refer to the influence of an arbitrary learning condition via the presented learning history information 123. In this case, the computer 100-3 outputs a two-dimensional array 400 of the specified learning condition.
[0176] According to the first embodiment, the computer 100-3 can efficiently and easily select learning data that improves prediction accuracy based on the influence score. By using the influence score using the predictor 122 generated under different learning conditions, it is possible to make it dependent on a specific learning condition. By generating the predictor 122 using the selected learning data, it is possible to provide the user with the predictor 122 with high prediction accuracy.
[0177] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment describes the configuration in detail to easily explain the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.
[0178] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.
[0179] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, Java (registered trademark), etc.
[0180] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.
[0181] In the above-mentioned embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all the control lines and information lines are shown in the product. All the components may be connected to each other. [Explanation of symbols]
[0182] 100 calculator 101 Terminal 105 Network 110 Data Reception Department 111 Result output section 120 Learning condition setting section 121 Learning Department 122 Predictor 123 Learning History Information 123 124 Learning Data Database 130 Impact score calculation section 131 Learning Data Selection Unit 132 Impact information 133 Impact Score Information 140 Test Data Database 201 Processor 202 Main storage 203 Secondary storage device 204 Network Interface 600 GUI 700 API
Claims
1. A method for selecting training data to be used to generate a predictor, the method being executed by a computer system, comprising the steps of: The computer system comprises: a computer having a processor and a memory coupled to the processor, Connect to a database that stores multiple pieces of training data and information for managing multiple predictors generated under different training conditions; The method for selecting learning data includes the steps of: a first step in which the computer selects a target predictor from among the plurality of predictors, and calculates, for each of a plurality of test data, an influence degree that represents a strength of an influence of the learning data on the prediction accuracy of the target predictor for the test data; a second step in which the computer records the predictor and the degree of influence of the training data on each of the plurality of test data in association with each other; a third step of the computer calculating, for each of the plurality of predictors, an influence score of the learning data based on the plurality of influence degrees of the learning data associated with the predictor; a fourth step in which the computer records each of the plurality of predictors and the influence score of the training data in association with each other; a fifth step of generating a learning data set by the computer selecting the learning data to be used from among the plurality of learning data based on the plurality of influence scores of each of the plurality of learning data; A method for selecting learning data, comprising:
2. The method for selecting training data according to claim 1, The fifth step includes: a step of executing, for each of the plurality of predictors, a selection process of selecting the learning data to be excluded based on the influence score of each of the plurality of learning data associated with the predictor; a step of selecting, by the computer, the learning data to be used from among a plurality of the learning data based on a result of the selection process for each of the plurality of the predictors; A method for selecting learning data, comprising:
3. The method for selecting learning data according to claim 2, The method for selecting learning data, wherein the learning conditions include a plurality of the learning data used to generate the predictor and an input order of the plurality of the learning data.
4. The method for selecting learning data according to claim 3, The method for selecting training data, wherein the training conditions include hyperparameters of a training process for generating the predictor.
5. The method for selecting learning data according to claim 4, a sixth step in which the computer generates the predictor under different learning conditions; The sixth step includes: a seventh step of generating a batch by the computer selecting a predetermined number of the learning data from the database and determining an input order of the selected multiple learning data; an eighth step in which the computer sets the hyperparameters; a ninth step in which the computer selects a target batch from the plurality of batches, executes the learning process using the set hyperparameters and the plurality of learning data included in the target batch, and stores the plurality of learning data included in the target batch, the determined input order, the set hyperparameters, and the generated predictor in the information in association with each other; a tenth step in which the computer determines whether or not to continue learning based on a result of the learning process; an eleventh step of determining whether or not processing has been completed for all the batches when it is determined that learning should be continued; a twelfth step of executing the ninth step and the tenth step when the computer determines that processing has not been completed for all of the batches; A method for selecting learning data, comprising:
6. The method for selecting learning data according to claim 5, a step of providing an interface for inputting the number of predictors to be generated, the number of training data constituting the batch, and values for adjusting the hyperparameters,
7. 1. A computer system comprising a computer having a processor and a memory coupled to the processor, Connect to a database that stores multiple pieces of training data and information for managing multiple predictors generated under different training conditions; The computer includes: a first process of selecting a target predictor from among the plurality of predictors, and calculating, for each of a plurality of test data, an influence degree that represents a strength of an influence of the learning data on a prediction accuracy of the target predictor for the test data; a second process of recording the predictor and the degree of influence of the learning data on each of the plurality of test data in association with each other; a third process of calculating, for each of the plurality of predictors, an influence score of the learning data based on the plurality of influence degrees of the learning data associated with the predictor; a fourth process of recording each of the plurality of predictors and the influence score of the training data in association with each other; a fifth process of generating a learning data set by selecting the learning data to be used from among the plurality of learning data based on the plurality of influence scores of each of the plurality of learning data; A computer system that executes the above.
8. 8. The computer system of claim 7, In the fifth process, performing a selection process for selecting, for each of the plurality of predictors, the learning data to be excluded based on the influence score of each of the plurality of learning data associated with the predictor; a computer system for selecting the learning data to be used from among a plurality of the learning data based on a result of the selection process for each of a plurality of the predictors,
9. 9. The computer system of claim 8, The computer system according to claim 1, wherein the learning conditions include a plurality of the learning data used to generate the predictor and an input order of the plurality of the learning data.
10. 10. The computer system of claim 9, The computer system according to claim 1, wherein the learning conditions include hyperparameters of a learning process for generating the predictor.
11. 11. The computer system of claim 10, The computer executes a sixth process of generating the predictor under different learning conditions; and In the sixth process, the computer a seventh process of generating a batch by selecting a predetermined number of the learning data from the database and determining an input order of the selected plurality of the learning data; an eighth process of setting the hyperparameters; a ninth process of selecting a target batch from the plurality of batches, executing the learning process using the set hyperparameters and the plurality of learning data included in the target batch, and storing the plurality of learning data included in the target batch, the determined input order, the set hyperparameters, and the generated predictor in the information in association with each other; a tenth process of determining whether or not to continue learning based on a result of the learning process; an eleventh process of determining whether or not processing has been completed for all the batches when it is determined that learning is to be continued; a twelfth process of executing the ninth process and the tenth process when it is determined that the processes have not been completed for all the batches; A computer system that executes the above.
12. 12. The computer system of claim 11, The computer system is characterized in that the computer provides an interface for inputting the number of predictors to be generated, the number of training data constituting the batch, and values for adjusting the hyperparameters.
Citation Information
Patent Citations
Method for analyzing learning data and computing system
JP2020030738A
Learning data refining method and computer system
JP2021033544A
