Rewriting program, rewriting method, and information processing device

Abstract

To provide a rewriting program, a method and an apparatus for calculating a score for an existing machine learning program.SOLUTION: An information processing apparatus 10 includes: a storage unit 11 which stores a machine learning program 14 that uses a machine learning library 13; and a processing unit 12 which detects, on the basis of a module name of a module included in the machine learning library, a first code 14-1 for calling a training function for training a machine learning model from the machine learning program, inserts, before the first code, a second code 15-2 for splitting first training data passed to the training function with the first code, into second training data and test data, rewrites the first code to pass the second training data to the training function, and inserts, after the first code, a third code 15-3 for calculating a score on the basis of prediction data, which is generated from the trained machine learning model and input data included in the test data, and correct answer data included in the test data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a rewriting program, a rewriting method, and an information processing device. [Background technology]

[0002] A computer may perform machine learning to train a machine learning model using training data. The training data may include input data to be input to the machine learning model and ground truth data corresponding to predicted data output by the machine learning model. The computer optimizes parameter values ​​included in the machine learning model so as to reduce an error between predicted data generated from the input data and the ground truth data.

[0003] Users who want to use computers to perform machine learning may create machine learning programs that specify the machine learning procedures. In machine learning programs, values ​​for various adjustment items are selected, such as the method of preprocessing training data, the structure of the machine learning model, and hyperparameters that control the behavior of the machine learning algorithm. These adjustment items affect the predictive accuracy of the machine learning model. Therefore, even if the task of the machine learning model and the training data used are the same, the predictive accuracy may vary depending on the machine learning program.

[0004] An information processing device has been proposed that divides a hierarchical program into multiple program elements, performs machine learning based on the attributes and hierarchy of each program element, and classifies the multiple program elements into multiple groups. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 066074 Summary of the Invention [Problem to be solved by the invention]

[0006] A computer may want to calculate a score for an existing machine learning program. The score may be the predictive accuracy of a machine learning model trained by that machine learning program. For example, there is AutoML (Auto Machine Learning), which reduces the burden on the user by generating a machine learning program to achieve a task when the user specifies the task and training data for the machine learning model. A computer may want to tune AutoML using an existing machine learning program with a high score so that AutoML generates a high-quality machine learning program.

[0007] Here, existing machine learning programs are not necessarily written to automatically calculate scores. In this regard, it is conceivable that a user manually edits an existing machine learning program so that a score is calculated at runtime. However, understanding and manually editing an existing machine learning program places a heavy burden on the user. For example, when collecting a large number of existing machine learning programs for tuning AutoML, manually editing these large number of machine learning programs places a heavy burden on the user. Therefore, in one aspect, the present invention aims to calculate a score for an existing machine learning program. [Means for solving the problem]

[0008] In one embodiment, a rewriting program is provided that causes a computer to perform the following processes: Detecting first code that calls a training function for training a machine learning model from a machine learning program that uses the machine learning library based on module names of modules included in the machine learning library; Inserting second code before the first code that splits first training data passed to the training function by the first code into second training data and test data; Rewriting the first code to pass the second training data to the training function; Inserting third code after the first code that calculates a score based on predicted data generated from the trained machine learning model and input data included in the test data, and correct answer data included in the test data.

[0009] In one aspect, a computer-implemented rewriting method is provided.In another aspect, an information processing device having a storage unit and a processing unit is provided. [Effects of the Invention]

[0010] On the one hand, it can calculate scores against existing machine learning programs. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. [Figure 2] FIG. 10 illustrates an example of hardware of an information processing apparatus according to a second embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of a flow of machine learning using AutoML. [Figure 4] FIG. 10 is a diagram illustrating an example of an API document and a class name list. [Figure 5] FIG. 10 is a diagram illustrating an example of rewriting a machine learning program. [Figure 6] FIG. 10 is a diagram illustrating an example of an abstract syntax tree before rewriting. [Figure 7] FIG. 10 is a diagram illustrating an example of a rewritten abstract syntax tree. [Figure 8]FIG. 2 is a block diagram illustrating an example of functions of the information processing device. [Figure 9] 10 is a flowchart illustrating an example of a procedure for calculating a score. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present embodiment will be described below with reference to the drawings. [First embodiment] A first embodiment will be described.

[0013] FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. The information processing device 10 of the first embodiment rewrites a machine learning program so that a score is calculated at runtime. The information processing device 10 may rewrite each of multiple machine learning programs. Furthermore, the information processing device 10 may calculate a score for the machine learning program by executing the rewritten machine learning program. The information processing device 10 may tune AutoML using the machine learning program to which a score has been assigned. When a task and training data for a machine learning model are specified, AutoML generates a machine learning program for achieving the task. The information processing device 10 may be a client device or a server device. The information processing device 10 may be called a computer, a rewriting device, or a machine learning device.

[0014] The information processing device 10 has a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory such as a random access memory (RAM), or a non-volatile storage such as a hard disk drive (HDD) or a flash memory. The processing unit 12 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). However, the processing unit 12 may also include an electronic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor executes a program stored in a memory such as a RAM (which may be the storage unit 11). A set of processors may be called a multiprocessor or simply a "processor."

[0015] The storage unit 11 stores a machine learning program 14. The machine learning program 14 may include source code written in a high-level language, or may include compiled object code. The machine learning program 14 uses a machine learning library 13. The machine learning library 13 is a library program that includes one or more reusable modules that define typical processes in machine learning. By using the machine learning library 13, a user can efficiently create the machine learning program 14.

[0016] The machine learning library 13 may include modules corresponding to types of machine learning models or types of machine learning algorithms. For example, the machine learning library 13 includes modules corresponding to machine learning models such as neural networks, decision trees, and support vector machines. The modules may include functions, and the machine learning program 14 may call the functions of the modules. The modules may correspond to classes in an object-oriented language, and the functions may correspond to methods in an object-oriented language. The functions may include a training function that trains the machine learning model, and a prediction function that causes the machine learning model to output predicted data corresponding to input data.

[0017] The machine learning model may be an image recognition model that determines object classes from image data, or a natural language processing model that performs natural language processing on text data, such as named entity extraction, machine translation, synonym search, and response generation.

[0018] Processing unit 12 analyzes machine learning program 14. If machine learning program 14 does not include code for calculating a score, processing unit 12 rewrites machine learning program 14 to include code for calculating a score. As a result, machine learning program 14 is rewritten into machine learning program 15. Machine learning program 15 may include source code or object code. Note that machine learning library 13 and machine learning program 15 may be stored in storage unit 11.

[0019] The processing unit 12 detects, from the machine learning program 14, code 14-1 that calls a training function for training a machine learning model, based on the module name of the module included in the machine learning library 13. For example, the processing unit 12 identifies a variable indicating an instance of the module by searching the machine learning program 14 for a statement that includes the module name. The module name may be the name of a model module corresponding to the type of machine learning model, and the variable may be a model variable that indicates an instance of the model module.

[0020] Then, for example, the processing unit 12 detects code 14-1 that calls a training function of the instance indicated by the variable. The training function optimizes parameter values ​​included in the machine learning model so as to reduce the error between predicted data generated from input data and ground truth data. The function name of the training function may be specified in advance or may be specified based on the machine learning library 13. The training function may be a fit function.

[0021] To detect the code 14-1, the processing unit 12 may analyze a document that describes how to use the machine learning library 13 and extract a module name from the document. The processing unit 12 may detect the code 14-1 based on the module name extracted from the document. The document may be called an API (Application Programming Interface) document. The document may include module names of modules included in the machine learning library 13 and function names of functions included in the modules. Note that the "code" may include one or more instructions or one or more lines of statements.

[0022] The processing unit 12 inserts code 15-2 before code 14-1. For example, the processing unit 12 inserts code 15-2 immediately before code 14-1. Code 15-2 divides the first training data passed to the training function in code 14-1 into second training data and test data. Code 15-2 may call a division function defined in the machine learning library 13. For example, the first training data includes a plurality of records, each of which includes input data and correct answer data. The correct answer data may be referred to as ground truth, teacher label, or teacher data. In this case, code 15-2 sets some of the records of the first training data as second training data and the remaining records as test data. For example, code 15-2 sets 10% of the records included in the first training data as test data.

[0023] The processing unit 12 rewrites the code 14-1 so that the second training data is passed to the training function instead of the first training data. As a result, the code 14-1 is rewritten as the code 15-1. Note that in the code 15-2, the input data and the correct answer data may be divided into input data for the training data, input data for the test data, correct answer data for the training data, and correct answer data for the test data. In the code 15-1, the input data for the training data and the correct answer data for the training data may be input to the training function.

[0024] The processing unit 12 inserts code 15-3 after code 14-1. For example, the processing unit 12 inserts code 15-3 immediately after code 14-1. Code 15-3 calculates a score based on predicted data output by the trained machine learning model and correct answer data included in the test data. The score may indicate the prediction accuracy of the machine learning model, such as a correct answer rate. Code 15-3 may call a score function defined in the machine learning library 13. For example, in code 15-3, the predicted data output by the machine learning model and correct answer data for the test data are input to the score function.

[0025] To generate prediction data, the processing unit 12 may search for code that calls a prediction function from the machine learning program 14. For example, the processing unit 12 searches for code that calls a prediction function of an instance indicated by the same variable as the code 14-1. The prediction function may be a predict function. If the corresponding code is detected, the processing unit 12 may rewrite the detected code. If the corresponding code is not detected, the processing unit 12 may insert code that calls the prediction function. In the rewritten code or inserted code, input data for the test data may be input to the prediction function, and the prediction data output by the prediction function may be stored in a variable.

[0026] This generates machine learning program 15 including codes 15-1, 15-2, and 15-3. Processing unit 12 may store the generated machine learning program 15 in non-volatile storage, may display it on a display device, or may transmit it to another information processing device. Processing unit 12 may also execute machine learning program 15 to calculate a score. In this case, processing unit 12 may store the calculated score in association with machine learning program 14 or machine learning program 15. Processing unit 12 may also display the calculated score on a display device or may transmit it to another information processing device.

[0027] The processing unit 12 may perform syntax analysis of the machine learning program 14 to generate an abstract syntax tree (AST). The abstract syntax tree represents the relationships between tokens such as module names, function names, and variable names in a tree structure. The processing unit 12 may use the generated abstract syntax tree to perform some or all of the following: detection of code 14-1, insertion of code 15-2, rewriting of code 14-1, and insertion of code 15-3. The processing unit 12 may rewrite the abstract syntax tree and convert the rewritten abstract syntax tree into the machine learning program 15.

[0028] As described above, the information processing device 10 of the first embodiment detects code 14-1 that calls a training function from the machine learning program 14 based on the module name of a module included in the machine learning library 13. The information processing device 10 inserts code 15-2 before code 14-1 that divides first training data passed to the training function into second training data and test data. The information processing device 10 rewrites code 14-1 so that the second training data is passed to the training function. The information processing device 10 inserts code 15-3 after code 14-1 that calculates a score based on predicted data generated from the machine learning model and input data included in the test data, and correct answer data included in the test data.

[0029] As a result, even if the machine learning program 14 is not written to calculate a score, a score for the machine learning program 14 is automatically calculated by executing the generated machine learning program 15. This reduces the burden of manually editing the machine learning program 14. For example, to tune AutoML, which generates a machine learning program from the task and training data of a machine learning model, multiple machine learning program samples with assigned scores may be used. In this case, the burden of manually editing multiple machine learning programs is reduced. In addition, an appropriate score indicating the predictive accuracy of the machine learning model is calculated for the machine learning program 14.

[0030] The information processing device 10 may extract module names from a document that describes how to use the machine learning library 13, and may search for the extracted module names in the machine learning program 14. This allows the code 14-1 that calls the training function to be detected with high accuracy. The information processing device 10 may also perform a syntax analysis of the machine learning program 14 to generate an abstract syntax tree, and may use the abstract syntax tree to perform processes such as detecting and rewriting the code 14-1. This allows the information processing device 10 to perform the above processes with high accuracy, regardless of how the machine learning program 14 is written.

[0031] The information processing device 10 may also detect code that calls a prediction function from the machine learning program 14, and rewrite the code so that input data included in the test data is passed to the prediction function. At this time, the hyperparameter values ​​input to the prediction function do not need to be changed. This allows prediction data corresponding to the input data to be generated so as to be consistent with other code included in the machine learning program 14.

[0032] Furthermore, the module may be a class representing a machine learning model, and the training function may be a method included in the class. This allows for highly accurate detection of code 14-1 for training the machine learning model. Furthermore, the information processing device 10 may execute the machine learning program 15 using the first training data, and may store the calculated score in association with the machine learning program 14. This allows a score to be assigned to the machine learning program 14, facilitating other tasks such as tuning AutoML.

[0033] [Second embodiment] Next, a second embodiment will be described. An information processing device 100 according to the second embodiment collects samples of multiple machine learning programs and assigns a score to each machine learning program. The information processing device 100 uses the multiple machine learning programs to which the scores have been assigned to train an AutoML that generates a machine learning program suited to a task. The information processing device 100 then uses the trained AutoML to generate a machine learning program suited to a desired task. The information processing device 100 then executes the generated machine learning program to generate a machine learning model. However, the above processing may be performed by a different information processing device.

[0034] The information processing device 100 may be a client device or a server device. The information processing device 100 may also be called a computer or a machine learning device. The information processing device 100 corresponds to the information processing device 10 of the first embodiment.

[0035] FIG. 2 illustrates an example of hardware of an information processing apparatus according to the second embodiment. The information processing device 100 has a CPU 101, a RAM 102, a HDD 103, a GPU 104, an input interface 105, a medium reader 106, and a communication interface 107, all connected via a bus. The CPU 101 corresponds to the processing unit 12 in the first embodiment. The RAM 102 or the HDD 103 corresponds to the storage unit 11 in the first embodiment.

[0036] The CPU 101 is a processor that executes program instructions. The CPU 101 loads programs and data stored in the HDD 103 into the RAM 102 and executes the programs. The information processing device 100 may have multiple processors.

[0037] The RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the CPU 101 and data used in calculations by the CPU 101. The information processing device 100 may have a type of volatile memory other than a RAM.

[0038] The HDD 103 is a nonvolatile storage that stores software programs such as an operating system (OS), middleware, and application software, as well as data. The information processing device 100 may also have other types of nonvolatile storage, such as a flash memory or an SSD (Solid State Drive).

[0039] The GPU 104 performs image processing in cooperation with the CPU 101 and outputs an image to a display device 111 connected to the information processing device 100. The display device 111 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, or a projector. Other types of output devices, such as a printer, may be connected to the information processing device 100. The GPU 104 may also be used as a GPGPU (General Purpose Computing on Graphics Processing Unit). The GPU 104 may execute a program in response to an instruction from the CPU 101. The information processing device 100 may have a volatile semiconductor memory other than the RAM 102 as a GPU memory.

[0040] The input interface 105 receives an input signal from an input device 112 connected to the information processing device 100. The input device 112 is, for example, a mouse, a touch panel, or a keyboard. A plurality of input devices may be connected to the information processing device 100.

[0041] The medium reader 106 is a reading device that reads programs and data recorded on the recording medium 113. The recording medium 113 is, for example, a magnetic disk, an optical disk, or a semiconductor memory. Magnetic disks include flexible disks (FDs) and HDDs. Optical disks include compact discs (CDs) and digital versatile discs (DVDs). The medium reader 106 copies the programs and data read from the recording medium 113 to other recording media such as the RAM 102 or the HDD 103. The read programs may be executed by the CPU 101.

[0042] The recording medium 113 may be a portable recording medium. The recording medium 113 may be used to distribute programs and data. The recording medium 113 and the HDD 103 may also be referred to as computer-readable recording media.

[0043] The communication interface 107 communicates with other information processing devices via the network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or a router, or may be a wireless communication interface connected to a wireless communication device such as a base station or an access point.

[0044] Next, we will explain AutoML. FIG. 3 is a diagram illustrating an example of the flow of machine learning using AutoML. The information processing device 100 collects sample data for training AutoML. The sample data associates a dataset, a problem definition, a machine learning program, and a score for each of a plurality of machine learning programs.

[0045] A dataset is training data input to a machine learning program. A dataset includes multiple records, and each record includes values ​​of two or more columns. A dataset may be table data. Some of the two or more columns are used as explanatory variables, and the values ​​of the explanatory variables are used as input data. Some of the other columns are used as objective variables, and the values ​​of the objective variables are used as correct answer data.

[0046] The problem definition indicates the task of prediction using a machine learning model. For example, the problem definition specifies which of two or more columns contained in a dataset corresponds to the target variable that the machine learning model predicts. A machine learning program is a script program that specifies the machine learning procedure. A machine learning program is source code written in a high-level language. A machine learning program may use a machine learning library. A machine learning program may include specifying how to preprocess training data, specifying the structure of the machine learning model, and specifying hyperparameter values ​​for the machine learning algorithm.

[0047] A score is a measurement of the predictive accuracy of a machine learning model generated by a machine learning program. Examples of score indicators include accuracy, precision, recall, F-measure, mean squared error (MSE), root mean squared error (RMSE), and coefficient of determination. If the machine learning program is written to calculate a score for a trained machine learning model, the score calculated by the machine learning program is saved. On the other hand, if the machine learning program is not written to calculate a score for a trained machine learning model, the score calculated by rewriting the machine learning program, as described below, is saved.

[0048] The sample data holds, for machine learning program 33a, a dataset 31a, a problem definition 32a, a machine learning program 33a, and a score 34a in association with each other. The sample data also holds, for machine learning program 33b, a dataset 31b, a problem definition 32b, a machine learning program 33b, and a score 34b in association with each other. The sample data includes, for example, hundreds to tens of thousands of machine learning programs.

[0049] The information processing device 100 uses this sample data to train AutoML 35. AutoML 35 is a program generator that generates, from a dataset 31 and a problem definition 32, a machine learning program 33 that trains a machine learning model 36 for solving the problem. AutoML 35 is implemented as software including a program. AutoML 35 may select a preprocessing method for generating input data for the machine learning model 36 from the dataset 31. Preprocessing of the dataset 31 may include selecting columns to be used as explanatory variables, normalizing column values, and the like.

[0050] Additionally, AutoML 35 may select a structure of a machine learning model that is suitable for the problem definition 32. Additionally, AutoML 35 may select hyperparameter values ​​of the machine learning algorithm so that the prediction accuracy of the machine learning model 36 is high. AutoML 35 may generate a machine learning program 33 to perform hyperparameter search.

[0051] When training AutoML 35, the information processing device 100 refers to the scores assigned to each machine learning program in the sample data. For example, the information processing device 100 analyzes the correlation between the code included in the machine learning program and the score, and learns how to write code that will result in a high score. Furthermore, for example, the information processing device 100 uses machine learning programs with high scores, such as machine learning programs with scores exceeding a threshold, to learn how to write a preferred machine learning program. As a result, AutoML 35 generates a machine learning program 33 that improves the prediction accuracy of the machine learning model 36.

[0052] The machine learning program 33 is a model generator that trains the machine learning model 36 using the dataset 31. The machine learning program 33 is a script program that defines a machine learning procedure suitable for the dataset 31 and the problem definition 32. The information processing device 100 executes the machine learning program 33. The information processing device 100 may compile the machine learning program 33 and convert it into object code.

[0053] The machine learning program 33 inputs input data to the machine learning model 36, calculates the error between the predicted data output by the machine learning model 36 and the correct answer data, and optimizes the parameter values ​​of the machine learning model 36 so as to reduce the error. The trained machine learning model 36 generates and outputs predicted data from the input data.

[0054] Here, existing machine learning programs such as the machine learning programs 33a and 33b may not be written to calculate scores. On the other hand, when training AutoML 35, the scores of existing machine learning algorithms are referenced. Therefore, for machine learning programs for which scores are not automatically calculated, the information processing device 100 rewrites and executes the machine learning programs so that scores are automatically calculated. The information processing device 100 associates the calculated scores with the original machine learning programs and stores them. As a result, the collected machine learning programs are assigned scores and used for training AutoML 35.

[0055] FIG. 4 shows an example of an API document and a class name list. To rewrite the machine learning program, the information processing device 100 acquires the API document 131 of the machine learning library and generates a class name list 132 from the API document 131. If there are multiple machine learning libraries, the information processing device 100 generates the class name list 132 from the multiple API documents.

[0056] The machine learning library includes a plurality of classes defined by an object-oriented language. The plurality of classes includes model classes corresponding to types of machine learning models. For example, the machine learning library includes model classes representing machine learning models such as a classification neural network, a regression neural network, a classification decision tree, a regression decision tree, and a support vector machine.

[0057] A class includes one or more methods. For example, a model class includes a fit function that accepts input data and ground truth data as arguments and trains an instance of a machine learning model. For example, a model class includes a predict function that accepts input data as arguments and outputs prediction data generated by an instance of a machine learning model.

[0058] API document 131 is a document that indicates the interface of the machine learning library. API document 131 describes the class name and type of each of multiple classes. API document 131 also describes the method names of the methods that each class has. Information processing device 100 extracts the class names of model classes from API document 131 and adds the extracted class names to class name list 132. Information processing device 100 does not need to extract class names of classes other than model classes. When there are multiple API documents, class name list 132 may contain a mixture of class names extracted from different API documents.

[0059] FIG. 5 is a diagram illustrating an example of rewriting a machine learning program. The information processing device 100 refers to the above-mentioned class name list 132 and rewrites the machine learning program 133 into the machine learning program 134. While the machine learning program 133 does not calculate a score, the machine learning program 134 does calculate a score.

[0060] The first line of the machine learning program 133 contains a statement to generate an instance of a model class and assign it to a model variable. The third line of the machine learning program 133 contains a statement to call the fit function of the instance indicated by the model variable. At this time, the input data X and the correct answer data y are passed as arguments to the fit function. The fifth line of the machine learning program 133 contains a statement to call the predict function of the instance indicated by the model variable and assign the predicted data to the data variable. At this time, certain input data and a hyperparameter value are passed as arguments to the predict function.

[0061] First, the information processing device 100 searches the machine learning program 133 for a class name included in the class name list 132. When any class name in the class name list 132 is detected, the information processing device 100 identifies a code for generating an instance of a model class for the detected class name, and identifies a model variable to which the instance is assigned. Here, the model variable is identified from the first line of the machine learning program 133.

[0062] Next, the information processing device 100 detects, from the machine learning program 133, a code that calls the fit function of the instance indicated by the model variable, and identifies the input data and supervised data to be passed to the fit function. Here, the input data X and supervised data y are identified from the third line of the machine learning program 133.

[0063] Next, the information processing device 100 generates code for dividing the input data records into training and test records, and for dividing the correct answer data records into training and test records, and inserts the code immediately before the call of the fit function. For example, a split function defined in a machine learning library is used for dividing the data. The ratio of the number of training data records to the number of test data records is, for example, 9 to 1. Next, the information processing device 100 changes the input data passed to the fit function to the divided training input data, and changes the correct answer data passed to the fit function to the divided training correct answer data. This generates the third and fourth lines of the machine learning program 134.

[0064] Next, the information processing device 100 searches the machine learning program 133 for code that calls the predict function of the instance indicated by the model variable. If code that calls the predict function is detected, the information processing device 100 copies the detected code and inserts it immediately after the detected code. For the copied code, the information processing device 100 changes the input data passed to the predict function to the input data for testing divided above, and changes the data variable to which the predicted data is assigned to a new data variable. At this time, other arguments such as hyperparameter values ​​are carried over without being changed.

[0065] On the other hand, if no code calling the predict function is detected, the information processing device 100 generates code calling the predict function of the instance indicated by the model variable and inserts it immediately after the call to the fit function. The generated code passes the test input data divided above as an argument to the predict function, and assigns the predicted data to a new data variable not included in the machine learning program 133. If the predict function has other arguments, specific initial values ​​are assigned to the other arguments. Here, line 5 of the machine learning program 133 is copied, and line 7 of the machine learning program 134 is generated.

[0066] Next, the information processing device 100 generates code that compares the predicted data assigned to the data variable with the divided correct answer data for the test to calculate a score, and inserts the code immediately after the call to the predict function. The score is, for example, the accuracy rate. For example, a score function defined in a machine learning library is used to calculate the score. In this case, the generated code passes the predicted data and correct answer data as arguments to the score function. This generates the ninth line of the machine learning program 134.

[0067] Incidentally, even if substantially the same machine learning procedure is indicated, the description method may differ depending on the machine learning program. To flexibly accommodate such variations in description method, the information processing device 100 performs the above-described code detection and rewriting on an abstract syntax tree instead of directly on the source code text. When rewriting the machine learning program 133 into the machine learning program 134, the information processing device 100 performs syntactic analysis on the machine learning program 133 to generate an abstract syntax tree. The abstract syntax tree is tree-structured data that indicates the relationships between syntax elements such as expressions, variables, classes, methods, and operators.

[0068] The information processing device 100 performs pattern matching on the abstract syntax tree to detect a subtree that corresponds to a specific code. The information processing device 100 rewrites the code by rewriting the detected subtree. The information processing device 100 also inserts code by inserting a new subtree into the abstract syntax tree based on the position of the detected subtree. The information processing device 100 then converts the rewritten abstract syntax tree into source code.

[0069] FIG. 6 is a diagram showing an example of an abstract syntax tree before rewriting. Abstract syntax tree 140 corresponds to machine learning program 133. Figure 6 shows a portion of the entire abstract syntax tree 140 that corresponds to the third line of machine learning program 133. Node 141 indicates input data X, which is the first argument of the fit function. Node 142 indicates correct answer data y, which is the second argument of the fit function.

[0070] FIG. 7 is a diagram showing an example of the abstract syntax tree after rewriting. Abstract syntax tree 150 corresponds to machine learning program 134. Figure 7 shows the portion of abstract syntax tree 150 that corresponds to lines 3 and 4 of machine learning program 134. Abstract syntax tree 150 is generated by rewriting abstract syntax tree 140.

[0071] Node 151 indicates a data variable that stores records that have been split for training out of input data X. Node 152 indicates a data variable that stores records that have been split for testing out of input data X. Node 153 indicates a data variable that stores records that have been split for training out of correct answer data y. Node 154 indicates a data variable that stores records that have been split for testing out of correct answer data y. A subtree including nodes 151, 152, 153, and 154 is newly added to abstract syntax tree 150.

[0072] Node 155 indicates the input data X, which is the first argument of the split function. Node 156 indicates the correct answer data y, which is the second argument of the split function. The subtree including nodes 155 and 156 is newly added to abstract syntax tree 150.

[0073] Node 157 indicates training input data, which is the first argument of the fit function. Node 157 indicates the same data variable as node 151. Node 158 indicates training correct answer data, which is the second argument of the fit function. Node 158 indicates the same data variable as node 153. The subtree including nodes 157 and 158 is a rewrite of the subtree included in abstract syntax tree 140. Node 157 corresponds to node 141 in abstract syntax tree 140. Node 158 corresponds to node 142 in abstract syntax tree 140. Information processing device 100 rewrites nodes 141 and 142 to nodes 157 and 158.

[0074] Next, the functions and processing procedures of the information processing device 100 will be described. FIG. 8 is a block diagram illustrating an example of functions of the information processing device. The information processing device 100 has an API document storage unit 121, a program storage unit 122, and an AutoML storage unit 123. These storage units are implemented using, for example, RAM 102 or HDD 103. The information processing device 100 also has a class name list generation unit 124, a program rewriting unit 125, a score calculation unit 126, an AutoML training unit 127, a program generation unit 128, and a model generation unit 129. These processing units are implemented using, for example, CPU 101 or GPU 104 and a program.

[0075] The API document storage unit 121 stores one or more API documents that indicate the interfaces of one or more machine learning libraries. The API document storage unit 121 also stores a class name list that lists class names extracted from the one or more API documents. The program storage unit 122 stores multiple machine learning programs collected as samples. The program storage unit 122 also stores, in association with each machine learning program, a dataset to be input to the machine learning program, a problem definition that indicates the machine learning task, and the score of the trained machine learning model.

[0076] The AutoML storage unit 123 stores AutoML programs and data. AutoML generates a machine learning program from a dataset and a problem definition. The class name list generation unit 124 searches for a model class corresponding to the type of machine learning model from the API document stored in the API document storage unit 121 and extracts the class name of the model class. The class name list generation unit 124 adds the extracted class name to a class name list.

[0077] Program rewriting unit 125 extracts machine learning programs that do not include code that outputs a score from among the machine learning programs stored in program storage unit 122. Program rewriting unit 125 refers to the class name list stored in API document storage unit 121, rewrites the extracted machine learning programs, and passes them to score calculation unit 126. Program rewriting unit 125 may save the rewritten machine learning programs in non-volatile storage, may display them on display device 111, or may transmit them to another information processing device.

[0078] The score calculation unit 126 inputs the corresponding data set into the machine learning program stored in the program storage unit 122, executes the machine learning program, and calculates a score. The score calculation unit 126 stores the calculated score in association with the machine learning program.

[0079] However, for machine learning programs that do not include code for outputting a score, score calculation unit 126 receives the rewritten machine learning program from program rewriting unit 125. Score calculation unit 126 executes the rewritten machine learning program in place of the original machine learning program and calculates a score corresponding to the original machine learning program. Score calculation unit 126 may store the calculated score in non-volatile storage, may display it on display device 111, or may transmit it to another information processing device.

[0080] The AutoML training unit 127 trains the AutoML stored in the AutoML storage unit 123 using the dataset, problem definition, machine learning program, and score stored in the program storage unit 122. At this time, the AutoML training unit 127 analyzes the relationship between the code included in the machine learning program and the score, and detects code patterns that contribute to improving the score. In this way, the AutoML is trained to generate a high-quality machine learning program that generates a machine learning model with a high score.

[0081] The program generation unit 128 receives a dataset and a problem definition. The program generation unit 128 generates a machine learning program by inputting the dataset and the problem definition into AutoML stored in the AutoML storage unit 123. The program generation unit 128 passes the generated machine learning program to the model generation unit 129. The program generation unit 128 may store the generated machine learning program in non-volatile storage, may display it on the display device 111, or may transmit it to another information processing device.

[0082] The model generation unit 129 inputs a dataset into the machine learning program, executes the machine learning program, and generates a trained machine learning model. The model generation unit 129 may store the trained machine learning model in a non-volatile storage, display it on the display device 111, or transmit it to another information processing device.

[0083] FIG. 9 is a flowchart showing an example of a procedure for calculating a score. (S10) The class name list generation unit 124 extracts class names of model classes corresponding to the types of machine learning models from the API documentation of the machine learning library. The class name list generation unit 124 generates a class name list that lists the extracted class names.

[0084] (S11) Program rewriting unit 125 generates an abstract syntax tree by syntactically analyzing the machine learning program. The following steps S12 to S20 are performed on the abstract syntax tree. Note that steps S11 to S21 are performed for each of the multiple machine learning programs.

[0085] (S12) The program rewriting unit 125 searches the machine learning program for any one of the class names listed in the class name list. The program rewriting unit 125 identifies the model variable to which the instance of the model class is assigned.

[0086] (S13) The program rewriting unit 125 searches the machine learning program for code F that calls the fit function for the model variable identified in step S12. (S14) The program rewriting unit 125 identifies the data set to be passed to the fit function in the code F. The program rewriting unit 125 inserts, before the code F, code for dividing the data set into training data and test data.

[0087] (S15) The program rewriting unit 125 rewrites the argument of the code F to a data variable indicating the training data divided in step S14. (S16) The program rewriting unit 125 searches the machine learning program for code P that calls the predict function for the model variable identified in step S12.

[0088] (S17) The program rewriting unit 125 determines whether the machine learning program contains code P. If code P exists, the process proceeds to step S18. If code P does not exist, the process proceeds to step S19.

[0089] (S18) The program rewriting unit 125 copies the code P and inserts it after the code P. The program rewriting unit 125 rewrites the arguments of the copied code with data variables that indicate the input data included in the test data divided in step S14. The program rewriting unit 125 also rewrites the output variables to which the predicted data is assigned with new data variables that are different from the other variables. Then, the process proceeds to step S20.

[0090] (S19) The program rewriting unit 125 generates code that calls the predict function for the model variable identified in step S12, and inserts the code after the code F. In the generated code, the input data included in the test data is passed as an argument to the predict function, and the predicted data output by the predict function is assigned to a new data variable.

[0091] (S20) Program rewriting unit 125 generates code that calculates a score by comparing the predicted data output by the predict function with the correct answer data included in the test data divided in step S14. Program rewriting unit 125 inserts the generated code after the code of step S18 or step S19.

[0092] (S21) Program rewriting unit 125 converts the rewritten abstract syntax tree into source code of a machine learning program. Score calculation unit 126 calculates a score by executing the rewritten machine learning program using the stored dataset. Score calculation unit 126 stores the calculated score in association with the original machine learning program.

[0093] As described above, the information processing device 100 according to the second embodiment uses AutoML to select appropriate data preprocessing, model structure, and hyperparameter values ​​according to the dataset and problem definition, and generate a machine learning program. This reduces the trial and error required by a user to cause a computer to perform machine learning, thereby reducing the burden on the user.

[0094] The information processing device 100 also trains AutoML using samples of machine learning programs to which scores have been assigned. This trains AutoML to generate high-quality machine learning programs that generate machine learning models with high scores. The information processing device 100 also rewrites machine learning programs that are not written to calculate the scores of the trained machine learning models so that the scores are calculated, and executes the rewritten machine learning programs. This reduces the burden of manually editing machine learning programs and assigns appropriate scores to the machine learning programs.

[0095] Furthermore, the information processing device 100 extracts the class names of model classes from the API document and searches the machine learning program for the extracted class names to identify model variables. This allows the model variables to be identified efficiently and with high accuracy. The information processing device 100 also performs syntax analysis of the machine learning program to generate an abstract syntax tree, and searches for and rewrites code on the abstract syntax tree. This allows the code to be searched for and rewritten efficiently and with high accuracy, regardless of how the machine learning program is written.

[0096] The information processing device 100 also rewrites the machine learning program so that a portion of the input dataset is divided as test data and a score is calculated. This eliminates the need to prepare separate test data, and the score of the machine learning model is calculated within the range of the dataset collected in conjunction with the machine learning program. Furthermore, if the machine learning program includes code that calls a predict function, the information processing device 100 copies that code and rewrites it into code that generates the desired prediction data. This allows other arguments to be inherited, maintaining consistency with surrounding code. [Explanation of symbols]

[0097] 10. Information processing equipment 11 Storage section 12 Processing section 13 Machine Learning Libraries 14,15 Machine Learning Programs 14-1, 15-1, 15-2, 15-3 Code

Claims

1. Detecting a first code that calls a training function that trains a machine learning model from a machine learning program that uses the machine learning library based on a module name of a module included in the machine learning library; inserting, before the first code, second code for splitting first training data passed to the training function by the first code into second training data and test data; rewriting the first code to pass the second training data to the training function; After the first code, insert a third code that calculates a score based on predicted data generated from the trained machine learning model and input data included in the test data, and correct answer data included in the test data. A rewrite program that causes a computer to execute a process.

2. extracting the module name from a document that describes how to use the machine learning library; detecting the first code includes a process of searching the machine learning program for the module name extracted from the document; The rewriting program according to claim 1.

3. detecting the first code includes a process of generating an abstract syntax tree by performing a syntax analysis of the machine learning program, and detecting the first code using the abstract syntax tree; The rewriting program according to claim 1.

4. The insertion of the third code includes a process of detecting, from the machine learning program, a fourth code that calls a prediction function that generates the prediction data using the machine learning model, and rewriting the fourth code so as to pass the input data to the prediction function. The rewriting program according to claim 1.

5. The module is a class representing the machine learning model, and the training function is a method included in the class. The rewriting program according to claim 1.

6. and causing the computer to further execute a process of executing the rewritten machine learning program using the first training data and storing the calculated score in association with the machine learning program before the rewriting. The rewriting program according to claim 1.

7. Detecting a first code that calls a training function that trains a machine learning model from a machine learning program that uses the machine learning library based on a module name of a module included in the machine learning library; inserting, before the first code, second code for splitting first training data passed to the training function by the first code into second training data and test data; rewriting the first code to pass the second training data to the training function; After the first code, insert a third code that calculates a score based on predicted data generated from the trained machine learning model and input data included in the test data, and correct answer data included in the test data. A rewriting method in which processing is performed by a computer.

8. a storage unit that stores a machine learning program that uses the machine learning library; a processing unit that detects, from the machine learning program, a first code that calls a training function that trains a machine learning model based on a module name of a module included in the machine learning library, inserts, before the first code, a second code that divides first training data passed to the training function by the first code into second training data and test data, rewrites the first code to pass the second training data to the training function, and inserts, after the first code, a third code that calculates a score based on predicted data generated from the trained machine learning model and input data included in the test data, and correct answer data included in the test data; An information processing device having the above.

Images