Machine learning automated execution system, machine learning automated execution method, and program

Abstract

To facilitate additional registration and management of a new machine learning algorithm in a system which comprehensively manages and executes machine learning algorithms.SOLUTION: A machine learning automatic execution system of an embodiment includes a registration receiving unit, a verification dataset storage unit, a verification unit, a feedback information generation unit, a first display control unit, and an algorithm storage unit. The registration receiving unit receives registration of a machine learning algorithm. In the verification dataset storage unit, a verification dataset for use in operation verification of the machine learning algorithm is stored. The verification unit executes the machine learning algorithm by use of the verification dataset to perform the operation verification. The feedback information generation unit generates feedback information based on a result of the operation verification. The first display control unit controls display of the generated feedback information. The machine learning algorithm is stored in the algorithm storage unit when the result of the operation verification satisfies a specific criterion.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a machine learning automatic execution system, a machine learning automatic execution method, and a program.

Background Art

[0002] The development of machine learning (Artificial Intelligence: AI) technology has enabled the development of AI capable of advanced inferences such as image recognition, text summarization, and speech recognition. In the field of AI, a description for explaining input data is called an AI model (or simply a model), and the process of updating the model according to the input data is called machine learning (or simply learning). Learning is roughly classified into supervised learning, unsupervised learning, and reinforcement learning according to the characteristics of the problem. In particular, in supervised learning, the model is updated so that the model outputs a more desirable inference result for the input data.

[0003] There are various description methods for models, such as neural networks, support vector machines, decision trees, etc. Among the models called neural networks, there are also various description methods such as autoencoders, convolutional neural networks (CNNs), and recurrent neural networks (RNNs). Also, for each model, there are various algorithms for learning that model. Examples of algorithms include stochastic gradient descent, genetic algorithms, EM algorithms, etc., and it is also possible to combine them.

[0004] The no-free-lunch theorem states that there is no general-purpose algorithm that is optimal for all problems that AI can address. In other words, anyone trying to develop a valuable model must select the appropriate algorithm for each specific problem the AI ​​is working on. Therefore, developing an AI model requires experts with advanced knowledge and implementation skills. Furthermore, executing an algorithm generally requires externally provided settings called hyperparameters, and even with the same algorithm, the accuracy of the trained model can vary greatly depending on the hyperparameters. Even AI experts need to explore hyperparameters through trial and error, which adds a significant time cost to AI model development. Consequently, those who want to use AI to solve real-world problems face high costs, at least in terms of requiring experts for model development and the time cost involved, which ultimately hinders the adoption of AI.

[0005] To address the above problems, the concept of Automated Machine Learning (AutoML) has been proposed as an attempt to reduce costs by automating model development. AutoML aims to minimize reliance on AI experts and the costs of trial and error, and there is active development of technologies to automate the trial and error of hyperparameters and the comparative evaluation of multiple models. As a result of technological development, it is becoming possible for even those without specialized knowledge of AI to create models with simple operations. For example, methods have been proposed to create multiple models using various algorithms and automatically select the desired model by comparing their suitability. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 6444494 [Overview of the project] [Problems that the invention aims to solve]

[0007] The AutoML system executes multiple pre-registered algorithms based on user commands, and then performs hyperparameter searches for each to provide the optimal model. As machine learning theory advances, algorithm developers are constantly creating new algorithms, and it is essential that the latest algorithms are quickly available in the AutoML system. Furthermore, algorithms may be customized according to the challenges and requests of those seeking to utilize AI models, and it is also essential that customized algorithms can be added to the AutoML system.

[0008] However, a unified mechanism for adding and registering new algorithms to the AutoML system has not been adequately considered until now. To run various algorithms in a common environment, it was necessary to understand and comprehensively manage the parameters provided externally at runtime, the conditions required for execution, and the resources. Furthermore, depending on the characteristics of the algorithm, specific preprocessing steps may be combined and used, requiring knowledge of the necessary preprocessing steps. When adding algorithms to the AutoML system, even algorithm developers sometimes did not accurately understand the conditions required for execution, and detailed verification and program modifications were time-consuming. In short, the lack of a mechanism to automate verification and comprehensively manage the results hindered algorithm developers from adding various algorithms to the AutoML system.

[0009] The problem that this invention aims to solve is to provide an automated machine learning execution system, an automated machine learning execution method, and a program that can easily add and manage new machine learning algorithms in a system that comprehensively manages and executes machine learning algorithms. [Means for solving the problem]

[0010] The automated machine learning execution system of this embodiment includes a registration reception unit, a verification dataset storage unit, a verification unit, a feedback information generation unit, a first display control unit, and an algorithm storage unit. The registration reception unit accepts registrations of machine learning algorithms. The verification dataset storage unit stores verification datasets used for verifying the operation of the machine learning algorithms. The verification unit performs operation verification by executing the machine learning algorithm using the verification datasets stored in the verification dataset storage unit. The feedback information generation unit generates feedback information based on the results of the operation verification. The first display control unit controls the display of the generated feedback information. The algorithm storage unit stores the machine learning algorithms whose operation verification results satisfy specific criteria. [Brief explanation of the drawing]

[0011] [Figure 1] A functional block diagram showing an example of the functional configuration of the automated machine learning execution system S according to the first embodiment. [Figure 2] A flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the first embodiment. [Figure 3] A flowchart showing an example of the learning process according to the first embodiment. [Figure 4] A figure showing an example of a verification dataset VD according to the second embodiment. [Figure 5] A flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the second embodiment. [Figure 6A] A diagram showing an example of feedback information according to the second embodiment. [Figure 6B] A diagram showing an example of feedback information according to the second embodiment. [Figure 6C] A diagram showing an example of feedback information according to the second embodiment. [Figure 7] A diagram showing an example of the functional block of the verification unit 12A according to the third embodiment. [Figure 8]A flowchart showing an example of the process for verifying the operation of the machine learning algorithm according to the third embodiment. [Figure 9] A flowchart showing an example of the learning process according to the third embodiment. [Figure 10] A flowchart showing an example of the process for verifying the operation of the machine learning algorithm according to the fourth embodiment. [Figure 11] A diagram showing an example of the verification result comparison screen according to the fifth embodiment. [Figure 12] A flowchart showing an example of the process for verifying the operation of the machine learning algorithm according to the sixth embodiment. [Figure 13] A flowchart showing an example of the process for verifying the operation of the machine learning algorithm according to the seventh embodiment. [Figure 14] A flowchart showing an example of the process for verifying the operation of the machine learning algorithm according to the eighth embodiment.

Best Mode for Carrying Out the Invention

[0012] Hereinafter, the machine learning automatic execution system, the machine learning automatic execution method, and the program of the embodiment will be described with reference to the drawings.

[0013] The machine learning automatic execution system of the embodiment is an integrated management environment for integrally managing machine learning algorithms. In the following description, the case where the machine learning automatic execution system is realized by a machine learning algorithm integrated management device and a machine learning execution device will be described as an example. However, the machine learning automatic execution system may be realized by a single device or may be realized by three or more devices. Also, each function of one of the machine learning algorithm integrated management device and the machine learning execution device may be provided in the other device. A part of the functions of the machine learning algorithm integrated management device and the machine learning execution device may be integrated. That is, the machine learning algorithm integrated management device, the machine learning execution device, and combinations thereof are examples of the "machine learning automatic execution system".

[0014] <First Embodiment> [Overall Configuration] FIG. 1 is a functional block diagram showing an example of the functional configuration of a machine learning automatic execution system S according to the first embodiment. The machine learning automatic execution system S includes, for example, a machine learning algorithm integrated management device 1 and a machine learning execution device 100. The machine learning algorithm integrated management device 1 and the machine learning execution device 100 are communicably connected via a communication network NW. The communication network NW includes, for example, the Internet, WAN (Wide Area Network), LAN (Local Area Network), a provider device, a radio base station, and the like.

[0015] When registering a new machine learning algorithm, the machine learning algorithm integrated management device 1 automatically verifies and confirms that the minimum specifications are satisfied. The machine learning algorithm integrated management device 1 is operated, for example, by an algorithm developer D (hereinafter referred to as "developer D") who develops a machine learning algorithm. The machine learning execution device 100 performs learning processing using the machine learning algorithm for which the operation verification by the machine learning algorithm integrated management device 1 has been completed, and generates a desired learned model. The machine learning execution device 100 is operated, for example, by an application user U (hereinafter referred to as "user U") who uses the model.

[0016] [Machine Learning Algorithm Integrated Management Device] The machine learning algorithm integrated management device 1 includes, for example, a control unit 10, a communication interface 20, a display unit 30, an input interface 40, and a storage unit 50. The communication interface 20 is a NIC or the like for executing communication with the machine learning execution device 100 and the like.

[0017] The display unit 30 displays various types of information. For example, the display unit 30 displays a GUI (Graphical User Interface) that accepts various operations from developer D, and feedback information showing the results of operational verification of the machine learning algorithm. The display unit 30 can be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or a touch panel. The display unit 30 may be provided separately from the machine learning algorithm integrated management device 1 and may communicate with the machine learning algorithm integrated management device 1 to display various types of information. Furthermore, if the display unit 30 is implemented as a touch panel, it may also incorporate the functions of the input interface 40.

[0018] The input interface 40 receives various input operations from developer D and outputs an electrical signal indicating the content of the received input operation to the control unit 10. The input interface 40 can be implemented, for example, by a keyboard, mouse, touch panel, etc.

[0019] The control unit 10 includes, for example, a registration acceptance unit 11, a verification unit 12, a feedback information generation unit 13, and a display control unit 14. Each functional unit of the control unit 10 is realized by a computer such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) executing a program. Furthermore, some or all of the functional units of the control unit 10 may be realized by hardware such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array), or they may be realized by the cooperation of software and hardware. The program may be stored in advance in the storage unit 50 (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM and installed when the storage medium is mounted on a drive device.

[0020] The registration reception unit 11 accepts registration of new machine learning algorithms (algorithm implementations) entered by developer D via the input interface 40. This registration may include the machine learning algorithm and its corresponding machine learning task, input / output data format, computing device (CPU, GPU), definition of the execution environment (Docker® image, etc.), definition of hyperparameters, etc. The algorithm implementation may be one in which a virtual environment is defined simultaneously, such as a Docker image, or it may be just a Python file of the program to be executed, assuming that the execution environment is defined in the system, or it may be written in an interactive analysis environment such as Jupyter Notebook. Methods for registering the algorithm implementation include, for example, directly uploading the actual file, entering the file's storage location, or using a dedicated API (Application Programming Interface) such as Docker.

[0021] Examples of machine learning tasks include regression, classification, anomaly detection, object detection, and segmentation. Machine learning tasks may also be combinations of several of these. They may also specify applications such as sales forecasting, visual inspection, plant anomaly detection, person detection, and cancer cell extraction. If a desired machine learning task has already been added by another algorithm developer, it may be possible to select it. For this purpose, tasks and algorithms added by other algorithm developers may be displayed in a list on the display unit 30. The data formats handled by the target tasks may include, for example, tabular data (structured data), image data, time series data, and audio data.

[0022] The computing device may input the resources required for execution if they are known. It may also input the acceptable execution time. If there are no items that developer D intends to input, default values ​​or recommended values ​​may be displayed in the display unit 30. In addition, parameters that can be specified when executing the algorithm and adjustable parameter definitions may be input. Parameter definitions may be set by inputting via the command line, input via the GUI, or input via a file such as JSON or YAML format. When limiting the possible values ​​of a variable, it is possible to list the values ​​discretely or set a numerical interval.

[0023] The verification unit 12 uses the verification dataset stored in the verification dataset storage unit 51 (described later) to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification. This operational verification checks whether the minimum specifications required by the system are met (for example, input / output specifications such as whether the file input / output paths are correct). If no errors occur as a result of this operational verification, it is guaranteed that the machine learning algorithm meets the required specifications and will operate normally on the automated machine learning execution system S. In this case, the verification unit 12 stores the operationally verified machine learning algorithm in the algorithm storage unit 52 (described later).

[0024] On the other hand, if an error occurs as a result of the operational verification, the machine learning algorithm does not meet the required specifications and is not guaranteed to operate correctly on the automated machine learning execution system S. In this case, it is assumed that there are parts of the machine learning algorithm that need to be corrected. When an error occurs during the operational verification by the verification unit 12 in this way, the feedback information generation unit 13 generates feedback information (information to prompt correction of the machine learning algorithm) to notify the developer D that an error has occurred. The feedback information generation unit 13 estimates the cause of the error from the information of the verification results. The types of errors are expected to be incompatible data input / output formats, insufficient memory, timeout, mismatch of execution environment, etc. Error messages output by the program may also be used as feedback information.

[0025] The display control unit 14 controls the display of feedback information generated by the feedback information generation unit 13 on the display unit 30. The display control unit 14 also displays a GUI on the display unit 30 for receiving various inputs and instructions from the developer D. The display control unit 14 displays the feedback information on the display unit 30 and prompts the developer D to make corrections. Because the environment in which the developer D conducts development and the environment in which the machine learning automated execution system S executes the program are not always exactly the same, unexpected errors may occur. By automatically performing verification for registering the machine learning algorithm with the machine learning automated execution system S and receiving the feedback from it, it becomes possible to respond smoothly. The display control unit 14 is an example of the "first display control unit".

[0026] The storage unit 50 includes, for example, a verification dataset storage unit 51, an algorithm storage unit 52, and a metadata storage unit 53. The verification dataset storage unit 51 stores the verification dataset used for operational verification by the verification unit 12. The verification dataset is intended solely for confirming the normal operation of the machine learning algorithm and is a standard dataset (for example, RGB image data in JPEG format with a data size of 224 x 224 [px], hereinafter referred to as the "standard format dataset"). Note that the developer D may be allowed to add verification datasets. Multiple standard format verification datasets may be set. If multiple datasets are set, registration of the machine learning algorithm is permitted only if no errors occur in all data or if the criteria are met. For example, in the case of an image dataset, both a dataset consisting of JPEG images and a dataset consisting of PNG images may be set as standard format datasets.

[0027] The algorithm storage unit 52 stores machine learning algorithms that have been verified by the verification unit 12 and meet specific criteria (verified (accepted)). The metadata storage unit 53 stores various information (metadata) related to the operational verification by the verification unit 12. The storage unit 50 is, for example, a storage device such as an HDD (Hard Disk Drive), RAM (Random Access Memory), or flash memory. The storage unit 50 may also be implemented by other storage devices connected via a communication network NW, such as a NAS (Network Attached Storage) or an external storage server device.

[0028] [Machine learning execution device] The machine learning execution device 100 includes, for example, a control unit 110, a communication interface 120, a display unit 130, an input interface 140, and a storage unit 150. The communication interface 120 is a NIC or the like for performing communication with the machine learning algorithm integrated management device 1, etc.

[0029] The display unit 130 displays various types of information. For example, the display unit 130 displays a GUI that accepts various operations from the user U, or the results of machine learning. The display unit 130 can be, for example, a liquid crystal display, an organic EL display, or a touch panel. The display unit 130 may be provided separately from the machine learning execution device 100 and communicate with the machine learning execution device 100 to display various types of information. Also, if the display unit 130 is implemented as a touch panel, it may also incorporate the functions of the input interface 140.

[0030] The input interface 140 receives various input operations from user U and outputs an electrical signal indicating the content of the received input operation to the control unit 110. The input interface 140 can be implemented, for example, by a keyboard, mouse, touch panel, etc.

[0031] The control unit 110 includes, for example, an acquisition unit 111, a learning control unit 112, a preprocessing unit 113, a learning unit 114, and a display control unit 115. Each functional unit of the control unit 110 is realized by a computer such as a CPU or GPU executing a program. Furthermore, some or all of the functional units of the control unit 110 may be realized by hardware such as an LSI, ASIC, or FPGA, or by the cooperation of software and hardware. The program may be stored in advance in the storage unit 150 (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is mounted on a drive device.

[0032] The acquisition unit 111, in response to a machine learning execution request from user U input via the input interface 140, acquires the machine learning algorithm, metadata, etc., to be used for machine learning from the machine learning algorithm integrated management device 1 (algorithm storage unit 52, metadata storage unit 53) via the communication interface 120. The acquisition unit 111 also acquires the training dataset uploaded by user U via the input interface 140 and stores it in the training dataset storage unit 151, which will be described later.

[0033] The learning control unit 112 controls the execution of machine learning by the preprocessing unit 113 and the learning unit 114 during machine learning. For example, the learning control unit 112 controls the processing content of the preprocessing unit 113 and various parameters of machine learning by the learning unit 114, according to the results of operational verification by the machine learning algorithm integrated management device 1. For items not specified by the user U, the learning control unit 112 controls settings related to the overall execution of machine learning, such as the algorithm to be executed, the scheduling of execution, whether or not specific preprocessing is performed and the parameters when preprocessing is performed, and the computing resources to be allocated to machine learning execution. The learning control unit 112 is an example of a "control unit".

[0034] The preprocessing unit 113 performs preprocessing on the training dataset stored in the training dataset storage unit 151 based on the control of the learning control unit 112.

[0035] Based on the control of the learning control unit 112, the learning unit 114 uses machine learning algorithms stored in the algorithm memory unit 52 of the machine learning algorithm integrated management device 1 to train a model on a training dataset. The learning unit 114 creates a model using machine learning methods such as neural networks, support vector machines, and decision trees.

[0036] The display control unit 115 controls the display unit 130 to display information such as the evaluation results of the model generated by machine learning by the learning unit 114. The display control unit 115 also displays a GUI on the display unit 130 to accept various inputs and instructions from the user U. The display control unit 115 is an example of a "second display control unit".

[0037] The storage unit 150 includes, for example, a training dataset storage unit 151 and an evaluation result storage unit 152. The training dataset storage unit 151 stores the training dataset that is the target of processing by the preprocessing unit 113 and the learning unit 114. The evaluation result storage unit 152 stores the evaluation results of the model generated by machine learning by the learning unit 114. The storage unit 150 is, for example, a storage device such as an HDD, RAM, or flash memory. However, the storage unit 150 may also be implemented by other storage devices connected via a communication network NW, such as a NAS or an external storage server device.

[0038] [Process for verifying operation] The following describes the process flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 2 is a flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the first embodiment.

[0039] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S101). This registration may include the definition of the execution environment (such as a Docker image) and the definition of hyperparameters.

[0040] Next, the verification unit 12 uses the verification dataset stored in the verification dataset storage unit 51 to execute the machine learning algorithm that was registered by the registration acceptance unit 11 and perform operational verification (step S103).

[0041] If an error occurs as a result of the operational verification by the verification unit 12, the machine learning algorithm will not meet the required specifications and will not be guaranteed to operate normally on the automated machine learning execution system S. If an error occurs in the operational verification by the verification unit 12 in this way (step S105: YES), the feedback information generation unit 13 generates feedback information (information to prompt correction of the machine learning algorithm) to notify the developer D that an error has occurred (step S109). Next, the display control unit 14 displays the feedback information generated by the feedback information generation unit 13 on the display unit 30 (step S111). By checking the feedback information displayed on the display unit 30, the developer D can understand that the machine learning algorithm does not meet the specifications and can correct the machine learning algorithm. The developer D re-registers the corrected machine learning algorithm via the input interface 40, and thereafter the process from step S101 onwards is repeated.

[0042] On the other hand, if no errors occur as a result of the operational verification by the verification unit 12, it is guaranteed that the machine learning algorithm meets the required specifications and will operate normally on the automated machine learning execution system S. In this case, if no errors occur during the operational verification by the verification unit 12 (step S105: NO), the verification unit 12 registers the machine learning algorithm that has been verified (treated as passed) in the algorithm storage unit 52 (step S107). With this, the processing of this flowchart is completed.

[0043] Furthermore, even if the machine learning algorithm's operation verification is completed successfully, the verification results may be saved and presented to developer D. For example, the prediction accuracy, execution time, and amount of memory used when training and evaluating the model on the verification dataset may be displayed so that developer D can confirm whether the behavior is as intended.

[0044] [Learning Process] Next, we will explain the flow of the learning process performed by the machine learning execution device 100. Figure 3 is a flowchart of an example of the learning process according to the first embodiment.

[0045] First, the acquisition unit 111 receives a request from user U to execute machine learning via the input interface 140 (step S201). The acquisition unit 111 also acquires the training dataset uploaded by user U via the input interface 140 and stores it in the training dataset storage unit 151 (step S203). The training dataset may be stored in the training dataset storage unit 151 in advance. Alternatively, the training dataset may be collected from one or more external storage devices connected to the machine learning automated execution system S via a network.

[0046] Next, the acquisition unit 111 acquires the machine learning algorithms to be machine-learned from the machine learning algorithm integrated management device 1 (algorithm storage unit 52) ​​via the communication interface 120 in response to the received execution request, and the display control unit 115 displays a list of the acquired machine learning algorithms on the display unit 130 (step S205). For example, the acquisition unit 111 acquires all machine learning algorithms registered in the algorithm storage unit 52. The display unit 130 displays a list of the acquired machine learning algorithms and allows user U to select the algorithm to be executed (step S207). The displayed machine learning algorithms may be classified by machine learning task, or filtered by the machine learning task entered by user U. The display method may include sorting by priority, such as algorithms with a proven track record, or highlighting algorithms with high priority. A specific algorithm may be selected by default.

[0047] Furthermore, the display unit 130 may display the hyperparameters of the algorithm registered in the algorithm storage unit 52 and accept user U's specifications for each hyperparameter. User U may input numerical values ​​or strings, or select from a list of options. Default values ​​may be set in advance.

[0048] Next, the learning unit 114, based on the control of the learning control unit 112, performs model training on the training dataset using the machine learning algorithm specified by user U (step S209). The learning control unit 112 selects values ​​from the candidate values ​​that each hyperparameter of the machine learning algorithm can take, which are registered in the algorithm memory unit 52, sets at least one combination of all hyperparameters, and instructs the learning unit 114 to perform model training with each setting. The candidate values ​​that the hyperparameters can take may be candidates specified in advance by developer D, or they may be candidates generated each time based on other settings such as the machine learning task. Since the accuracy of a machine learning model changes depending on the hyperparameters even when using the same algorithm, the accuracy of models trained with multiple different hyperparameter settings may be compared. A strategy for hyperparameter search to obtain the optimal hyperparameters may be set. Strategies such as random search, grid search, and Bayesian optimization may be used.

[0049] Next, the display control unit 115 displays information such as the evaluation results of the model generated by machine learning by the learning unit 114 on the display unit 130 (step S211). User U can understand the information such as the model evaluation results by checking the display unit 130. This completes the processing of this flowchart.

[0050] In other words, the second display control unit (display control unit 115) displays a list of machine learning algorithms stored in the algorithm memory unit 52, and the learning unit 114 performs model training using the machine learning algorithm specified by user U from among the machine learning algorithms included in the displayed list. Alternatively, the learning unit 114 may select one or more machine learning algorithms from those stored in the algorithm memory unit 52 based on a machine learning task specified by user U, and perform model training using each of the selected machine learning algorithms. The second display control unit displays items of hyperparameters defined in the machine learning algorithms stored in the algorithm memory unit 52, and the learning unit 114 performs model training using the hyperparameter values ​​specified by the user for the displayed hyperparameter items. Alternatively, the learning unit 114 may select at least one hyperparameter setting defined in the machine learning algorithms stored in the algorithm memory unit 52, and perform model training using each of the selected hyperparameter settings.

[0051] According to the first embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it becomes possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, in addition to the machine learning algorithm implementation itself, it accepts setting information including machine learning tasks, hyperparameters and their domains, and computing resources, and can automatically verify and confirm that it meets the minimum specifications, making it easier for developers D to add and register new algorithms. In other words, it solves the problem of there being no way to accept algorithms newly proposed by developers D, or even if there is a way, the developer themselves must verify whether the algorithm meets the specifications for new registration.

[0052] <Second Embodiment> The second embodiment will now be described. The automated machine learning execution system S of the second embodiment has the same configuration as the automated machine learning execution system S of the first embodiment. In the automated machine learning execution system S of the second embodiment, the content of the verification dataset used during operation verification differs from that of the first embodiment. The automated machine learning execution system S of the second embodiment will now be described, focusing on the differences from the first embodiment.

[0053] Figure 4 shows an example of a verification dataset VD according to the second embodiment. As shown in Figure 4, the verification dataset VD includes multiple types of verification datasets. For example, the verification dataset VD includes a standard format dataset (Data 1) in addition to extended format verification data (Data 2 and Data 3) (hereinafter referred to as "extended format datasets"). An extended format dataset differs from the standard format dataset in at least one of its data formats. For example, Data 1 (standard format dataset) is an RGB image dataset in JPEG format with an image size of 224 × 224 [px]. On the other hand, Data 2 (extended format 1) is an RGB image dataset in TIFF format with an image size of 224 × 224 [px]. Data 3 (extended format 2) is an RGB image dataset in JPEG format with non-uniform image sizes.

[0054] Some image recognition algorithms, due to their specifications, can only handle specific resolutions. Furthermore, even algorithms capable of handling variable-size images through precise implementation may be designed for only specific resolutions due to implementation difficulty, and may be unable to process other image types. To address these situations, we will perform operational verification using multiple types of validation datasets.

[0055] [Process for verifying operation] The following describes the process flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 5 is a flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the second embodiment.

[0056] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S301).

[0057] Next, the verification unit 12 uses one of the multiple verification datasets stored in the verification dataset storage unit 51 to execute the machine learning algorithm that was registered by the registration acceptance unit 11 and perform operational verification (step S303).

[0058] Next, the verification unit 12 determines whether or not operational verification has been completed for all verification datasets stored in the verification dataset storage unit 51 (step S305). If the verification unit 12 determines that operational verification has not been completed for all verification datasets (step S305: NO), it uses one of the multiple types of verification datasets stored in the verification dataset storage unit 51 that has not yet undergone operational verification to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification (step S303), and then repeats the subsequent processing.

[0059] Meanwhile, if the verification unit 12 determines that the operational verification has been completed for all verification datasets (step S305: YES), it determines whether an error has occurred in any of the operational verifications for multiple types of verification datasets (step S307). If it determines that an error has occurred in the operational verification (step S307: YES), the feedback information generation unit 13 generates feedback information (step S311). Next, the display control unit 14 displays the feedback information generated by the feedback information generation unit 13 on the display unit 30 (step S313).

[0060] Figures 6A to 6C show an example of feedback information according to the second embodiment. For example, if the operation is verified using data 1 (standard format), data 2 (extended format 1), and data 3 (extended format 2) as shown in Figure 4, and no errors occur with data 1 (standard format) and data 2 (extended format 1) (pass), but an error occurs with data 3 (extended format 2) (fail), then a first feedback screen will be displayed on the display unit 30, showing the message "An error occurs when the image size is uneven. Please improve the implementation," as shown in Figure 6A. By checking this feedback screen, developer D can understand that the machine learning algorithm does not meet the specifications and can make corrections to the machine learning algorithm. Developer D then re-registers the corrected machine learning algorithm via the input interface 40, and the process from step S301 onwards is repeated thereafter.

[0061] Furthermore, as shown in Figure 6B, the feedback information may also show the validation results for each item in the validation dataset. Alternatively, as shown in Figure 6C, it may show an accuracy index value (e.g., F-score) for each validation dataset.

[0062] On the other hand, if the verification unit 12 determines that no errors have occurred during the operational verification (step S307: NO), it registers the machine learning algorithm that has been verified (treated as passed) in the algorithm storage unit 52 (step S309). With this, the processing of this flowchart is completed.

[0063] In other words, the verification dataset includes at least one standard format dataset and at least one extended format dataset. If the verification unit 12 finds that the operational verification using the standard format dataset is successful and the operational verification using the extended format dataset is unsuccessful, it estimates the cause of failure from the difference in data format between the standard format dataset and the extended format dataset. The feedback information generation unit 13 generates feedback information including the estimated cause of failure.

[0064] Furthermore, a standard format dataset is an image dataset unified by a specific file format, resolution, number of channels, and color depth, while an extended format dataset is an image dataset that differs in at least one of the file format, resolution, number of channels, color depth, and number of images, or an image dataset that differs in the number of images. The verification unit 12 may estimate the difference between the standard format dataset and the extended format dataset as the cause of failure.

[0065] According to the second embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it becomes possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, by performing operational verification using multiple types of verification datasets, it becomes possible to automatically verify and manage the execution conditions of new machine learning algorithms, which are difficult for even developer D to comprehensively grasp. In other words, it becomes easier to add machine learning algorithms, in the sense that the applicability of new machine learning algorithms to realistic datasets can be confirmed without increasing the burden on developer D.

[0066] As another example, when adding an algorithm for a regression task of tabular data, suppose we set a standard format dataset consisting only of real-valued variables. In this case, we may set extended format datasets that include missing values, categorical variables, text data, time data, or datasets with different sample sizes. Regression algorithms for tabular data can be categorized into those that can train and predict models even when missing values ​​are present, and those that cannot. For example, random forests and gradient boosting trees can directly handle data with missing values, but support vector machines and multilayer perceptrons (MLPs) cannot handle data with missing values ​​in their input, so they must be used in combination with methods such as missing value imputation.

[0067] In other words, the standard format dataset is a tabular dataset consisting only of real-valued variables, while the extended format dataset is a tabular dataset containing at least one of categorical variables, text data, time data, and missing values, or a tabular dataset with a different sample size. The validation unit 12 estimates the difference between the standard format dataset and the extended format dataset as the cause of failure.

[0068] When setting up an extended format dataset, the differences between the extended format dataset and the standard format dataset are saved. If, as a result of the verification, no errors occur in the standard format dataset but errors occur in the extended format dataset, the feedback information generation unit 13 estimates the difference as the cause of the error and can provide feedback to developer D through the display unit 30. Developer D can then add preprocessing or other steps to the algorithm so that the algorithm can be used with the extended format data as well.

[0069] <Third Embodiment> The third embodiment will now be described. The machine learning automated execution system S of the third embodiment differs from that of the first embodiment in the processing content during operation verification. The machine learning automated execution system S of the third embodiment will now be described, focusing on the differences from the first embodiment.

[0070] Figure 7 shows an example of a functional block of the verification unit 12A provided in the machine learning algorithm integrated management device 1 according to the third embodiment. As shown in Figure 7, the verification unit 12A includes, for example, a verification data editing unit 12-1 and a metadata registration unit 12-2.

[0071] The verification data editing unit 12-1 edits the verification dataset (data 1, standard format) stored in the verification dataset storage unit 51 to generate data in which at least one item of the data format differs from that of data 1. For example, the verification data editing unit 12-1 edits data 1 (standard format, RGB image data in JPEG format with an image size of 224 x 224 [px]) to generate data 2, which is RGB image data in TIFF format with an image size of 224 x 224 [px]. The verification data editing unit 12-1 also edits data 1 to generate data 3, which is RGB image data in JPEG format with non-uniform image sizes.

[0072] Based on the results of operational verification using multiple types of verification data as described above, the metadata registration unit 12-2 registers metadata defining the preprocessing to be applied during machine learning execution, associating it with the machine learning algorithm, in the metadata storage unit 53. The metadata may be represented in, for example, JSON format.

[0073] [Process for verifying operation] The following describes the process flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 8 is a flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the third embodiment.

[0074] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S401).

[0075] Next, the verification data editing unit 12-1 edits the verification dataset (data 1, standard format) stored in the verification dataset storage unit 51 to generate data in which at least one item of the data format differs from that of data 1 (step S403).

[0076] Next, the verification unit 12 uses one of the multiple verification datasets generated by the verification data editing unit 12-1 to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification (step S405).

[0077] Next, the verification unit 12 determines whether or not all operational verification of the multiple types of verification datasets has been completed (step S407). If the verification unit 12 determines that all operational verification of the multiple types of verification datasets has not been completed (step S407: NO), it uses one of the multiple types of verification datasets whose operational verification has not been performed to execute the machine learning algorithm that was registered by the registration acceptance unit 11 and perform operational verification (step S405), and then repeats the subsequent processing.

[0078] If the verification unit 12 determines that all operational verifications of the multiple types of verification datasets have been completed (step S407: YES), it determines whether or not an error has occurred in any of the operational verifications of the multiple types of verification datasets (step S409). If it determines that an error has occurred in the operational verification (step S409: YES), the metadata registration unit 12-2 generates metadata that defines the preprocessing to be applied when machine learning is executed, based on the results of the operational verification using the multiple types of verification data, and registers it in the metadata storage unit 53 (step S413).

[0079] For example, if an operational verification is performed using each of the above data 1 (standard format), data 2 (edited data), and data 3 (edited data), and no errors occur with data 1 (standard format) and data 2 (edited data), but errors occur with data 3 (edited data), the metadata registration unit 12-2 generates metadata indicating that preprocessing to adjust (uniformize) the image size is necessary based on the difference between data 1 (no errors) and data 3 (with errors), namely "uneven image size," and registers this metadata (metadata indicating that no change in file format is necessary) in the metadata storage unit 53.

[0080] On the other hand, if the verification unit 12 determines that no errors have occurred during the operational verification (step S409: NO), or after the metadata registration by the metadata registration unit 12-2 is completed, it registers the operationally verified machine learning algorithm in the algorithm storage unit 52 (step S411). With this, the processing of this flowchart is completed.

[0081] [Learning Process] Next, we will explain the flow of the learning process performed by the machine learning execution device 100. Figure 9 is a flowchart showing an example of the learning process according to the third embodiment.

[0082] First, the acquisition unit 111 receives a request from user U to execute machine learning via the input interface 140 (step S501). The acquisition unit 111 also acquires the training dataset uploaded by user U via the input interface 140 and stores it in the training dataset storage unit 151 (step S503).

[0083] Next, the acquisition unit 111 acquires the machine learning algorithms and metadata to be machine-learned from the machine learning algorithm integrated management device 1 (algorithm storage unit 52, metadata storage unit 53) via the communication interface 120 in response to the received execution request, and the display control unit 115 displays a list of acquired machine learning algorithms on the display unit 130 (step S505). The display unit 130 displays a list of acquired machine learning algorithms and allows the user U to select the algorithm to be executed (step S507). Alternatively, the display unit 130 may display a list of machine learning tasks and allow the user U to select a machine learning task, and then use one or more machine learning algorithms stored in the algorithm storage unit 52 corresponding to the selected machine learning task to perform training on the following model.

[0084] Next, the preprocessing unit 113 performs preprocessing on the training dataset stored in the training dataset storage unit 151 based on the metadata acquired by the acquisition unit 111, based on the control of the learning control unit 112 (step S509). For example, if the metadata associated with the machine learning algorithm indicates that preprocessing to adjust (uniformize) the image size is necessary, the preprocessing unit 113 performs image size adjustment (uniformization) on the training dataset as preprocessing.

[0085] Furthermore, multiple preprocessing methods may be set for converting an extended format dataset to a standard format dataset. For example, in a machine learning task for image recognition, for the preprocessing of converting a rectangular image to a square image, a preprocessing method that changes the aspect ratio may be applied, or a process that crops the image may be performed. In a regression task for tabular data, for the task of imputing missing values, median imputation may be used, imputation may be performed using the K-nearest neighbors method, or imputation may be performed using the multiple imputation method. If multiple preprocessing methods are set, the machine learning execution device 100 may execute each process, adopt the one with the better evaluation result, and display the result on the display unit 130.

[0086] Next, the learning unit 114, based on the control by the learning control unit 112, performs model training on the pre-processed training dataset by the pre-processing unit 113 using the machine learning algorithm specified by the user U (step S511). Since the pre-processed training dataset used for this training has undergone image size adjustment (uniformization), no errors occur when the machine learning algorithm is executed.

[0087] Next, the display control unit 115 displays information such as the evaluation results of the model generated by machine learning by the learning unit 114 on the display unit 130 (step S513). User U can understand the information such as the model evaluation results by checking the display unit 130. This completes the processing of this flowchart.

[0088] In other words, the validation dataset includes at least one standard format dataset and at least one extended format dataset, and further comprises a metadata registration unit 12-2 that stores items in the extended format dataset in the metadata storage unit 53 as preprocessing to be applied when machine learning is executed if the operational verification using the standard format dataset is successful and the operational verification using the extended format dataset is unsuccessful, and a preprocessing unit 113 that refers to the metadata storage unit 53 and performs processing to convert the training dataset requiring preprocessing into the standard format.

[0089] According to the third embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it becomes possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, the automated machine learning execution system S can actively control the preprocessing required for machine learning execution that the developer D was unaware of when registering a new machine learning algorithm. As a result, the developer D does not need to be aware of the detailed specifications of the execution environment, and the burden related to registering machine learning algorithms is reduced. In addition, it is possible to prevent unnecessary preprocessing from being performed.

[0090] <Fourth Embodiment> The fourth embodiment will now be described. The machine learning automated execution system S of the fourth embodiment differs from that of the third embodiment in the processing content during operation verification. The machine learning automated execution system S of the fourth embodiment will now be described, focusing on the differences from the third embodiment.

[0091] The verification data editing unit 12-1 edits the verification dataset (Data 1, standard format) stored in the verification dataset storage unit 51 to generate data with a different data structure (number of normal and abnormal data) from Data 1. For example, the verification data editing unit 12-1 edits Data 1 (image classification dataset, 500 normal data, 500 abnormal data) to generate Data 2 (image classification dataset, 5,000 normal data, 500 abnormal data).

[0092] [Process for verifying operation] The following describes the process flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 10 is a flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the fourth embodiment.

[0093] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S601).

[0094] Next, the verification data editing unit 12-1 edits the verification dataset (data 1, standard format) stored in the verification dataset storage unit 51 to generate data with a different data structure (number of normal and abnormal data) from data 1 (data in which the number of normal and abnormal data is unbalanced) (step S603).

[0095] Next, the verification unit 12 uses one of the multiple verification datasets generated by the verification data editing unit 12-1 to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification (step S605).

[0096] Next, the verification unit 12 determines whether or not all operational verification of the multiple types of verification datasets has been completed (step S607). If the verification unit 12 determines that all operational verification of the multiple types of verification datasets has not been completed (step S607: NO), it uses one of the multiple types of verification datasets whose operational verification has not been performed to execute the machine learning algorithm that was registered by the registration acceptance unit 11 and perform operational verification (step S605), and then repeats the subsequent processing.

[0097] On the other hand, if the verification unit 12 determines that all operational verifications of multiple types of verification datasets have been completed (step S607: YES), it determines whether there are any verification results in which the accuracy (e.g., F-score) of the operational verifications on multiple types of verification datasets is lower than a predetermined threshold (step S609). If it is determined that there are verification results in which the accuracy is lower than a predetermined threshold (step S609: YES), the metadata registration unit 12-2 generates metadata that defines the preprocessing to be applied when machine learning is executed based on the results of the operational verifications using multiple types of verification data, and registers it in the metadata storage unit 53 (step S613).

[0098] For example, if operational verification is performed using Data 1 (standard format) and Data 2 (edited data) as described above, and the accuracy of Data 1 (standard format) exceeds a predetermined threshold, but the accuracy of Data 2 (edited data) is lower than the predetermined threshold, the metadata registration unit 12-2 generates metadata indicating that preprocessing to balance the data structure is necessary, based on the difference between Data 1 and Data 2, which is "data with an imbalance in the number of normal data and abnormal data," and registers it in the metadata storage unit 53. In this way, machine learning algorithms that are not designed to handle imbalanced datasets may experience a significant decrease in prediction accuracy. In such cases, by setting preprocessing to balance the data structure, the machine learning execution device 100 can solve this problem by adding preprocessing to thin out normal data through undersampling, thereby reducing the effort required for algorithm developers to make modifications.

[0099] On the other hand, if the verification unit 12 determines that there are no verification results in which the accuracy is lower than a predetermined threshold during the operational verification (step S609: NO), or after the metadata registration by the metadata registration unit 12-2 is completed, it registers the operationally verified machine learning algorithm in the algorithm storage unit 52 (step S611). With this, the processing of this flowchart is completed. In the learning process executed afterward, preprocessing is performed to balance the data structure of the training data based on the metadata. This makes it possible to perform appropriate learning processing while avoiding a decrease in accuracy.

[0100] In other words, the validation dataset includes at least one standard format dataset and at least one extended format dataset. The validation unit 12 executes a machine learning algorithm to perform a model training accuracy test. If the accuracy test using the standard format dataset meets the accuracy criteria, but the accuracy test using the extended format dataset does not, the validation unit 12 estimates the cause of failure based on the differences in data properties between the standard format dataset and the extended format dataset. The feedback information generation unit 13 then generates feedback information including the estimated cause of failure.

[0101] Furthermore, the standard format dataset is an image dataset that takes image data consisting only of normalized luminance values ​​as input and has no bias in the values ​​or labels of the target variable, while the extended format dataset is one of the following: the input images contain outliers, the target variable has a bias in values ​​or labels, or the number of images differs. The verification unit 12 estimates the difference between the standard format dataset and the extended format dataset as the cause of failure.

[0102] Furthermore, the standard format dataset consists only of normalized real-valued variables and is a tabular dataset with no bias in the values ​​or labels of the target variable, while the extended format dataset contains outliers in the explanatory variables, has bias in the values ​​or labels of the target variable, or has different sample sizes, and the validation unit 12 estimates the differences between the standard format dataset and the extended format dataset as reasons for failure.

[0103] Furthermore, the validation dataset includes at least one standard format dataset and at least one extended format dataset. The validation unit 12 executes a machine learning algorithm to perform a model learning accuracy test. If the accuracy test using the standard format dataset satisfies the accuracy criteria, but the accuracy test using the extended format dataset does not, the validation unit 12 further includes a metadata registration unit 12-2 that stores items in the extended format dataset in the metadata storage unit 53 as preprocessing to be applied when machine learning is executed. The validation unit 12 also includes a preprocessing unit 113 that refers to the metadata storage unit 53 and performs a process to convert datasets requiring preprocessing to the standard format.

[0104] According to the fourth embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it becomes possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, the automated machine learning execution system S can actively control the preprocessing required for machine learning execution that the developer D was unaware of when registering a new machine learning algorithm. As a result, the developer D does not need to be aware of the specifications of the execution environment in detail, and the burden related to registering machine learning algorithms is reduced. In addition, it is possible to prevent unnecessary preprocessing from being performed.

[0105] <Fifth Embodiment> The fifth embodiment will now be described. The machine learning automated execution system S of the fifth embodiment differs from that of the first embodiment in the processing content during operation verification. The machine learning automated execution system S of the fifth embodiment will now be described, focusing on the differences from the first embodiment.

[0106] In the fifth embodiment of the automated machine learning execution system S, the feedback information generation unit 13 of the machine learning algorithm integration management device 1 generates comparison result information to present to the developer D, comparing the results of operational verification of an existing machine learning algorithm with the results of operational verification of a new machine learning algorithm, in addition to or instead of providing feedback information to notify that an error has occurred. The display control unit 14 displays the comparison result information on the display unit 30.

[0107] Figure 11 shows an example of a verification result comparison screen according to the fifth embodiment. As shown in Figure 11, a comparison of the operational verification results of existing machine learning algorithms (A, B) and the operational verification results of the new machine learning algorithm is presented with respect to "accuracy," "execution time," and "GPU memory usage." Developer D can review this verification result comparison screen and then decide whether or not to register the new machine learning algorithm based on its effectiveness.

[0108] As described above, the machine learning algorithm integrated management device 1 may allow the registration of multiple machine learning algorithms for a single machine learning task. When developer D registers a new algorithm, if other algorithms for the same type of machine learning task are already registered, the device may perform verification on those algorithms as well, and the results may be compared. The verification results may also be stored in the memory unit 50, and if verification results have already been stored, those results may be used.

[0109] Furthermore, when comparing the results with those of other algorithms, one or more criteria regarding accuracy, execution time, computational resources, etc., may be established, and new machine learning algorithms may only be registered if they meet the criteria. Alternatively, when comparing the results with those of other algorithms, information regarding accuracy, execution time, computational resources, etc., may be presented to developer D, and the algorithm developer may be allowed to choose whether or not to register the new machine learning algorithm.

[0110] In other words, the verification unit 12 compares the results of the operational verification of the machine learning algorithm with the results of the operational verification of other similar machine learning algorithms, with respect to at least one of the following: accuracy, execution time, and computing resources.

[0111] According to the fifth embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it is possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, by presenting a verification result comparison screen to the developer D, it is possible to judge the effectiveness of the new machine learning algorithm and decide whether or not to register it. In other words, it is possible to decide whether or not to register a new machine learning algorithm based on a relative evaluation with other algorithms. This makes it possible to support algorithm developers who have developed a new machine learning algorithm in deciding whether or not to register that algorithm with the integrated machine learning algorithm automated machine learning execution system S, and to prevent algorithms that cannot be expected to perform relatively well from being registered unnecessarily.

[0112] <Sixth Embodiment> The sixth embodiment will now be described. The automated machine learning execution system S of the sixth embodiment has the same configuration as the automated machine learning execution system S of the third embodiment. The difference between the automated machine learning execution system S of the sixth embodiment and the third embodiment is that, during operational verification, the verification dataset that resulted in errors is standardized and operational verification is performed again, thereby presenting the developer D with a method for resolving the errors (content of metadata). The automated machine learning execution system S of the sixth embodiment will now be described, focusing on the differences from the third embodiment.

[0113] [Process for verifying operation] The following describes the processing flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 12 is a flowchart showing an example of the processing for verifying the operation of a machine learning algorithm according to the sixth embodiment. The explanation will use the example where the verification dataset storage unit 51 stores data 1 (standard format) and data 3 (extended format 2) as shown in Figure 4.

[0114] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S701).

[0115] Next, the verification unit 12 uses one of the multiple verification datasets stored in the verification dataset storage unit 51 to execute the machine learning algorithm that was registered by the registration acceptance unit 11 and perform operational verification (step S703).

[0116] Next, the verification unit 12 determines whether or not an error has occurred as a result of the operational verification (step S705). If the verification unit 12 determines that an error has occurred in the operational verification (step S705: YES), the verification data editing unit 12-1 standardizes the verification dataset that has an error (step S713) and performs the operational verification again (step S703). Standardization means changing the data format to a predetermined standard format. For example, if data 3 (extended format 2) has an error, the verification data editing unit 12-1 unifies the image size (standardizes) in order to make data 3 a standard format.

[0117] Next, the verification unit 12 determines whether or not operational verification has been completed for all verification datasets stored in the verification dataset storage unit 51 (step S707). If the verification unit 12 determines that operational verification has not been completed for all verification datasets (step S707: NO), it uses one of the multiple types of verification datasets stored in the verification dataset storage unit 51 that has not yet undergone operational verification to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification (step S703), and then repeats the subsequent processing.

[0118] Meanwhile, if the verification unit 12 determines that it has completed operational verification on all verification datasets stored in the verification dataset storage unit 51 (step S707: YES), it determines whether an error has occurred in any of the operational verifications on the multiple types of verification datasets (step S709). If it determines that an error has occurred in the operational verification (step S709: YES), the feedback information generation unit 13 generates feedback information (step S715). Next, the display control unit 14 displays the feedback information generated by the feedback information generation unit 13 on the display unit 30 (step S717). If developer D confirms the feedback information and inputs an instruction to permit registration, the metadata registration unit 12-2 generates metadata defining the preprocessing to be applied when machine learning is executed and registers it in the metadata storage unit 53 (step S719).

[0119] For example, during operational verification, if data 3 (extended format 2) results in an error, and the error is resolved as a result of standardizing data 3 (unifying the image size), the feedback information will include content indicating how the error was resolved, such as, "An error occurred with data that had uneven image sizes, but this can be resolved by standardizing the data."

[0120] On the other hand, if it is determined that no errors have occurred during the operational verification (step S709: NO), or if the metadata registration is complete, the verification unit 12 registers the operationally verified machine learning algorithm in the algorithm storage unit 52 (step S711). With this, the processing of this flowchart is completed.

[0121] Furthermore, even if there are no problems running the extended format dataset, the verification data editing unit 12-1 may perform the process of converting the extended format to a standard format dataset, and the results in that case may be displayed on the display unit 30. This allows for confirmation of side effects caused by uniformly applying preprocessing. Also, if the verification result is that "there are no functional problems even without preprocessing, but performance improves with preprocessing," then that machine learning algorithm may be treated as requiring that preprocessing.

[0122] In other words, the verification data editing unit 12-1 further includes a standardization process that converts an extended format dataset into a standard format dataset, and the verification unit 12 performs operational verification using the standardized dataset.

[0123] According to the sixth embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it is possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, error resolution methods (contents of preprocessing) through standardization can be presented to the developer D for confirmation. This allows the developer D to confirm that preprocessing not intended by them is not performed automatically.

[0124] <Seventh Embodiment> The seventh embodiment will now be described. The machine learning automated execution system S of the seventh embodiment has the same configuration as the machine learning automated execution system S of the third embodiment. The machine learning automated execution system S of the seventh embodiment differs from the third embodiment in that it changes the pattern of computer resources (CPU, GPU, etc.) (hereinafter referred to as the "verification resource pattern") during operation verification. The machine learning automated execution system S of the seventh embodiment will now be described, focusing on the differences from the third embodiment.

[0125] [Process for verifying operation] The following describes the process flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 13 is a flowchart showing an example of the process for verifying the operation of a machine learning algorithm according to the seventh embodiment.

[0126] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S801).

[0127] Next, the verification unit 12 uses one of several predefined verification resource patterns to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification (step S803). For example, the verification unit 12 uses verification resource pattern 1 (2 CPUs) to perform operational verification.

[0128] Next, the verification unit 12 determines whether or not the operational verification of all of the multiple verification resource patterns has been completed (step S805). If the verification unit 12 determines that the operational verification of all of the multiple verification resource patterns has not been completed (step S805: NO), it uses one of the multiple verification resource patterns whose operational verification has not been performed to execute the machine learning algorithm that was registered by the registration acceptance unit 11 and perform operational verification (step S803), and then repeats the subsequent processing. For example, the verification unit 12 performs operational verification using verification resource pattern 2 (a combination of 2 CPUs and 1 GPU).

[0129] On the other hand, if the verification unit 12 determines that it has completed operational verification of all of the multiple verification resource patterns (step S805: YES), the metadata registration unit 12-2 generates metadata that defines the resource conditions to be applied when machine learning is executed, based on the results of the operational verification of the multiple verification resource patterns, and registers it in the metadata storage unit 53 (step S807).

[0130] For example, if the operational verification process using verification resource pattern 1 (2 CPUs) takes 4 hours, and the operational verification process using verification resource pattern 2 (a combination of 2 CPUs and 1 GPU) takes 1 hour, the metadata will include a statement indicating that either verification resource pattern 1 or verification resource pattern 2 will be selected depending on the resource status of the machine learning execution environment.

[0131] Next, the verification unit 12 registers the verified machine learning algorithm in the algorithm storage unit 52 (step S809). This completes the process shown in this flowchart.

[0132] As described above, for example, focusing on computational resources and computation time, we can assume a situation where the computation time for a certain validation dataset is recorded in the metadata storage unit 53 for two cases: when the model is trained using only the CPU, and when the model is trained using both the CPU and the GPU. It is known that AI algorithms such as deep learning can have their processing time shortened by using a GPU, but since GPUs are more expensive hardware than CPUs, it is not always advisable to use a GPU. There are several possible methods for the learning control unit 112 to set the resources. For example, it may calculate the usage cost per unit time and determine which is more cost-effective: training the model using only the CPU or training the model using both the CPU and the GPU, and then select the appropriate method. Alternatively, if one of the two is to be prioritized, for example, if it is desirable to complete training quickly even if it is slightly more expensive, a cost function including weighting may be set, and the computational resources may be determined based on its output. For example, the cost function J can be defined as shown in equation (1) below.

[0133] J = Number of CPUs used × CPU usage time × Cost per CPU used per unit of time + Number of GPUs used × GPU usage time × Cost per GPU per unit of time ... Equation (1)

[0134] In machine learning execution environments, multiple users perform model training, so the GPU equipped with the training device may be occupied by other users. In such cases, if the algorithm can run on the CPU alone, it may be possible to perform model training using only the CPU.

[0135] Furthermore, if variations are possible in the execution conditions and settings of a machine learning algorithm, the operation may be verified for multiple patterns. If a domain is given for the hyperparameters of a machine learning algorithm, such as parameters that determine the model structure of the machine learning algorithm or parameters that control model learning, verification may be performed when the maximum value and the minimum value are selected for the parameters.

[0136] Specifically, the verification unit 12 executes the machine learning algorithm using at least one computing resource allocation pattern, and the first display control unit (display control unit 14) displays feedback information including the success or failure of the execution, processing speed, and amount of memory used. The verification unit 12 also executes the machine learning algorithm using multiple computing resource allocation patterns, and the control unit (learning control unit 112) allocates resources for machine learning execution based on the results of the operational verification by the verification unit 12 using the multiple computing resource allocation patterns.

[0137] According to the seventh embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it becomes possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, by performing operational verification using multiple verification resource patterns, it becomes possible to automatically verify and manage the execution conditions of new machine learning algorithms, which are difficult for even developers to comprehensively grasp. For example, regarding the CPU and GPU required as computing resources, the effort required of developer D to perform prior verification and evaluation and adjust them according to the actual situation at execution is reduced.

[0138] <Eighth Embodiment> The eighth embodiment will now be described. The machine learning automated execution system S of the eighth embodiment has the same configuration as the machine learning automated execution system S of the third embodiment. The machine learning automated execution system S of the eighth embodiment differs from the third embodiment in that it measures the amount of memory used during operation verification. The machine learning automated execution system S of the eighth embodiment will now be described, focusing on the differences from the third embodiment.

[0139] [Process for verifying operation] The following describes the processing flow for verifying the operation of a machine learning algorithm by the machine learning algorithm integrated management device 1. Figure 14 is a flowchart showing an example of the processing for verifying the operation of a machine learning algorithm according to the eighth embodiment. The explanation will use as an example the case where the verification dataset storage unit 51 stores two verification datasets with different data configurations: verification pattern 1 (image classification dataset, 500 normal data, 500 abnormal data) and verification pattern 2 (image classification dataset, 500 normal data, 5,000 abnormal data).

[0140] First, the registration reception unit 11 accepts the registration of a new machine learning algorithm entered by developer D via the input interface 40 (step S901).

[0141] Next, the verification unit 12 uses the verification dataset of one of the multiple verification patterns (for example, verification pattern 1) to allocate a sufficient amount of memory (for example, 256 GB) and executes the machine learning algorithm registered by the registration acceptance unit 11 to perform operational verification (step S903).

[0142] Next, the verification unit 12 determines whether or not all operational verifications of the multiple verification patterns have been completed (step S905). If the verification unit 12 determines that all operational verifications of the multiple verification patterns have not been completed (step S905: NO), it uses one of the multiple verification patterns whose operational verification has not yet been performed to execute the machine learning algorithm registered by the registration acceptance unit 11 and perform operational verification (step S903), and then repeats the subsequent processing. For example, the verification unit 12 performs operational verification using the verification dataset for verification pattern 2.

[0143] If the verification unit 12 determines that all operational verifications of multiple verification patterns have been completed (step S905: YES), the metadata registration unit 12-2 generates metadata that defines the conditions regarding memory usage to be applied when machine learning is executed, based on the results of the operational verifications of the multiple verification patterns, and registers it in the metadata storage unit 53 (step S907).

[0144] For example, if the maximum memory usage during operational verification in verification pattern 1 is 11.2GB and the maximum memory usage during operational verification in verification pattern 2 is 11.3GB, the metadata will include a statement indicating that "12GB" should be allocated as memory for machine learning.

[0145] Next, the verification unit 12 registers the verified machine learning algorithm in the algorithm storage unit 52 (step S909). This completes the process shown in this flowchart.

[0146] As described above, the verification unit 12 allocates a sufficient amount of memory for verification and records the amount of memory used when executing the algorithm. The learning unit 114 of the machine learning execution device 100 allocates slightly more memory than the amount of memory used in verification and executes the algorithm. This allocates the memory necessary for execution while preventing the allocation of excessive memory, allowing the entire system to use memory efficiently.

[0147] The memory used when executing an algorithm may depend on the size of the dataset. To address this, the validation unit 12 performs validation with multiple datasets of different sizes and records the memory used. The machine learning execution device 100 may allocate a value greater than the largest memory usage size obtained from the validation unit during model training. Alternatively, an approximation line or curve may be fitted to the dataset size and memory usage to estimate the amount of memory used from the dataset size. In this case, the machine learning execution device 100 estimates the required amount of memory according to the dataset set by user U, allocates it, and performs model training.

[0148] According to the eighth embodiment of the automated machine learning execution system S described above, when registering a new machine learning algorithm with the automated machine learning execution system S, it becomes possible to automatically verify and confirm that it meets the minimum specifications. Furthermore, by performing operational verification using multiple verification patterns, it becomes possible to automatically verify and manage the execution conditions of new machine learning algorithms, which are difficult for even developers to comprehensively grasp. In other words, the effort required of the developer D to perform prior verification and evaluation of the memory necessary as a computing resource and to adjust it according to the actual situation at execution is reduced.

[0149] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0150] 1…Machine learning algorithm integrated management device, 10…Control unit, 11…Registration reception unit, 12…Verification unit, 12-1…Verification data editing unit, 12-2…Metadata registration unit, 13…Feedback information generation unit, 14…Display control unit, 20…Communication interface, 30…Display unit, 40…Input interface, 50…Storage unit, 51…Verification dataset storage unit, 52…Algorithm storage unit, 53…Metadata storage unit, 100…Machine learning execution device, 110…Control unit, 111…Acquisition unit, 112…Learning control unit, 113…Preprocessing unit, 114…Learning unit, 115…Display control unit, 120…Communication interface, 130…Display unit, 140…Input interface, 150…Storage unit, 151…Learning dataset storage unit, 152…Evaluation result storage unit, S…Machine learning automated execution system

Claims

1. A registration department that will accept new registrations of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that registers at least one machine learning algorithm used for training a model by the learning unit, Equipped with, The verification unit, if the results of the operational verification of the machine learning algorithm meet certain criteria, permits registration of the machine learning algorithm as an operationally verified machine learning algorithm in the algorithm storage unit. Machine learning automated execution system.

2. A control unit that controls the execution of machine learning, A training dataset storage unit for storing the training dataset used for the aforementioned machine learning, A preprocessing unit performs preprocessing on the training dataset stored in the training dataset storage unit based on the control of the control unit, Based on the control by the control unit, the learning unit uses the machine learning algorithm stored in the algorithm storage unit to train the model on the preprocessed training dataset, An evaluation result storage unit for storing the evaluation results of the learned model, A second display control unit controls the display of the evaluation results stored in the evaluation result storage unit, Furthermore, The machine learning automated execution system according to claim 1.

3. The second display control unit displays a list of the machine learning algorithms stored in the algorithm storage unit. The learning unit performs training on the model using the machine learning algorithm specified by the user from among the machine learning algorithms included in the displayed list. The machine learning automated execution system according to claim 2.

4. The learning unit selects one or more machine learning algorithms stored in the algorithm memory unit based on a machine learning task specified by the user, and uses the selected machine learning algorithms to train the model. The machine learning automated execution system according to claim 2.

5. The second display control unit displays the hyperparameter items defined in the machine learning algorithm stored in the algorithm storage unit. The learning unit performs training of the model using the hyperparameter values ​​specified by the user for the displayed hyperparameter items. The machine learning automated execution system according to claim 2.

6. The learning unit selects at least one hyperparameter setting defined in the machine learning algorithm stored in the algorithm memory unit, and uses the selected hyperparameter setting to train the model. The machine learning automated execution system according to claim 2.

7. A registration acceptance unit that accepts registration of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that stores the machine learning algorithm whose operational verification results satisfy a specific criterion, Equipped with, The verification dataset includes at least one standard format dataset and at least one extended format dataset. If the verification unit determines that the operational verification using the standard format dataset is successful, but the operational verification using the extended format dataset is unsuccessful, it estimates the cause of failure based on the difference in data format between the standard format dataset and the extended format dataset. The feedback information generation unit generates the feedback information including the estimated cause of failure. Machine learning automated execution system.

8. The aforementioned standard format dataset is an image dataset unified by a specific file format, resolution, number of channels, and color depth. The aforementioned extended format dataset is an image dataset that differs in at least one of the file format, resolution, number of channels, color depth, and number of images, or an image dataset that differs in the number of images. The verification unit estimates the difference between the standard format dataset and the extended format dataset as the cause of failure. The machine learning automated execution system according to claim 7.

9. The aforementioned standard format dataset is a tabular dataset consisting only of real-valued variables. The aforementioned extended format dataset is a tabular dataset containing at least one of categorical variables, text data, time data, and missing values, or a tabular dataset with different sample sizes. The verification unit estimates the difference between the standard format dataset and the extended format dataset as the cause of failure. The machine learning automated execution system according to claim 7.

10. A registration acceptance unit that accepts registration of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that stores the machine learning algorithm whose operational verification results satisfy a specific criterion, Equipped with, The verification dataset includes at least one standard format dataset and at least one extended format dataset. The verification unit executes the machine learning algorithm to perform a model training accuracy test, and if the results of the accuracy test using the standard format dataset meet the accuracy criteria, but the results of the accuracy test using the extended format dataset do not meet the accuracy criteria, the verification unit estimates the cause of failure from the differences in data properties between the standard format dataset and the extended format dataset. The feedback information generation unit generates the feedback information including the estimated cause of failure. Machine learning automated execution system.

11. The aforementioned standard format dataset is an image dataset that takes image data consisting only of normalized luminance values ​​as input, and has no bias in the values ​​or labels of the target variable. The aforementioned extended format datasets include those in which the input images contain outliers, those in which the target variable has biases in values ​​or labels, and those with different numbers of images. The verification unit estimates the difference between the standard format dataset and the extended format dataset as the cause of failure. The machine learning automated execution system according to claim 10.

12. The aforementioned standard format dataset consists only of normalized real-valued variables and is a tabular dataset that is unbiased in the values ​​and labels of the target variable. The aforementioned extended format datasets include those with outliers in the explanatory variables, those with bias in the values ​​or labels of the dependent variable, or those with different sample sizes. The verification unit estimates the difference between the standard format dataset and the extended format dataset as the cause of failure. The machine learning automated execution system according to claim 10.

13. A registration acceptance unit that accepts registration of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that stores the machine learning algorithm whose operational verification results satisfy a specific criterion, A control unit that controls the execution of machine learning, A training dataset storage unit for storing the training dataset used for the aforementioned machine learning, A preprocessing unit performs preprocessing on the training dataset stored in the training dataset storage unit based on the control of the control unit, A learning unit, based on control by the control unit, uses the machine learning algorithm stored in the algorithm storage unit to train a model on the preprocessed training dataset. An evaluation result storage unit for storing the evaluation results of the learned model, A second display control unit controls the display of the evaluation results stored in the evaluation result storage unit, Equipped with, The verification dataset includes at least one standard format dataset and at least one extended format dataset. If the operational verification using the standard format dataset is successful, but the operational verification using the extended format dataset is unsuccessful, the metadata registration unit stores the items of the extended format dataset in the metadata storage unit as preprocessing to be applied when machine learning is executed. A preprocessing unit that, by referring to the metadata storage unit, performs a process to convert the training dataset requiring preprocessing into a standard format, Furthermore, Machine learning automated execution system.

14. A registration acceptance unit that accepts registration of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that stores the machine learning algorithm whose operational verification results satisfy a specific criterion, A control unit that controls the execution of machine learning, A training dataset storage unit for storing the training dataset used for the aforementioned machine learning, A preprocessing unit performs preprocessing on the training dataset stored in the training dataset storage unit based on the control of the control unit, A learning unit, based on control by the control unit, uses the machine learning algorithm stored in the algorithm storage unit to train a model on the preprocessed training dataset. An evaluation result storage unit for storing the evaluation results of the learned model, A second display control unit controls the display of the evaluation results stored in the evaluation result storage unit, Equipped with, The verification dataset includes at least one standard format dataset and at least one extended format dataset. The verification unit executes the machine learning algorithm to perform an accuracy test of the model learning, If the results of the accuracy test using the standard format dataset meet the accuracy criteria, but the results of the accuracy test using the extended format dataset do not meet the accuracy criteria, a metadata registration unit stores the items of the extended format dataset in the metadata storage unit as preprocessing to be applied when machine learning is executed. A preprocessing unit that refers to the metadata storage unit and performs a process to convert the dataset requiring preprocessing into a standard format, Furthermore, Machine learning automated execution system.

15. The verification unit compares the results of the operational verification of the machine learning algorithm with the results of the operational verification of other similar machine learning algorithms, with respect to at least one of the following: accuracy, execution time, and computing resources. The machine learning automated execution system according to claim 1 or 2.

16. It also includes a verification data editing unit that performs standardization processing to convert extended format datasets into standard format datasets. The verification unit performs operational verification using a standardized dataset. The machine learning automated execution system according to claim 3 or 4.

17. A registration acceptance unit that accepts registration of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that stores the machine learning algorithm whose operational verification results satisfy a specific criterion, Equipped with, The verification unit executes the machine learning algorithm using at least one computing resource allocation pattern. The first display control unit displays the feedback information, including the success or failure of execution, processing speed, and amount of memory used. Machine learning automated execution system.

18. A registration acceptance unit that accepts registration of machine learning algorithms, A verification dataset storage unit for storing a verification dataset used to verify the operation of the machine learning algorithm, A verification unit that performs operational verification by executing the machine learning algorithm using the verification dataset stored in the verification dataset storage unit, A feedback information generation unit that generates feedback information based on the results of the aforementioned operational verification, A first display control unit that controls the display of the generated feedback information, An algorithm storage unit that stores the machine learning algorithm whose operational verification results satisfy a specific criterion, A control unit that controls the execution of machine learning, A training dataset storage unit for storing the training dataset used for the aforementioned machine learning, A preprocessing unit performs preprocessing on the training dataset stored in the training dataset storage unit based on the control of the control unit, A learning unit, based on control by the control unit, uses the machine learning algorithm stored in the algorithm storage unit to train a model on the preprocessed training dataset. An evaluation result storage unit for storing the evaluation results of the learned model, A second display control unit controls the display of the evaluation results stored in the evaluation result storage unit, Equipped with, The verification unit executes the machine learning algorithm with multiple computing resource allocation patterns. The control unit allocates resources for machine learning execution based on the results of the operational verification by the verification unit for the multiple computing resource allocation patterns. Machine learning automated execution system.

19. Computers We are now accepting new registrations for machine learning algorithms. The machine learning algorithm is executed using the verification dataset stored in the verification dataset storage unit, which is used to verify the operation of the machine learning algorithm, and the operation is verified. Based on the results of the aforementioned operational verification, feedback information is generated, Control the display of the generated feedback information, If the results of the operational verification of the machine learning algorithm meet certain criteria, at least one of the machine learning algorithms used for training a model by the learning unit is registered as an operationally verified machine learning algorithm, and registration to the algorithm storage unit is permitted. Methods for automating machine learning execution.

20. On the computer, We are now accepting new registrations for machine learning algorithms. The machine learning algorithm is executed using the verification dataset stored in the verification dataset storage unit, which is used to verify the operation of the machine learning algorithm, and the operation is verified. Based on the results of the aforementioned operational verification, feedback information is generated. Control the display of the generated feedback information, If the results of the operational verification of the machine learning algorithm meet certain criteria, at least one of the machine learning algorithms used for training a model by the learning unit is registered as an operationally verified machine learning algorithm, and registration to the algorithm storage unit is permitted. program.

Images