Development system and learning management system

Abstract

To provide a development system capable of constructing an MLOps system by combining predetermined platforms, and achieving an efficient MLOps system leveraging the characteristics of each platform.SOLUTION: A development system 100 comprises a first pipeline 120 which processes training data for training in an external learning system 400 and trains a machine learning model 401 in the external learning system 400 using the processed training data, and a second pipeline 130 which deploys the machine learning model 401 trained with the training data to an operation system 200.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to development systems and learning management systems. [Background technology]

[0002] In recent years, in the field of machine learning, MLOps systems, a coined term combining machine learning (ML) and operations, have become widespread. MLOps systems are a flow that smoothly advances machine learning, and for example, by pipeline-izing the development process and the operations process, they aim to facilitate data processing and communication. MLOps can also improve productivity by automating version management, deployment, and other processes (see Non-Patent Documents 1 and 2).

[0003] According to a reference translation of the abstract of Non-Patent Document 1, "The ultimate goal of all industrial machine learning (ML) projects is to develop ML products and rapidly deploy them in production. However, automating and operating ML products is extremely difficult, so many ML efforts fall short of expectations. The Machine Learning Operations (MLOps) paradigm addresses this issue. MLOps encompasses several aspects, such as best practices, a set of concepts, and a development culture. However, MLOps is still a vague term, and its impact for researchers and practitioners is unclear. To address this gap, we conduct a mixed-methods study, including a literature review, tool review, and expert interviews. As a result of these studies, we contribute to the body of knowledge by providing a consolidated overview of the necessary principles, components, roles, and associated architectures and workflows. Furthermore, we provide a comprehensive definition of MLOps and highlight open challenges in this field. Finally, this work provides guidance to ML researchers and practitioners who want to automate and operate ML products using a specified set of technologies."

[0004] Furthermore, Non-Patent Document 2, MLOps Level 1: Automating ML Pipelines, states that "The goal of Level 1 is to perform continuous training of models by automating the ML pipeline, thereby enabling continuous delivery of model prediction services. To automate the process of retraining models in a production environment using new data, automated data and model validation procedures, as well as pipeline triggers and metadata management, must be introduced into the pipeline."

[0005] Furthermore, the following is described as a feature of FIG. 3 in Non-Patent Document 2. " Rapid testing: ML testing steps are integrated, and automating the movement between steps allows for faster test iterations and improves readiness to move the entire pipeline to production. CT of models in production: Models are automatically trained in production with new data based on live pipeline triggers described in the next section. Test-to-production symmetry: The pipeline implementation used in development or test environments is used in pre-production and production environments. This is a key aspect of MLOps methodologies that integrate DevOps. Modularized code for components and pipelines: To build ML pipelines, components must be reusable, composable, and potentially shareable across ML pipelines. Therefore, while EDA code can reside within a notebook, the component source code should be modularized. Components should also ideally be containerized for the following operations: Separating the execution environment and custom code runtime. · Make your code reproducible between development and production environments. Isolating each component in the pipeline. Each component can have its own version of the runtime environment, different languages ​​and libraries. Continuous Model Delivery: A production ML pipeline continuously provides prediction services for new models trained on new data. It automates the model deployment step, providing trained and validated models as prediction services for online predictions. Pipeline deployment: Level 0 deploys the trained model in production as a prediction service. Level 1 deploys the entire training pipeline, which runs automatically and repeatedly to serve the trained model as a prediction service. [Prior art documents] [Patent documents]

[0006] [Non-Patent Document 1] Dominik Kreuzberger, Niklas Kuhl, and Sebastian Hirschl. "Machine Learning Operations (MLOps): Overview, Definition, and Architecture", IEEE Access, 27 March 2023, pp. 31866-31879, [online], [Retrieved April 2, 2024], Internet<https: / / ieeexplore.ieee.org / document / 10081336> [Non-patent document 2] Cloud Architecture Center, [online], [Retrieved April 2, 2024], Internet <https: / / cloud.google.com / architecture / mlops-continuous-delivery-and-automation-pipelines-in-machine-learning?hl=jahttps%3A%2F%2Fieeexplore.ieee.org%2Fdocument%2F10081336> Summary of the Invention [Problem to be solved by the invention]

[0007] The purpose of MLOps is to ensure the smooth operation and management of AI products. Therefore, in the MLOps flow, the data, source code, devices, etc. used in all steps must exist on the same platform (PF) that is easy to consolidate. However, when choosing a platform based on the learning environment, which is the most important aspect of machine learning, it can sometimes be difficult to select a platform that is specialized for databases, model management, and user interfaces (UIs).

[0008] Therefore, if it is not possible to combine platforms that are suited to the purpose of each step in the MLOps flow, it is expected that the work efficiency of developing AI products using MLOps systems will decrease.

[0009] Non-Patent Document 1 achieves rapid response to corrections to trained models through a continuous machine learning flow, which is the basic mechanism of MLOps. However, the MLOps flow in Non-Patent Document 1 has difficulty responding when the accuracy of the trained model is significantly degraded and cannot be completely corrected. In this case, Non-Patent Document 1 requires a separate, intensive process to improve accuracy using a flow specialized for machine learning.

[0010] On the other hand, Non-Patent Document 2 expands MLOps into two parts. Specifically, it separates it into a development part and an operations part. The development part concentrates on processing to improve the accuracy of the algorithm. In contrast, the operations part responds quickly to minute deteriorations in accuracy in the field. This allows Non-Patent Document 2 to smoothly perform updates to improve the accuracy of machine learning models. Therefore, Non-Patent Document 2 can respond to the deterioration in accuracy of Non-Patent Document 1.

[0011] However, the technology in Non-Patent Document 2 requires that the data, source code, and devices used in all steps in the flow be placed on the same platform, which is easy to consolidate. In other words, Non-Patent Document 2 includes a development part that requires large learning resources, which means that the platform on which MLOps is placed is selected based on the learning environment.

[0012] As a result, for steps other than machine learning, it becomes impossible to select a platform specialized for the database, model management, and UI that is suited to the functions and purposes of each step, which reduces the efficiency of MLOps system development.

[0013] In response to these issues, the present invention aims to provide a development system and learning management system that can build an MLOps system by combining specified platforms and realizes an efficient MLOps system that makes use of the characteristics of each platform. [Means for solving the problem]

[0014] That is, the above-mentioned problems of the present invention are solved by the following configuration. (1) a first pipeline that processes training data for training in an external training system and trains a machine learning model in the external training system using the processed training data; a second pipeline that deploys the machine learning model trained with the training data into an operational system; A development system having: (2) The first pipeline is a transmitting unit that transmits information including the processed learning data to an external learning system; The second pipeline is a receiving unit that receives a machine learning model trained by the external learning system; a model deployment unit that deploys the machine learning model into an operational system; The development system according to (1) above. (3) The transmitting unit transmitting the second pipeline including information necessary for executing the second pipeline; (2) The development system according to (2). (4) the information necessary for executing the second pipeline includes a processing instruction trigger code for starting processing of the second pipeline; (3) A development system according to (3). (5) The first pipeline is converting the learning data into a data format suitable for a machine learning model of the external learning system, and transmitting the learning data in the converted data format to the external learning system by the transmitting unit; (2) The development system according to (2). (6) Further comprising a conversion script for converting the data into a data format suitable for a machine learning model of an external learning system; (5) A development system according to (5). (7) The second pipeline is receiving a machine learning model trained by the external learning system, and converting the received machine learning model into a format usable by the operational system; (1) A development system according to (1). (8) The model deploy unit: transmitting the machine learning model to the operational system; (2) The development system according to (2). (9) The receiving unit Accepting a designation of a receiving format of a predetermined machine learning framework, and converting the received machine learning model into the predetermined machine learning framework of the designated receiving format; (7) A development system according to (7). (10) The first pipeline is The communication device further includes a data set unit that prepares data to be sent to the transmitting unit and transfers the data. (2) A development system according to (2). (11) A first pipeline that processes training data for training in an external training system and trains a machine learning model in the external training system using the processed training data; a second pipeline that deploys a machine learning model trained with the training data; an operational system for evaluating the machine learning model; A learning management system with (12) The operation system is a receiving unit that receives a machine learning model from the second pipeline; a learning unit that causes the machine learning model received from the receiving unit to be re-learned; an evaluation unit that performs inference using the machine learning model received from the learning unit and calculates an evaluation result including a performance index; a management unit that manages the re-trained model received from the evaluation unit and transfers it to an external system; The learning management system according to (11) above, (13) The management department: When a re-learning command is received from the outside, the first pipeline is caused to re-execute the process. (12) A learning management system according to (12). [Effects of the Invention]

[0015] According to the present invention, an MLOps system can be constructed by combining predetermined platforms, and an efficient MLOps system can be realized that makes the most of the characteristics of each platform. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing an example of the main configuration of a learning management system according to a first embodiment. [Figure 2] 1 is a block diagram showing the hardware configuration of a learning management system according to a first embodiment. [Figure 3] FIG. 10 is an explanatory diagram showing a setting screen for transmitting learning data through a first pipeline; [Figure 4]FIG. 10 is an explanatory diagram showing the body of an API sent from a development system to an external learning system. [Figure 5] FIG. 10 is an explanatory diagram showing parameter information received by a receiving unit. [Figure 6] This is a flowchart showing the process by which the second pipeline converts a machine learning model received from an external learning system into a format that can be used by the operational system. [Figure 7] This is a schematic diagram showing the concept of the second pipeline accepting a specified receiving format and converting it into a predetermined machine learning framework. [Figure 8] FIG. 10 is a block diagram showing an example of the main configuration of a learning management system according to a second embodiment. [Figure 9] FIG. 10 is a block diagram showing the hardware configuration of a learning management system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following describes in detail embodiments of the present invention. Note that the embodiments described below are examples for realizing the present invention. Therefore, the present invention should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments. Furthermore, identical components are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0018] <Comparative Example> We will explain the conventional technology as a comparative example. For example, when releasing an AI product externally, there are cases where you want to manage MLOps using a cloud server that is suitable for external release. In this case, you want to use a cloud server, but there are also cases where you want to perform machine learning using your own company's abundant on-premise resources. In other words, there are times when you want to use a system owned by your company for machine learning, rather than a cloud server.

[0019] In addition, users may prefer to use cloud servers for management because they have an easy-to-understand UI and facilitate source management. However, cloud servers often have poor learning capabilities. For this reason, users may prefer to use high-performance yet inexpensive learning resources such as ABCI (AI Bridging Cloud Infrastructure).

[0020] Furthermore, there are cases where companies want to use cloud servers specialized for data management, but it is expected that such cloud servers will be difficult to apply to their own in-house platforms that are specialized for UI.

[0021] These are some of the problems that have existed in the past. In particular, with MLOps, systems that are easy to manage, have a user-friendly UI, and are not difficult to implement are generally preferred. However, it has become common for MLOps to be run on the same platform.

[0022] This makes it difficult to use separate platforms or hardware resources for each step. Furthermore, if services specialized in machine learning and services specialized in data management are implemented on different platforms, it becomes extremely difficult to incorporate MLOps flows.

[0023] Therefore, the development systems 100 and 101 according to this embodiment can select a platform suited to the purpose of each step, and can perform efficient development using an MLOps system.

[0024] Additionally, MLOps requires all flows to be connected as a single system. To achieve this, a script or service is required to compile and manage the entire system. This is generally called an orchestrator. The orchestrator must have access to the source code and hardware used in each step of the MLOps flow.

[0025] However, orchestrators have limited access to resources, which means that there are combinations of systems and hardware that are difficult to incorporate into a single MLOps system.

[0026] Therefore, the development systems 100 and 101 according to this embodiment can build an MLOps system even if there are systems or hardware combinations that are difficult to incorporate into a single MLOps system. Therefore, the development systems 100 and 101 according to this embodiment can solve these problems and realize an efficient MLOps system that makes the most of the characteristics of each platform.

[0027] First Embodiment [Overall configuration of learning management system] 1 is a block diagram showing an example of the main configuration of a learning management system 300 according to the first embodiment. As shown in FIG. 1, the learning management system 300 according to the first embodiment is configured to include a development system 100 and an operation system 200.

[0028] The development system 100 is configured to include a first pipeline 120 and a second pipeline 130.

[0029] The first pipeline 120 processes the training data for training by the external training system 400 (external training system), and uses the processed training data to train a machine learning model in the external training system 400. The first pipeline 120 includes a transmission unit 122.

[0030] The transmitting unit 122 transmits information including the processed learning data to the external learning system 400. The transmitting unit 122 is configured to include code 123 and a conversion script 124. The transmitting unit 122 may transmit information required for executing the second pipeline 130. In this case, the information required for executing the second pipeline 130 may include code 123 (see FIG. 4) of a processing command trigger for starting processing of the second pipeline 130.

[0031] Furthermore, first pipeline 120 converts the learning data into a data format suitable for machine learning model 401 of external learning system 400, and transmits the learning data in the converted data format to external learning system 400 via transmission unit 122. First pipeline 120 is provided with conversion script 124, and is thereby able to convert the learning data into a data format suitable for machine learning model 401 of external learning system 400. Note that conversion script 124 is provided in transmission unit 122, for example, as shown in FIG. 1 .

[0032] The second pipeline 130 deploys the machine learning model 401, which has been trained using training data in the external learning system 400, to the operation system 200. The second pipeline 130 includes a receiving unit 131 and a model deploying unit 132.

[0033] The receiving unit 131 receives the machine learning model 401 trained by the external learning system 400. The receiving unit 131 is configured with a conversion script 133. The receiving unit 131 accepts a designation of the reception format of a predetermined machine learning framework, and converts the received machine learning model 401 into the predetermined machine learning framework of the reception format designated by the conversion script 133. This processing will be described later with reference to FIGS. 6 and 7. Note that the conversion script 133, FIGS. 6 and 7 are merely examples and are not limited thereto.

[0034] The model deploy unit 132 deploys the machine learning model 401 to the operation system 200. The model deploy unit 132 transmits the machine learning model 401 to the operation system 200.

[0035] This allows the second pipeline 130 to receive the machine learning model 401 trained by the external learning system 400 and convert the received machine learning model 401 into a format that can be used by the operation system 200.

[0036] The operation system 200 evaluates the machine learning model 401. The operation system 200 represents a production environment for MLOps, and has an inference function for an AI algorithm. The operation system 200 will be described in detail in a second embodiment.

[0037] External learning system 400 is an external system that performs predetermined machine learning and is not included in learning management system 300. External learning system 400 can perform predetermined machine learning on a platform different from that of learning management system 300, and is not limited to one in particular. In other words, external learning system 400 represents a learning environment provided externally.

[0038] 2 is a block diagram showing the hardware configuration of a learning management system 300 according to the first embodiment. As shown in FIG. 2, the learning management system 300 is configured to include a development system 100 and an operation system 200.

[0039] The development system 100 shown in FIG. 3 is configured to include a CPU (Central Processing Unit) 110, a ROM (Read Only Memory) 111, a RAM (Random Access Memory) 112, a storage unit 113, an input unit 114, a display unit 115, and a communication unit 116.

[0040] The development system 100 is configured to include a first pipeline 120 and a second pipeline 130 .

[0041] The CPU 110 is a central processing unit, and embodies a first pipeline 120 and a second pipeline 130 shown in FIGS. 1 and 2 by executing a predetermined program stored in the ROM 111 or the storage unit 113.

[0042] The RAM 112 functions as a work area for temporarily storing various programs that are read from the ROM 111 and that can be executed by the CPU 110, input data, output data, and parameters.

[0043] The storage unit 113 is a large-capacity storage device, and is configured, for example, by a hard disk drive, a nonvolatile memory, or the like.

[0044] The input unit 114 is configured to include a keyboard having cursor keys, numeric input keys, and various function keys, and a pointing device such as a mouse. The input unit 114 outputs press signals of keys pressed on the keyboard and operation signals of the mouse as input signals to the CPU 110. The CPU 110 executes various processes based on the operation signals from the input unit 114.

[0045] Display unit 115 is configured to include a monitor such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display). Display unit 115 displays various screens in accordance with instructions of a display signal input from CPU 110. Note that display unit 115 and input unit 114 may also employ touch panel displays.

[0046] The communication unit 116 includes a communication interface and communicates with external devices on a network. The communication unit 116 is connected to the external learning system 400 (FIG. 1), and transmits information including learning data to the external learning system 400, for example.

[0047] The operation system 200 shown in FIG. 2 is configured to include a CPU 210, a ROM 211, a RAM 212, a storage unit 213, an input unit 214, a display unit 215, and a communication unit 216.

[0048] The CPU 210 is a central processing unit, and executes a predetermined program stored in the ROM 211 or the storage unit 213 to embody components according to a second embodiment (see FIGS. 8 and 9) described later.

[0049] The RAM 212 functions as a work area for temporarily storing various programs that are read from the ROM 211 and that can be executed by the CPU 210, input data, output data, and parameters.

[0050] The storage unit 213 is a large-capacity storage device, and is configured, for example, by a hard disk drive, a nonvolatile memory, or the like.

[0051] The input unit 214 is configured to include a keyboard having cursor keys, numeric input keys, and various function keys, and a pointing device such as a mouse. The input unit 214 outputs press signals of keys pressed on the keyboard and operation signals from the mouse as input signals to the CPU 210. The CPU 210 executes various processes based on the operation signals from the input unit 214.

[0052] The display unit 215 is configured to include a monitor such as a CRT or LCD. The display unit 215 displays various screens in accordance with instructions of a display signal input from the CPU 210. Note that the display unit 215 and the input unit 214 may also be touch panel displays.

[0053] The communication unit 216 includes a communication interface and communicates with external devices on a network. The communication unit 216 is connected to the external learning system 400 (FIG. 1), and receives, for example, the machine learning model 401 trained by the external learning system 400.

[0054] First Embodiment [Learning Management System Operation] The learning management system 300 according to the first embodiment operates as follows.

[0055] First, the first pipeline 120 of the development system 100 receives predetermined settings from the MLOps launch portal in order to perform machine learning in the external learning system 400.

[0056] Fig. 3 is an explanatory diagram showing a setting screen for transmitting learning data from the first pipeline 120. As shown in Fig. 3, on a setting screen 1221, items 1222 to 1226 are selected using a pull-down menu. Items 1222 to 1226 are configured as combo boxes.

[0057] In item 1222, the provider to which the learning data will be provided is selected. For example, in item 1222, Company A is selected. In item 1223, an algorithm for converting the data into a format suitable for the next step is specified. For example, in item 1223, the "object detection" algorithm is specified.

[0058] Item 1224 specifies a learning instance that indicates the dataset to be used for machine learning. "YYY" is specified in item 1224. Item 1225 specifies the data type to be sent. "ZZZ" is specified in item 1225. When "ZZZ" specified in item 1225 is sent, it is converted into a data format suitable for "object detection" in item 1223 and sent.

[0059] Item 1226 specifies the registration destination of the learning model. Item 1226 specifies "AAA" as the registration destination of the learning model. In this embodiment, by specifying the registration destination of the learning model, the external learning system 400 can download the registered learning data through internal processing using an API (Application Programming Interface).

[0060] In this way, the transmitting unit 122 converts the data into a format suitable for the next step. The transmitting unit 122 also sets the next step to download the data using an API or the like. After learning in the external learning system 400, the transmitting unit 122 notifies the external learning system 400 of a process command trigger that activates the next step using code 123.

[0061] 4 is an explanatory diagram showing the body 1151 of the API sent from the development system 100 to the external learning system 400. As shown in FIG. 4, the body 1151 indicates the code 123 of the processing instruction trigger of the API sent to the external learning system 400.

[0062] The transmission unit 122 transmits information including information required for executing the second pipeline 130. In this case, the transmission unit 122 transmits information including code 123 of a processing command trigger for starting processing of the second pipeline 130 via the API.

[0063] Body 1151, for example, states that the data type is "ms-coo" and the data storage location is "Registry R." Body 1151 also states that the experiment ID is "XXXXXX." Body 1151 also states a command for external learning system 400 to download learning data and a command for the post-learning flow as a subsequent step execution command.

[0064] In this embodiment, when the external learning system 400 executes a command to download learning data as a command indicating the start of processing, the external learning system 400 executes the command and starts downloading. Then, the external learning system 400 performs predetermined machine learning.

[0065] After the machine learning is completed, the external learning system 400 executes the command for the post-learning flow, which is specified as the subsequent step execution command, and then executes the post-learning flow. As a result, in this embodiment, the learning management system 300 instructs the external learning system 400 to have the development system 100 execute the next step of the flow continuously.

[0066] When the subsequent step execution command is executed by the external learning system 400, the development system 100 receives the machine learning model 401 trained by the external learning system 400 via the receiver 131 of the second pipeline 130. In this case, the receiver 131 acquires the experiment result storage location information and receives the trained machine learning model 401 and parameter information from the experiment result storage location by downloading it.

[0067] Upon receiving the parameter information, the receiving unit 131 converts the parameter information into a data format of JSON (JavaScript Object Notation).

[0068] FIG. 5 is an explanatory diagram showing parameter information 1311 received by the receiving unit 131. The parameter information 1311 in FIG. 5 has name and value columns. The name column of the parameter information 1311 lists information such as augmentation, batch size, machine learning model, class information, dataset ID, dataset version, depth, epoch, input size, and maximum epoch. The value column of the parameter information 1311 lists each value corresponding to the name. Note that the parameter information 1311 is an example and is not limited to this.

[0069] In this way, the receiving unit 131 converts parameters, meta-information about the experiment, and information about the storage location of the weight file into a format suitable for the next step using commands for the post-learning flow. The receiving unit 131 also acquires the data itself. By storing the weight file and meta-information in the appropriate storage location, the receiving unit 131 can continue the next flow in the next step, thereby enabling a smooth MLOps system flow.

[0070] Furthermore, the second pipeline 130 converts the machine learning model 401 received from the external learning system 400 into a format that can be used by the operation system 200. In this case, the receiving unit 131, for example, accepts a designation of a receiving format of a predetermined machine learning framework, and converts the received machine learning model 401 into the predetermined machine learning framework of the designated receiving format.

[0071] FIG. 6 is a flowchart showing the process in which the second pipeline 130 converts the machine learning model 401 received from the external learning system 400 into a format usable by the operation system 200.

[0072] First, the second pipeline 130 receives, at the input unit 114, a specification of the receiving format of the machine learning framework (step S01).

[0073] Next, the receiving unit 131 receives the trained machine learning model 401 (step S03). Then, the receiving unit 131 converts the received machine learning model 401 into a predetermined machine learning framework of the specified receiving format (step S05).

[0074] The order of steps S01 and S03 is not limited to this. That is, the second pipeline 130 can convert the trained machine learning model 401 into a predetermined machine learning framework by simply accepting the designation of the receiving format at the input unit 114 and receiving the trained machine learning model 401.

[0075] 7 is a schematic diagram showing the concept of the second pipeline accepting a designation of a receiving format and converting it into a predetermined machine learning framework. As shown in FIG. 7, the receiving unit 131 receives a machine learning model 401 from an external learning system 400.

[0076] The receiving unit 131 has, for example, a conversion script 133 according to the purpose (see FIG. 1). The conversion script 133 has, for example, three scripts, conversion scripts 1331 to 1333.

[0077] Conversion script 1331 is a conversion script for a device-compatible process for a CPU. Conversion script 1332 is a conversion script for a device-compatible process for a GPU (Graphics Processing Unit). Conversion script 1333 is a conversion script for a device-compatible process for an FPGA (Field Programmable Gate Array).

[0078] The receiving unit 131 can convert the received machine learning model 401 into a model for CPU 1334 by applying a conversion script 1331 to the received machine learning model 401. The receiving unit 131 can also convert the received machine learning model 401 into a model for GPU 1335 by applying a conversion script 1332 to the received machine learning model 401. The receiving unit 131 can also convert the received machine learning model 401 into a model for FPGA 1336 by applying a conversion script 1333 to the received machine learning model 401. In this way, the receiving unit 131 can convert into a predetermined machine learning framework of a specified reception format.

[0079] As described above, the development system 100 according to the first embodiment has a first pipeline 120 and a second pipeline 130. The first pipeline 120 processes the training data for training by the external training system 400, and trains the machine learning model 401 in the external training system 400 using the processed training data. The second pipeline 130 deploys the machine learning model 401, which has trained the training data, to the operation system 200.

[0080] According to the development system 100 of the first embodiment, the first pipeline 120 trains the machine learning model 401 in the external learning system 400. This allows the external learning system 400 to perform machine learning on a platform different from the learning management system 300. Furthermore, the second pipeline 130 can deploy the machine learning model 401 trained in the external learning system 400 to the operation system 200.

[0081] As a result, development system 100 according to the first embodiment can train machine learning model 401 in, for example, external learning system 400 that constitutes a platform. Therefore, development system 100 according to the first embodiment can perform machine learning on a platform different from learning management system 300.

[0082] Therefore, the development system 100 according to the first embodiment can build an MLOps system by combining predetermined platforms, and can realize an efficient MLOps system that makes the most of the characteristics of each platform.

[0083] However, the development system 100 according to the first embodiment is not limited to this. For example, the first pipeline 120 may include a transmitter 122 that transmits information including the processed learning data to the external learning system 400.

[0084] The transmitting unit 122 may transmit information necessary for executing the second pipeline 130. In this case, the information necessary for executing the second pipeline 130 may include a process command trigger code 123 for starting processing of the second pipeline 130.

[0085] In addition, the first pipeline 120 may convert the learning data into a data format suitable for the machine learning model 401 of the external learning system 400, and transmit the learning data in the converted data format to the external learning system 400 via the transmitting unit 122.

[0086] The first pipeline 120 may further include a conversion script 124 that converts the data into a data format suitable for the machine learning model 401 of the external learning system 400.

[0087] The second pipeline 130 may also have a receiving unit 131 that receives the machine learning model 401 trained by the external learning system 400, and a model deployment unit 132 that deploys the machine learning model 401 to the operation system 200.

[0088] In addition, the second pipeline 130 may receive the machine learning model 401 trained by the external learning system 400 and convert the received machine learning model 401 into a format that can be used by the operation system 200.

[0089] Furthermore, the receiving unit 131 may accept a designation of a receiving format of a predetermined machine learning framework, and convert the received machine learning model 401 into the predetermined machine learning framework of the designated receiving format.

[0090] In addition, the model deploy unit 132 may transmit the machine learning model 401 to the operation system 200.

[0091] Second Embodiment Fig. 8 is a block diagram showing an example of the main configuration of a learning management system 301 according to the second embodiment. Fig. 9 is a block diagram showing the hardware configuration of the learning management system 301 according to the second embodiment.

[0092] As shown in FIG. 8, the development system 101 of the learning management system 301 according to the second embodiment is configured such that a dataset 121 is further provided in the first pipeline 120 of the development system 100.

[0093] The data set 121 prepares data to be sent to the transmitting unit 122 and transfers the data.

[0094] Furthermore, the learning management system 301 according to the second embodiment is configured such that the operation system 201 further includes a receiving unit 220, a learning unit 221, an evaluation unit 222, and a management unit 223.

[0095] The receiving unit 220 receives the machine learning model 401 from the second pipeline 130. The learning unit 221 causes the machine learning model 401 received from the receiving unit 220 to undergo re-learning. The evaluation unit 222 performs inference using the machine learning model 401 received from the learning unit 221, and calculates an evaluation result including a performance index.

[0096] The management unit 223 manages the re-learned machine learning model 401 received from the evaluation unit 222 and transfers it to the external learning system 400. Furthermore, upon receiving a re-learning command from the outside, the management unit 223 may cause the first pipeline 120 to re-execute the process.

[0097] As described above, the learning management system 301 according to the second embodiment is configured to include a data set 121 in the development system 101. The data set 121 prepares data to be sent to the transmitting unit 122 and transfers the data.

[0098] Furthermore, the learning management system 301 according to the second embodiment is configured to include a receiving unit 220, a learning unit 221, an evaluation unit 222, and a management unit 223 in an operation system 201. The receiving unit 220 receives a machine learning model 401 from the second pipeline 130. The learning unit 221 causes the machine learning model 401 received from the receiving unit 220 to undergo re-learning. The evaluation unit 222 performs inference using the machine learning model 401 received from the learning unit 221 and calculates evaluation results including a performance index. The management unit 223 manages the re-learned machine learning model 401 received from the evaluation unit 222 and transfers it to the external learning system 400. Furthermore, upon receiving a re-learning command from an external source, the management unit 223 can cause the first pipeline 120 to re-execute processing.

[0099] According to the learning management system 301 of the second embodiment, the development system 101 is provided with the dataset 121, and thus can use, for example, its own database for the first pipeline 120. This allows the development system 101 to use its own dedicated database for the first pipeline 120 as one MLOps system.

[0100] Furthermore, according to the learning management system 301 of the second embodiment, the development system 101 can realize, for example, a company's own on-premise server that can be directly connected to the external learning system 400 for the second pipeline 130. This allows the development system 101 to receive the machine learning model 401 for the second pipeline 130, and to use an easy-to-manage, large-capacity company server.

[0101] Furthermore, according to the learning management system 301 of the second embodiment, the operation system 201 is configured to include a receiving unit 220, a learning unit 221, an evaluation unit 222, and a management unit 223. This allows the operation system 201 to apply cloud services that are easy to deploy to customers. Therefore, the operation system 201 can provide an optimal platform according to the destination and form of provision.

[0102] Furthermore, when the management unit 223 of the operational system 201 receives a re-learning command from the outside, it can cause the first pipeline 120 to re-execute the processing, thereby maintaining and improving the accuracy of inference in the evaluation unit 222.

[0103] As a result, the learning management system 301 according to the second embodiment can be deployed separately on platforms suited to each purpose, making it possible to build an efficient MLOps system.

[0104] (Variation) The present invention is not limited to the above-described embodiment, and modifications can be made without departing from the spirit of the present invention, for example, the following (a) to (b).

[0105] (a) The learning management system 300 according to the first embodiment may be configured to include a first pipeline 120, a second pipeline 130, and an operation system 200. The first pipeline 120 processes training data for training by an external training system 400, and trains a machine learning model 401 in the external training system 400 using the processed training data. The second pipeline 130 deploys the machine learning model 401 that has trained the training data. The operation system 200 evaluates the machine learning model 401.

[0106] (b) The external learning system 400 can perform learning in a predetermined environment that is optimal for machine learning. In this case, the external learning system 400 can be operated as an on-premise system by an outsourced party. Note that the external learning system 400 is not limited to an outsourced party, and may also be a dedicated machine learning server within the company. [Explanation of symbols]

[0107] 100,101 Development System 120 First Pipeline 121 datasets 122 Transmitter 123 Code 124 Conversion script 130 Second Pipeline 131 Receiving unit 132 Model Deployment Department 133 Conversion script 200,201 Operational System 220 Receiving Department 221 Learning Department 222 Evaluation Department 223 Management Department 300,301 Learning Management Systems 400 External Learning System (External Learning System) 401 Machine Learning Models

Claims

1. a first pipeline that processes training data for training in an external training system and trains a machine learning model in the external training system using the processed training data; a second pipeline that deploys the machine learning model trained with the training data into an operational system; A development system having:

2. The first pipeline comprises: a transmitting unit that transmits information including the processed learning data to an external learning system; The second pipeline comprises: a receiving unit that receives a machine learning model trained by the external learning system; a model deployment unit that deploys the machine learning model into an operational system; 10. The development system of claim 1, comprising:

3. The transmission unit transmitting the second pipeline including information necessary for executing the second pipeline; The development system of claim 2 .

4. The information necessary for executing the second pipeline includes a processing instruction trigger code for starting processing of the second pipeline. The development system of claim 3 .

5. The first pipeline comprises: converting the learning data into a data format suitable for a machine learning model of the external learning system, and transmitting the learning data in the converted data format to the external learning system by the transmitting unit; The development system of claim 2 .

6. Further comprising a conversion script for converting the data into a data format suitable for a machine learning model of the external learning system.

6. The development system of claim 5.

7. The second pipeline comprises: receiving a machine learning model trained by the external learning system, and converting the received machine learning model into a format usable by the operational system; The development system of claim 2 .

8. The model deploy unit: transmitting the machine learning model to the operational system; The development system of claim 2 .

9. The receiving unit Accepting a designation of a receiving format of a predetermined machine learning framework, and converting the received machine learning model into the predetermined machine learning framework of the designated receiving format; The development system of claim 7.

10. The first pipeline comprises: The communication device further includes a data set unit that prepares data to be sent to the transmitting unit and transfers the data. The development system of claim 2 .

11. a first pipeline that processes training data for training in an external training system and trains a machine learning model in the external training system using the processed training data; a second pipeline that deploys a machine learning model trained with the training data; an operational system for evaluating the machine learning model; A learning management system comprising:

12. The operation system includes: a receiving unit that receives a machine learning model from the second pipeline; a learning unit that causes the machine learning model received from the receiving unit to be re-learned; an evaluation unit that performs inference using the machine learning model received from the learning unit and calculates an evaluation result including a performance index; a management unit that manages the re-trained model received from the evaluation unit and transfers it to an external system; The learning management system of claim 11 , comprising:

13. The management unit When a re-learning command is received from the outside, the first pipeline is caused to re-execute the process. The learning management system of claim 12.

Images