Management device, management method, and program

Abstract

To provide a management device, a management method and a program capable of achieving efficient version management of learning data.SOLUTION: According to an embodiment, a management device includes a data processing part, a data management part and an evaluation part. The data processing part performs at least one pre-processing for generating a learning data set. The data management part preserves the generated learning data set. The evaluation part evaluates a model created by using the generated learning data set. The data management part temporarily preserves the generated learning data set, and determines whether or not to actually preserve the generated learning data set on the basis of an evaluation result of the model by the evaluation part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a management apparatus, a management method, and a program. [Background technology]

[0002] Machine learning / AI (artificial intelligence) models are created by executing learning algorithms using training data. When developing and operating a model using machine learning, a series of workflows are required, such as preprocessing of raw data used as training data, feature extraction, model creation, and model verification. The concept and methodology of such a workflow is called MLOps (Machine Learning Operation) and is implemented using a machine learning pipeline.

[0003] A machine learning pipeline is composed of multiple components. A component is something that realizes some function in developing and operating a model. For example, a component is something that realizes the function of creating a model. In such a machine learning pipeline, a model developer inputs the necessary parameters and issues an execution command to develop a model. In order to develop a highly accurate model, the parameters are changed many times and the model is reconstructed. Then, the model developer can compare the created models to select the one with the highest accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-127964 [Patent Document 2] JP 2020-57287 A Summary of the Invention [Problem to be solved by the invention]

[0005] When comparing the evaluation results of models created using a machine learning pipeline, it is sometimes necessary to check what kind of training data and parameters were used in models with high accuracy in the evaluation results. It is known to perform version management of such training data and parameters. This version management uses a method in which the training data used when creating a certain model (old training data) is left intact, while the training data used when creating another model (new data) is saved. In other words, by leaving both the old data and the new data, management is performed from old versions to new versions.

[0006] In the conventional data version management described above, a set of training data is managed separately each time a model is created, and even if there is duplication of training data, the duplicated parts are not removed and stored, which causes storage capacity to be strained. In addition, since it is not possible to grasp the differences between training data, it is not easy to compare the model creation conditions. In addition, in order to perform version management, model developers need to prepare source code that describes the processing contents for implementing version management of training data separately from the machine learning pipeline, and the implementation method differs depending on the model developer, resulting in coding errors and time waste. In addition, when data preprocessing consists of multiple steps, if preprocessing at one step fails, it is necessary to perform data preprocessing from scratch, which takes time to rerun.

[0007] The problem to be solved by the present invention is to provide a management device, a management method, and a program capable of realizing efficient version management of learning data. [Means for solving the problem]

[0008] The management device of the embodiment has a data processing unit, a data management unit, and an evaluation unit. The data processing unit performs at least one preprocessing to generate a learning dataset. The data management unit stores the generated learning dataset. The evaluation unit evaluates a model created using the generated learning dataset. The data management unit temporarily stores the generated learning dataset, and determines whether or not to permanently store the generated learning dataset based on a result of the evaluation of the model by the evaluation unit. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a functional block diagram showing an example of a functional configuration of a management device 1 according to the embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of components constituting a machine learning pipeline MLP according to the embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a data configuration of a learning data storage unit 41 according to the embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a data configuration of a model storage unit 42 according to the embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a data configuration of a metadata storage unit 43 according to the embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a data configuration of a source code storage unit 44 according to the embodiment. [Figure 7] 4 is a flowchart showing an example of a workflow of version management by the management device 1 according to the embodiment. [Figure 8] 6 is a flowchart showing an example of a workflow for adding a version management function by the management device 1 according to the embodiment. [Figure 9] FIG. 11 is a sequence diagram for explaining an operation when performing version management (initial operation) according to the embodiment. [Figure 10] FIG. 11 is a sequence diagram for explaining an operation when performing version management (addition of difference data) according to the embodiment. [Figure 11] FIG. 11 is a sequence diagram for explaining an operation when performing version management (deletion of differential data) according to the embodiment. [Figure 12]FIG. 11 is a sequence diagram for explaining an operation when performing version management (comparison) according to the embodiment. [Figure 13] FIG. 4 is a sequence diagram for explaining an operation when performing version management (model management) according to the embodiment. [Figure 14] FIG. 11 is a sequence diagram for explaining an operation when performing version management (model evaluation) according to the embodiment. [Figure 15] FIG. 11 is a sequence diagram for explaining an operation when performing version management (addition of a version management function) according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A management apparatus, a management method, and a program according to an embodiment will be described below with reference to the drawings.

[0011] The management device according to the embodiment performs version management of learning data used in creating a machine learning model. The management device also manages the created model, metadata indicating various information related to model creation, source code for creating each component constituting a machine learning pipeline, and the like.

[0012] [Overall configuration] 1 is a functional block diagram showing an example of a functional configuration of a management device 1 according to an embodiment. The management device 1 includes, for example, a control unit 10, an input interface 20, a display unit 30, and a storage unit 40. The control unit 10 includes, for example, an acquisition unit 11, a data processing unit 12, a learning unit 13, an evaluation unit 14, a management unit 15, and a display control unit 16.

[0013] Each functional unit of the control unit 10 is realized by a central processing unit (CPU) (computer) executing a program. In addition, some or all of the functional units of the control unit 10 may be realized by hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), or may be realized by software and hardware working together. The program may be stored in the storage unit 40 (a storage device having a non-transient storage medium) in advance, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed by mounting the storage medium in a drive device.

[0014] The acquisition unit 11 includes, for example, a parameter acquisition unit 111 and a raw data acquisition unit 112. The parameter acquisition unit 111 acquires various kinds of parameter information input by a user (for example, a model developer) via the input interface 20. The parameter information includes, for example, reference destination information and parameters for acquiring data required when creating a model, authentication information and a URL (Uniform Resource Locator) of a version control system such as Git required for data version control, and the like. The acquisition unit 11 may acquire the parameter information from an external device (not shown) connected to the management device 1 via a network.

[0015] The raw data acquiring unit 112 acquires raw data stored in the storage unit 40 by using information on a reference destination from which data acquired by the parameter acquiring unit 111 is acquired. This raw data includes raw data for learning and raw data for evaluation.

[0016] The data processing unit 12 performs preprocessing on the raw data for learning acquired by the raw data acquisition unit 112. Preprocessing of raw data refers to performing some kind of processing (editing, processing, merging, extraction, etc.) on raw data. Preprocessing of raw data refers to preparing collected raw data so that it is easy to learn, and creating learning data. Learning data refers to data for which all preprocessing has been completed. In other words, the data processing unit 12 performs at least one preprocessing to generate a learning dataset.

[0017] The learning unit 13 creates a model using the learning data (preprocessed data) created by the preprocessing of the data processing unit 12 and the parameters acquired by the parameter acquisition unit 111. The learning unit 13 creates the model by machine learning using a technique such as a neural network or SVM (Support Vector Machine).

[0018] The evaluation unit 14 uses the evaluation data to evaluate the model created by the learning unit 13. The evaluation unit 14 acquires raw data for evaluation from the learning data storage unit 41 using the evaluation information of the model acquired by the parameter acquisition unit 111, and calculates evaluation indices such as a correct answer rate (accuracy) and a confusion matrix (confusion matrix) using the evaluation data obtained by performing preprocessing on the raw data. The evaluation unit 14 judges whether the created model is pass / fail based on the calculated evaluation indices. For example, the evaluation unit 14 compares the calculated correct answer rate with a threshold value previously determined by the model developer, and judges whether the calculated correct answer rate exceeds the threshold value, thereby judging whether the model is pass / fail. That is, the evaluation unit 14 evaluates the model created using the generated learning dataset.

[0019] The management unit 15 includes, for example, a data management unit 151, a model management unit 152, a metadata management unit 153, and a source code management unit 154. The data management unit 151 performs version management of the learning data in each step of the workflow. The data management unit 151 is realized by additionally implementing a component having a version management function of the learning data to the function of the machine learning pipeline created in advance by the model developer. The component is executed by receiving, for example, a Git URL and authentication information passed as input to the machine learning pipeline in order to perform version management of the learning data.

[0020] That is, the data management unit 151 performs a process of saving the generated learning data set. The data management unit 151 temporarily saves the generated learning data set, and determines whether or not to permanently save the generated learning data set based on the evaluation result of the model by the evaluation unit 14. The data management unit 151 determines whether or not there is a difference between the learning data set newly generated by the preprocessing by the data processing unit 12 and the saved preprocessed learning data set. At least one preprocessing includes a first preprocessing and a second preprocessing, and the data processing unit 12 performs the second preprocessing on the temporarily saved first preprocessed first learning data set to generate a second preprocessed data set. The data management unit 151 temporarily saves the second preprocessed data. The data management unit 151 determines whether or not there is a difference between the first preprocessed data set and the second preprocessed data set. When the data management unit 151 determines that there is a difference between the first preprocessed data set and the second preprocessed data set, the data management unit 151 temporarily saves the second preprocessed data. If the model evaluation result is successful, the data management unit 151 determines to permanently store the generated learning dataset. If the model evaluation result is unsuccessful, the data management unit 151 determines not to permanently store the generated learning dataset, and discards the temporarily stored learning dataset. The data management unit 151 generates a branch for version management, and temporarily stores the generated learning dataset in the branch. The data management unit 151 temporarily stores the generated learning dataset each time each of a plurality of preprocessing processes is completed. The data management unit 151 stores metadata related to the model learning process.

[0021] The model management unit 152 manages the version of the model created by the learning unit 13. The metadata management unit 153 manages metadata. Metadata is data indicating the relationship between data such as a model, learning data, and parameters. The metadata management unit 153 manages metadata (hyperparameters, etc.) used when the learning unit 13 creates a model for each version of the model. The source code management unit 154 has a function of managing source code. The source code is a computer program indicating the content of the processing of each component constituting the machine learning pipeline. For example, the source code is a text file in which the content of the processing of a component having a function of creating a model is described in a programming language.

[0022] The display control unit 16 causes the display unit 30 to display the evaluation results and the like by the evaluation unit 14. In addition, the display control unit 16 causes the display unit 30 to display a GUI (Graphical User Interface) for receiving various inputs and instructions from the model developer.

[0023] The input interface 20 accepts various input operations by the model developer, and outputs an electrical signal indicating the content of the accepted input operation to the control unit 10. The input interface 20 is realized by, for example, a keyboard, a mouse, a touch panel, or the like.

[0024] The display unit 30 displays various information. For example, the display unit 30 displays a GUI that accepts various operations by the model developer. The display unit 30 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, a touch panel, or the like. The display unit 30 may be provided separately from the management device 1 and display various information by communicating with the management device 1. When the display unit 30 is realized by a touch panel, it may also have the function of the input interface 20 described above.

[0025] The storage unit 40 is a storage device such as a hard disk drive (HDD), a random access memory (RAM), or a flash memory. The storage unit 40 includes, for example, a learning data storage unit 41, a model storage unit 42, a metadata storage unit 43, and a source code storage unit 44. The learning data storage unit 41 stores preprocessed data, learning data, evaluation data, and the like. The model storage unit 42 stores a model created by the learning unit 13. The metadata storage unit 43 stores metadata used in creating the model. The source code storage unit 44 stores source code.

[0026] [Machine learning pipeline configuration] 2 is a diagram showing a list of components necessary for implementing a workflow according to an embodiment in a machine learning pipeline MLP and an example of the execution order of the components. In the machine learning pipeline MLP shown in FIG. 2, a total of seven components CP1 to CP7 are defined to be executed in order.

[0027] The component CP1 has a function of creating a branch (Create Branch). The component CP2 has a function of creating training data by performing preprocessing (Training Data Create). The component CP3 has a function of temporarily storing preprocessed data (Data commit). The component CP4 has a function of creating a model using training data (Model training). The component CP5 has a function of evaluating the created model (Model evaluation). The component CP6 has a function of permanently storing training data (Training Data merge). The component CP7 has a function of saving metadata (Metadata Save). In this embodiment, both temporary storage and permanent storage refer to physically storing various data in the storage unit 41. Temporary storage refers to temporarily (provisionally) storing data in the storage unit 41 before a final decision to store the data is made. Permanent storage refers to storing data in the storage unit 41 after a final decision to store the data is made (changing the temporarily stored data to a permanently stored state).

[0028] Among the components CP1 to CP7, the components CP1, CP3, CP6, and CP7 are components provided by the management device 1 to realize version management. On the other hand, the components CP2, CP4, and CP5 are components prepared by the model developer himself. The model developer can easily realize a function for version management by using the management device 1. Note that the machine learning pipeline MLP shown in FIG. 2 is an example, and may be composed of other components. For example, the number and execution order of components prepared by the model developer himself can be changed.

[0029] [Data structure of memory section] FIG. 3 is a diagram showing an example of a data configuration of the learning data storage unit 41 according to the embodiment. The learning data storage unit 41 stores information such as a pipeline execution ID, a commit ID, a folder name, a learning data file name, commit information, and a registration date. The pipeline execution ID is an identifier for distinguishing execution of the machine learning pipeline MLP. The pipeline execution ID is issued by the management device 1 (e.g., the control unit 10) every time the machine learning pipeline MLP is executed. The commit ID is an identifier for distinguishing storage of data in the learning data storage unit 41. The commit ID is issued by the management device 1 (e.g., the control unit 10, a management program of the learning data storage unit 41, etc.) every time data storage in the learning data storage unit 41 is executed. The folder name indicates the name of a storage location in the learning data storage unit 41. The learning data file name is the file name of the learning data. The commit information is information related to storage of data in the learning data storage unit 41. The registration date is the date on which the learning data is registered.

[0030] 4 is a diagram showing an example of the data configuration of the model storage unit 42 according to the embodiment. The model storage unit 42 stores information such as a folder name, a model file name, a model ID, and a creation date. The folder name indicates the name of a storage location in the model storage unit 42. The model file name is the file name of the model. The creation date is the date when the model was created.

[0031] 5 is a diagram showing an example of a data configuration of the metadata storage unit 43 according to the embodiment. The metadata storage unit 43 stores information such as a metadata ID, a model ID, a commit ID, an evaluation result, model information data, and evaluation data. The metadata ID is an identifier for distinguishing metadata. The evaluation result is the evaluation result of the model by the evaluation unit 14. The model information data is information explaining the model. The evaluation data is information indicating the evaluation data used in the evaluation by the evaluation unit 14.

[0032] 6 is a diagram showing an example of a data configuration of the source code storage unit 44 according to the embodiment. The source code storage unit 44 stores information such as a folder name corresponding to each function, source code, and registration date. The folder name corresponding to each function indicates the name of the storage location in the source code storage unit 44. The source code is a program that realizes each function. The registration date is the date when the source code is registered.

[0033] [Version control process workflow] The overall flow (workflow) of the version management process of the management device 1 will be described below. FIG. 7 is a flowchart showing an example of a workflow of the version management process by the management device 1 according to the embodiment. The management device 1 performs version management of various data by executing a workflow preset on a machine learning pipeline. The flowchart shown in FIG. 7 is executed based on an instruction to execute the workflow by the model developer via the input interface 20, for example. The flowchart shown in FIG. 7 will be described taking as an example a case where there are multiple pre-processing steps (1, ..., n).

[0034] First, the data management unit 151 creates a branch (step S101). A branch is a function for branching the process of model development on a machine learning pipeline. Since the branched branch does not affect the created model, it is possible to create a new model while leaving the created model as it is. For example, the data management unit 151 uses a pipeline execution ID as the name of the branch.

[0035] Next, the data processing unit 12 performs preprocessing on the learning raw data acquired by the raw data acquisition unit 112 (step S103). When step S103 is executed for the first time, the data processing unit 12 performs the first preprocessing (1) of the multiple preprocessing (1, ..., n).

[0036] Next, the data management unit 151 compares the preprocessed data of the most recent step S103 with the preprocessed data of the previous step S103 (preprocessed data temporarily stored in the learning data storage unit 41) and determines whether there is a difference between the two preprocessed data (step S105). When step S105 is executed for the first time, the previous preprocessed data does not exist, so it is determined that there is a difference. When the data management unit 151 determines that there is a difference between the two preprocessed data (step S105; YES), it temporarily stores the most recent preprocessed data in the learning data storage unit 41 (step S107). On the other hand, when the data management unit 151 determines that there is no difference between the two preprocessed data (step S105; NO), it does not temporarily store the most recent preprocessed data. A situation in which there is no difference between the two preprocessed data is, for example, a case in which no change in data occurs due to the preprocessing of the most recent step S103.

[0037] Next, the data processing unit 12 judges whether or not all pre-processing has been completed (step S109). If the data processing unit 12 judges that all pre-processing has not been completed (step S109; NO), the data processing unit 12 returns to step S103, performs the uncompleted pre-processing on the most recently pre-processed data, and repeats the subsequent processing.

[0038] On the other hand, if the data processing unit 12 determines that all preprocessing has been completed (step S109; YES), the learning unit 13 creates a model using the preprocessed data for which all preprocessing has been completed as learning data (step S111).

[0039] Next, the evaluation unit 14 evaluates the created model and judges whether the model passes or fails (step S113). If the evaluation unit 14 judges that the model passes (step S113; YES), the data management unit 151 permanently saves the changes in the most recent preprocessed data, for which all preprocessing has been completed, in the learning data storage unit 41 (step S115). Next, the metadata management unit 153 saves a commit ID, which is issued at the time of permanent saving and indicates the storage location of the learning data, as metadata in the metadata storage unit 43 (step S117). In addition, the model management unit 152 saves the created model in the model storage unit 42.

[0040] On the other hand, if the evaluation unit 14 judges that the model is unsuccessful (step S113; NO), the data management unit 151 discards the most recently preprocessed data for which all preprocessing has been completed (step S119). The data management unit 151 also deletes the preprocessed data temporarily stored in the learning data storage unit 41. This completes the processing of this flowchart.

[0041] Component CP1 of the machine learning pipeline MLP shown in FIG. 2 has a function to execute the process of step S101 described above. Component CP2 has a function to execute the process of step S103 described above. Component CP3 has a function to execute the process of steps S105 and S107 described above. Component CP4 has a function to execute the process of step S111 described above. Component CP5 has a function to execute the process of step S113 described above. Component CP6 has a function to execute the process of step S115 described above. Component CP7 has a function to execute the process of step S117 described above.

[0042] For example, when learning data temporarily stored in a branch is merged with learning data in main for learning data stored in the Git main repository managed for each machine learning pipeline, only the difference data between the newly stored learning data in main and the temporarily stored (or temporarily deleted) learning data is merged. After the learning data is merged with main, the temporarily stored (or temporarily deleted) learning data in the branch is deleted, so that the learning data existing in the repository managed for each machine learning pipeline does not have duplicate learning data, and deduplication can be realized. In this embodiment, both temporary deletion and final deletion refer to physically deleting various data from the storage unit 41. Temporary deletion refers to temporarily (provisionally) deleting data from the storage unit 41 before a final deletion decision is made. Final deletion refers to deleting data from the storage unit 41 after a final deletion decision is made (changing the temporarily deleted data to a final deletion state, completely erasing the data).

[0043] An example will be described in which the Git repository name where the learning data is saved is set to the pipeline ID, which is the identifier of the machine learning pipeline. The pipeline execution ID, which is the execution identifier of the machine learning pipeline, is created as the Git branch name for version management of the learning data. The pipeline execution ID is set to the same pipeline execution ID when the workflow of this embodiment is re-executed after preprocessing fails. The pipeline execution ID can uniquely determine which machine learning pipeline the learning data was used in. If the learning data is already permanently saved in the repository, it is determined whether there is a difference between the learning data permanently saved in the repository and the preprocessed data to be permanently saved that was used when creating a new model in the machine learning pipeline, and if there is a difference, the differential data is temporarily saved in a branch whose branch name is the pipeline execution ID. The differential data refers to a group of differential data between the preprocessed data that has already been temporarily saved and the preprocessed data to be temporarily saved.

[0044] If the preprocessed data partially deleted from the already permanently stored learning data is the difference portion, the difference data partially deleted from the learning data is temporarily stored. After the created model is evaluated by the evaluation unit, if the evaluation result is successful, the difference of the learning data used for learning is permanently stored. If the evaluation result is unsuccessful, the branch which is the difference information of the temporarily stored learning data is discarded. The learning data is temporarily deleted, and if the evaluation result is successful, the difference of the deleted learning data is permanently deleted, and if the evaluation result is unsuccessful, the branch of the temporarily deleted learning data is deleted. When the learning data is permanently saved (or permanently deleted), the commit ID which is the identifier of the saved location is saved as metadata in the metadata storage unit 43.

[0045] [Workflow for adding version control to machine learning pipelines] Next, a workflow for adding a component having a data version management function to a machine learning pipeline created in advance by a model developer will be described. Fig. 8 is a flowchart showing an example of a workflow for adding a version management function by the management device 1 according to the embodiment.

[0046] First, the parameter acquisition unit 111 acquires input parameters (access destination information and authentication information of the version management tool for learning data) input by the model developer via the input interface 20 (step S201).

[0047] Next, the data management unit 151 embeds the acquired input parameters in the component that performs the version management of data (step S203), thereby completing the creation of the component that performs the version management of data.

[0048] Next, the data management unit 151 determines whether the machine learning pipeline created by the model developer satisfies the conditions for adding a version management function (step S205). For example, the data management unit 151 determines whether the parameter arguments of the components included in the machine learning pipeline created by the model developer are in a predetermined format. These parameter arguments are set in order to use version management of data. If the model developer wishes to use version management, the model developer sets the parameter arguments in accordance with the rules predetermined for the machine learning pipeline. On the other hand, if the model developer does not wish to use version management, the model developer does not set these parameter arguments.

[0049] For example, a condition for adding a version management function is that the name of a parameter argument related to the learning data of a machine learning pipeline component is a predetermined name (e.g., "training_commit_data"). If a parameter argument name that meets this condition exists in a component created by a model developer, a component having a data version management function for the learning data is added before the learning component, and a new machine learning pipeline is created. Note that in this embodiment, a parameter argument is used as a condition for adding a version management function, but this is not limited to this, and other rules may be used.

[0050] When the data management unit 151 determines that the condition for adding the version management function is satisfied (step S205; YES), the data management unit 151 adds a component having a data version management function to the machine learning pipeline created by the model developer (step S207). Next, the data management unit 151 executes the machine learning pipeline having the version management function (step S209).

[0051] On the other hand, if the data management unit 151 determines that the condition for adding the version management function is not satisfied (step S205; NO), the data management unit 151 does not add a component with a version management function, and executes the machine learning pipeline created by the model developer (step S211). This completes the processing of this flowchart.

[0052] [Flow for version management of learning data (first time)] Next, the initial operation when performing version management of learning data will be described with reference to a sequence diagram (FIG. 9) showing data exchange between functional blocks of the management device 1. First, the data processing unit 12 transmits a pipeline execution ID to the data management unit 151 (S1). Next, the data management unit 151 creates a branch for saving the learning data in the learning data storage unit 41 under the name of the pipeline execution ID (S2). Next, the data processing unit 12 performs preprocessing to create learning data to be used for model creation (S3), and transmits the preprocessed data to the data management unit 151 (S4). The data management unit 151 temporarily saves the preprocessed data in the created branch in the learning data storage unit 41 (S5).

[0053] Next, after the learning unit 13 creates a model (learning step) and the evaluation unit 14 evaluates the model (evaluation step), if the evaluation of the model by the evaluation unit 14 passes, the data management unit 151 transmits a permanent storage command to the preprocessed data temporarily stored in the learning data storage unit 41, and the preprocessed data is permanently stored in the learning data storage unit 41 (S6). If the permanent storage process ends normally, the data management unit 151 transmits the commit ID received from the learning data storage unit 41 to the metadata management unit 153 (S7). The metadata management unit 153 stores the received commit ID together with model information (hyperparameters, etc.) as metadata in the metadata storage unit 43 (S8). On the other hand, if the evaluation of the model by the evaluation unit 14 fails, the data management unit 151 transmits a command to the learning data storage unit 41 to discard the temporarily stored preprocessed data (S9). As a result, the preprocessed data temporarily stored in the metadata storage unit 43 is discarded.

[0054] [Flow for version management of learning data (adding differential data)] Next, the operation of adding differential data when performing version management of learning data will be described with reference to a sequence diagram (FIG. 10) showing data exchange between functional blocks of the management device 1. First, the data processing unit 12 performs preprocessing to create learning data used for model creation (S11), and transmits the preprocessed data and the pipeline execution ID to the data management unit 151 (S12). Next, the data management unit 151 acquires past learning data stored in the learning data storage unit 41, compares the new preprocessed data transmitted from the data processing unit 12 with the past learning data, and determines whether there is a difference between the two (S13). Here, since differential data exists (the new preprocessed data includes new data that does not exist in the past learning data), the data management unit 151 temporarily stores the differential data in the branch created in the learning data storage unit 41 (S14).

[0055] Next, after the learning unit 13 creates a model (learning step) and the evaluation unit 14 evaluates the model (evaluation step), if the evaluation of the model by the evaluation unit 14 passes, the data management unit 151 transmits a permanent storage command to the preprocessed data (differential data) temporarily stored in the learning data storage unit 41, and thus the data is permanently stored in the learning data storage unit 41 (S15). If the permanent storage process ends normally, the data management unit 151 transmits the commit ID received from the learning data storage unit 41 to the metadata management unit 153 (S16). The metadata management unit 153 registers the received commit ID together with model information (hyperparameters, etc.) as metadata in the metadata storage unit 43 (S17). On the other hand, if the evaluation of the model by the evaluation unit 14 fails, the data management unit 151 transmits a command to the learning data storage unit 41 to discard the temporarily stored preprocessed data (differential data) (S18). As a result, the preprocessed data (differential data) temporarily stored in the metadata storage unit 43 is discarded.

[0056] [Flow for version management of learning data (deleting differential data)] Next, the operation of deleting differential data when performing version management of learning data will be described with reference to a sequence diagram (FIG. 11) showing data exchange between functional blocks of the management device 1. First, the data processing unit 12 performs preprocessing to create learning data to be used for model creation (S21), and transmits the preprocessed data and the pipeline execution ID to the data management unit 151 (S22). Next, the data management unit 151 acquires past learning data stored in the learning data storage unit 41, compares the new preprocessed data transmitted from the data processing unit 12 with the past learning data, and determines whether there is a difference between the two (S23). Here, since differential data exists (at least a part of the past learning data has been deleted in the new preprocessed data), the data management unit 151 provisionally deletes the differential data from the branch created in the learning data storage unit 41 (S24).

[0057] Next, after the learning unit 13 creates a model (learning step) and the evaluation unit 14 evaluates the model (evaluation step), if the evaluation of the model by the evaluation unit 14 passes, the data management unit 151 transmits a definitive deletion command to the preprocessed data (differential data) temporarily stored in the learning data storage unit 41, and thus the definitive deletion from the learning data storage unit 41 is performed (S25). If the definitive deletion ends normally, the data management unit 151 transmits the commit ID received from the learning data storage unit 41 to the metadata management unit 153 (S26). The metadata management unit 153 registers the received commit ID together with model information (hyperparameters, etc.) as metadata in the metadata storage unit 43 (S27). On the other hand, if the evaluation of the model by the evaluation unit 14 fails, the data management unit 151 transmits a command to restore the temporarily deleted preprocessed data (differential data) to the learning data storage unit 41 (S28). As a result, the preprocessed data (differential data) temporarily deleted in the metadata storage unit 43 is restored.

[0058] [Comparison process flow for version management of learning data] Next, the operation of the comparison process when performing version management of learning data will be described with reference to a sequence diagram (FIG. 12) showing data exchange between functional blocks of the management device 1. First, the data management unit 151 transmits a metadata ID to the metadata management unit 153 in order to obtain a pipeline execution ID that identifies a branch that stores past learning data (S31). The metadata management unit 153 obtains a pipeline execution ID corresponding to the metadata ID received from the metadata storage unit 43 (S32) and transmits it to the data management unit 151. The data management unit 151 obtains past learning data (previously stored learning data) from the learning data storage unit 41 based on the obtained pipeline execution ID (S33).

[0059] Next, the data management unit 151 compares the acquired past learning data with the preprocessed learning data newly acquired from the data processing unit 12, and determines whether there is a difference between the two (S34). If there is difference data, the data management unit 151 creates a branch with the acquired pipeline execution ID as the branch name and temporarily stores it in the learning data storage unit 41 (S35). As a result, the data management unit 151 acquires from the learning data storage unit 41 a commit ID that indicates the location where the difference data is temporarily stored.

[0060] [Model management process flow for version management of training data] Next, the operation of the model management process when managing the version of the learning data will be described with reference to a sequence diagram (FIG. 13) showing data exchange between functional blocks of the management device 1. First, the data management unit 151 acquires the preprocessed data temporarily stored in a branch created in the learning data storage unit 41 (S41). Next, the data management unit 151 transmits the acquired preprocessed data to the learning unit 13 (S42).

[0061] The learning unit 13 stores the model created by learning the preprocessed data received from the data management unit 151 in the model storage unit 42 (S43). Here, the learning unit 13 stores in the model storage unit 42 the model that has been evaluated by the evaluation unit 14 and passed the evaluation. If the saving is completed normally, the learning unit 13 acquires the model ID from the model storage unit 42. Next, the learning unit 13 transmits the metadata (hyperparameters, etc.) used when creating the model and the model ID to the metadata management unit 153 (S44).

[0062] Next, the metadata management unit 153 stores the metadata received from the learning unit 13 in the metadata storage unit 43 (S45). If the metadata storage has ended normally, the metadata management unit 153 obtains a metadata ID from the metadata storage unit 43 and transmits it to the learning unit 13. The learning unit 13 transmits the metadata ID and the model ID to the data management unit 151.

[0063] [Model evaluation process flow for version management of training data] Next, the operation of the model evaluation process when performing version management of the learning data will be described with reference to a sequence diagram (FIG. 14) showing data exchange between the functional blocks of the management device 1. First, the data management unit 151 acquires raw evaluation data stored in the learning data storage unit 41 (S51). Next, the data management unit 151 performs preprocessing on the acquired raw evaluation data (S52), and transmits the preprocessed data to the evaluation unit 14 as evaluation data (S53).

[0064] Next, the evaluation unit 14 transmits the model ID of the model to be evaluated to the model management unit 152 (S54). The model management unit 152 acquires the model corresponding to the model ID received from the evaluation unit 14 from the model storage unit 42 (S55) and transmits it to the evaluation unit 14. The evaluation unit 14 performs model evaluation using the model received from the model management unit 152 and the evaluation data received from the data management unit 151 (S56). The evaluation unit 14 transmits the evaluation result to the metadata management unit 153 (S57). The metadata management unit 153 stores the evaluation result received from the evaluation unit 14 in the metadata storage unit 43 (S58). When the storage of the evaluation result (metadata) is normally completed, the metadata management unit 153 acquires the metadata ID from the metadata storage unit 43 and transmits it together with the model ID to the evaluation unit 14. The learning unit 13 transmits the metadata ID and the model ID to the data management unit 151. Note that models that have failed the evaluation may be deleted from the model storage unit 42, but the metadata (evaluation results) may be left intact. For example, failure information may be stored in the metadata (evaluation results).

[0065] [Flow to automatically add a component that manages versioning of learning data] Next, the operation of automatically adding a component that performs version management of learning data will be described with reference to a sequence diagram (FIG. 15) showing data exchange between functional blocks of the management device 1. First, the parameter acquisition unit 111 acquires input parameters (such as a storage destination of learning data and authentication information of a storage tool) input by a model developer, and transmits them to the data management unit 151 (S61). The authentication information includes, for example, an IP address and a user name of the storage tool, a password (access key or secret), and the like. Next, the data management unit 151 uses the authentication information received from the parameter acquisition unit 111 to create a component having a function similar to a source code for version management of learning data (S62). For example, the data management unit 151 embeds authentication information in a component having a version management function.

[0066] Next, the data management unit 151 acquires information required to determine where the created component should be added to the machine learning pipeline created in advance by the model developer from the learning unit 13. Specifically, the data management unit 151 acquires the name of an input parameter related to the learning data of the component having a learning function. The data management unit 151 determines whether the argument name of the parameter argument of the learning data, which is one of the input parameters of the component having a learning function, is a predetermined argument name (whether the version management function addition condition is satisfied or not) (S63). For example, the argument name is previously determined as "training_commit_data", and the data management unit 151 determines whether the argument name of the learning data of the component having a learning function created in advance by the model developer is "training_commit_data". If the argument name of the parameter argument matches the predetermined argument name, the data management unit 151 adds a component having a function of version management of the learning data before the component having a learning function created by the model developer, and creates a machine learning pipeline (S64). Next, the data management unit 151 executes the created machine learning pipeline with the version management function (S65). On the other hand, if the argument name of the parameter argument does not match the predetermined argument name, the data management unit 151 executes the machine learning pipeline created by the model developer as is, without adding a component having a function of version management of the learning data (S66).

[0067] According to the management device 1 of the embodiment configured as described above, by implementing version management of learning data in combination with the machine learning pipeline, it is possible to realize efficient version management of learning data. In addition, it is possible to confirm metadata and learning data that are the basis for deduplication of learning data and comparison between models created by the machine learning pipeline. When performing version management of learning data, by having a function of temporarily saving preprocessed data each time preprocessing of raw learning data is performed, even if the machine learning pipeline fails during execution, the process up to the preprocessing where the temporary saving was performed can be omitted, and the time required for the preprocessing of learning data can be reduced. In addition, by realizing the same function in a component of the machine learning pipeline without creating source code describing the processing contents for realizing version management for learning data, it is no longer necessary to program a code for version management of learning data, and it is possible to reduce time and eliminate coding errors in model development.

[0068] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0069] 1...management device, 10...control unit, 11...acquisition unit, 111...parameter acquisition unit, 112...raw data acquisition unit, 12...data processing unit, 13...learning unit, 14...evaluation unit, 15...management unit, 151...data management unit, 152...model management unit, 153...metadata management unit, 154...source code management unit, 16...display control unit, 20...input interface, 30...display unit, 40...storage unit, 41...learning data storage unit, 42...model storage unit, 43...metadata storage unit, 44...source code storage unit

Claims

1. A data processing unit that performs at least one pre-processing to generate a training data set; A data management unit that performs processing to store the generated learning data set; An evaluation unit that evaluates a model created using the generated learning dataset; Equipped with The data management unit Temporarily storing the generated learning data set; determining whether or not to permanently store the generated learning data set based on a result of the evaluation of the model by the evaluation unit; Management device.

2. The data management unit determines whether or not there is a difference between a learning data set newly generated by the preprocessing by the data processing unit and the stored preprocessed learning data set. The management device according to claim 1 .

3. The at least one pretreatment includes a first pretreatment and a second pretreatment, The data processing unit performs the second preprocessing on the temporarily stored first preprocessed first learning data set to generate a second preprocessed data set; The data management unit temporarily stores the second preprocessed data. The management device according to claim 1 .

4. The data management unit determines whether or not there is a difference between the first pre-processed data set and the second pre-processed data set. The management device according to claim 3 .

5. When the data management unit determines that there is a difference between the first preprocessed data set and the second preprocessed data set, the data management unit temporarily stores the second preprocessed data. The management device according to claim 4.

6. The data management unit determines to store the generated learning data set when the evaluation result of the model is acceptable. The management device according to claim 1 .

7. When the evaluation result of the model is unsatisfactory, the data management unit determines not to permanently store the generated learning data set, and discards the temporarily stored learning data set. The management device according to claim 6.

8. The data management unit generates a branch for performing version management, and temporarily stores the generated learning data set in the branch. The management device according to claim 1 .

9. The data management unit temporarily stores the generated learning data set each time each of the plurality of preprocessing processes is completed. The management device according to claim 1 .

10. The data management unit stores metadata related to the model learning process. The management device according to claim 1 .

11. The computer performing at least one pre-processing step to generate a training data set; storing the generated training data set; Evaluating a model created using the generated learning dataset.

1. A method of management comprising: Storing the training data set includes: Temporarily storing the generated learning data set; and determining whether to permanently store the generated learning data set based on an evaluation result of the model. Management method.

12. On the computer, performing at least one pre-processing step to generate a training data set; Storing the generated learning data set; Evaluating a model created using the generated learning data set; A program, Storing the training data set includes: Temporarily storing the generated learning data set; and determining whether to permanently store the generated learning data set based on an evaluation result of the model. program.

Images