Data management and distribution method and device of machine learning platform and storage medium

Abstract

The invention relates to a data management and distribution method and device for a machine learning platform and a storage medium, and the method comprises the steps that a data system is controlled to manage and share data based on a Git version, the data storage comprises a plurality of machine learning training cluster available areas, each available area is provided with a two-stage storage structure, and the two-stage storage structure is provided with a plurality of data storage units; the primary storage is used for mirroring a git source station; the second-level storage is used for carrying out file storage according to commit id, each data version is stored in the second-level storage, each version has the corresponding commit id, and different training tasks share and call the second-level storage. Data are managed and shared through a Git version control system. In the face of multiple training task requirements, version data required by the tasks are directly called based on the custom resource CR under the Kubernetes platform, sharing of storage space and distribution operation among multiple tasks is achieved, and the data storage and transmission efficiency is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and in particular to a method, device, and storage medium for data management and distribution of a machine learning platform. Background Art

[0002] With the rapid development of AI technology represented by large models, the volume of training data and model data has increased rapidly. How to effectively manage data, including precise version control, efficient data distribution, and convenient training and use, has become a new challenge facing AIInfra technology.

[0003] In existing technologies, when multiple versions of data are used simultaneously, multiple cross-cluster transmission operations are often required, resulting in repeated data storage work and insufficient transmission efficiency. Summary of the invention

[0004] In view of this, the present disclosure proposes a method, device and storage medium for data management and distribution of a machine learning platform.

[0005] This method manages and shares data based on the Git version control system. When faced with multi-zone training task requirements, a two-level storage structure is set up in each zone. The first-level storage is used to mirror the git source station, and the second-level storage is used to store files by commitid. Argo Workflow is used for task DAG scheduling. By sharing storage space and distribution operations among multiple tasks, the efficiency of data storage and transmission can be effectively improved.

[0006] According to one aspect of the present application, a method for data management and distribution of a machine learning platform is provided, comprising: Based on the Git version control data system to manage and share data, data storage includes multiple machine learning training cluster availability zones, each of which is set up with a two-level storage structure, where the first-level storage is used to mirror the git source station; The secondary storage is used to store files according to commit id, and each data version is stored in the secondary storage, and each version has a corresponding commit id identifier. Different training tasks share the call to the secondary storage.

[0007] In a possible implementation, Argo Workflow is used to schedule the task DAG. When data needs to be called in a machine learning training, the commit id of the data to be called is determined. According to the commit id to be called, determine whether the data to be called is ready through the custom resources of Kubernetes; When the data to be called is prepared, the currently executed machine learning training uses the data to be called for training.

[0008] In a possible implementation, when determining whether the data to be called is ready through a custom resource of Kubernetes according to the commit id to be called, the method further includes: When the data to be called is not ready, check whether the CR corresponding to the commit id to be called is being executed according to the commit id to be called, and obtain a judgment result; When the CR is being executed, suspend the CR to be executed; When the CR to be executed can be executed, the data to be called is ready.

[0009] In a possible implementation, the determination result of checking whether the CR corresponding to the commit id to be called is obtained during execution further includes: When the CR is not being executed, the CR application is submitted and CR data preparation is performed.

[0010] In a possible implementation, the CR data preparation includes determining whether a specified commit id is obtained from the primary storage; When a specified commit ID is obtained from the primary storage, based on the obtained specified commit ID, data corresponding to the obtained commit ID is used as the secondary storage; When the specified commit id is not obtained, a fetch operation of the primary storage is triggered to obtain the specified commit id again.

[0011] In a possible implementation, when the specified commit ID is not obtained, a fetch operation of the primary storage is triggered, including: When the specified commit id is not obtained, before triggering the fetch operation of the primary storage, check whether there is a fetch operation of the primary storage being executed; When there is a fetch operation of the primary storage being executed, the CR enters a waiting mode; When the fetch operation of the primary storage is completed, the CR that enters the waiting mode obtains the specified commit id again.

[0012] In a possible implementation, when data needs to be called in the machine learning training, ArgoWorkflow is used for task scheduling, and a training task is described as a DAG. When the DAG is executed to the data initialization node, it is checked whether the data preparation is completed; The data initialization node of the DAG is implemented based on CR.

[0013] In a possible implementation, the CR preparation completion status check is implemented by a custom resource controller; The state of the CR includes at least one of an initial state, first-level data preparation in progress, first-level data preparation completed, data preparation failed, and data preparation completed.

[0014] According to another aspect of the present application, a device for data management and distribution of a machine learning platform is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above method.

[0015] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0016] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0018] Figure 1 A block diagram showing the secondary storage structure of the available area corresponding to the training cluster of an embodiment of the present application is shown; Figure 2 A schematic diagram of the DAG execution process of an embodiment of the present application is shown; Figure 3 A schematic diagram showing a specific logic flow of a custom controller according to an embodiment of the present application; Figure 4 A block diagram showing the device structure of data management and distribution of a machine learning platform according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0020] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0021] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0022] Figure 1 The method of data management and distribution of a machine learning platform of an embodiment is shown. In the method of this embodiment, based on the Git version control data system management and sharing of data, the data storage includes multiple machine learning training cluster available areas, each of which is provided with a two-level storage structure, wherein the first-level storage is used to mirror the git source station, so as to maintain a complete Git source station mirror in the available area corresponding to the training cluster. The number of available areas for specific data storage corresponds to the number of clusters for machine learning training using the data stored on the storage platform. Of course, in the actual processing process, the specific number of available areas can be set in advance, or the platform can automatically divide the corresponding available areas when the machine learning is started, and the corresponding processing method can be executed according to the needs of the practical process. Moreover, for the number of specific available areas, at least 2 or more, it can also be more, such as 3, 4, 5, ..., even more than ten or more than dozens, hundreds are all achievable. It can also be set according to the operating capacity of the data storage management platform.

[0023] A secondary storage structure is set up in each availability zone, where the primary storage is used to mirror the git source station; the secondary storage is used to store files by commit id, and each data version is stored in the secondary storage, and each version has a corresponding commit id. Different training tasks in a single training cluster share the secondary storage in the availability zone corresponding to the training cluster. During the call process, the source data is the mirror data obtained by mirroring the git source station data in the primary storage of the availability zone corresponding to the training cluster; the secondary storage is the version data distributed based on the mirror data of the primary storage with the version as the identifier. In the scheme of this embodiment, a secondary storage structure is set up to store version data called by the training cluster. Different training tasks in the training cluster can call data version backups that have been called by other tasks, so that multiple training tasks in the training cluster can call each other's shared version data, reducing the amount of data storage and increasing the speed of data preparation and calling, thereby improving machine learning and training efficiency.

[0024] Specifically, the git source station is mirrored in the primary storage of the available zone corresponding to the training cluster. The gitfetch operation is performed periodically to achieve synchronization between the primary storage and the git source station. In a specific embodiment of the present application, a scheduled task is set through the crontab instruction, and a git fetch operation is performed at 2 am every day to achieve synchronization between the primary storage and the git source station.

[0025] Among them, the operation cycle of git fetch can be adjusted according to the actual task requirements and is not specifically limited in this application.

[0026] The entire storage platform is divided into a main availability zone and availability zones corresponding to multiple specific training clusters. A main availability zone can be set on the data platform, in which git source station data is stored. There are multiple availability zones of corresponding numbers for multiple training clusters. Each availability zone is used to store the mirror repository of the git source station and the data called during the training of the corresponding training cluster task. Specifically, in one embodiment of the present application, in availability zone A corresponding to training cluster A, the first-level storage takes gitrepo path as the unit, and performs snapshot cache of git source data; the second-level storage takes git commit id as the unit, and performs snapshot cache of git data. For a main availability zone, there can be multiple training clusters and availability zones corresponding to the training cluster at the same time; when more than one training task in a training cluster uses the same version of data, based on the specified commitid, the version data in the secondary storage can be called by multiple training tasks, thereby realizing the distribution and sharing of version data among multiple tasks.

[0027] This application uses Argo Workflow to schedule training tasks. For a training task in a training cluster, the different execution stages of a training task, such as data preparation and training execution, are described as a directed acyclic graph (DAG). The steps of the task are described as nodes, and the dependencies between tasks are described as directed edges; the dependencies are implemented by specifying the output of one node as the input of another node. In this DAG, each node has specific execution logic and execution conditions. A special data initialization node is set at the starting point of the DAG for preparing training data.

[0028] Among them, the data initialization node is defined through the custom resource (CR) of Kubernetes. Each CR uses the two fields of git repo path and commit id to uniquely identify the specific version data, where the git repo path field represents the primary storage of specific data, and the commit id represents the secondary storage of specific data. For tasks with the same git repo path field but different commit ids, only the data path of the primary storage is shared, and the version data of the secondary storage is not shared. Perform a checkout operation in the primary storage, take out the specified commit id, and obtain the corresponding version data from the primary storage image based on the extracted commit id, and use the version data corresponding to the obtained commit id as the secondary storage. When multiple training tasks use the same CR, as long as the CR has been reached, the data can be reused, thereby achieving the purpose of sharing the CR node in multiple DAGs, and then continue with the subsequent process of the DAG. Specifically, the Argo Workflow DAG execution process is as follows: Figure 2 As shown in the figure, when DAG starts to execute to the data initial node CR, the Kubernetes API system first checks whether the CR is ready (that is, data preparation is completed). If the CR already exists and the status is completed, the system directly executes the downstream node of the training task; if it is not ready, wait for the data to be pulled; if the CR is not ready, the data pull is not completed, or it does not exist (no other tasks use it). Start the initialization node, continue to pull and prepare data, and recheck the CR status. The CR status is mainly divided into the initial state of data being declared (Declared), the first-level data preparation is in progress (Fetching), the first-level data preparation is completed (FetchchCompleted), the data preparation fails (Failed), and the data preparation is completed (Ready).

[0029] When a CR is created, it indicates that a training task declares that it needs a specific version of data. For a CR, a specific version of data can be used normally only when the CR status is Ready. The called specific version of data is completely placed in the secondary storage in the available zone. If other training tasks subsequently pull the same version of data, they can directly call it in the secondary storage without repeated data pulling.

[0030] Furthermore, a custom controller is implemented based on the Kubernetes API extension mechanism to process the CR, prepare the specific version data declared in the CR, and feed back the status changes during the processing to the CR so that other services on Kubernetes can also obtain the CR status.

[0031] Specifically, Figure 3 As shown in the figure, the specific logic of the custom controller when checking whether the CR is ready is as follows: when there is version data to be called, specify a CR according to the git repo path and commit id provided by the training task, and determine whether the CR is in the Declared state; if the CR is in the Declared state, perform a git checkout operation on the primary storage; if the CR is not in the Declared state, first update the Declared state of the CR to True, and then perform a git checkout operation on the primary storage.

[0032] Then check whether the checkout operation is successful. If successful, update the CR Ready status to True and the Fetching status to False. The CR data is ready. If the checkout operation fails due to data not being synchronized, the fetch operation of the primary storage is triggered to update the git mirror data of the primary storage from the git source station. Preferably, before executing the fetch operation, the controller first determines whether the CR is in the Fetching state, that is, whether there is a fetch operation of the primary storage in progress, and waits for the current fetch operation to be completed; If the current CR is not in the Fetching state, that is, there is no fetch operation in progress, check the FetchCompleted state of the CR. When the FetchCompleted state is True, directly perform the checkout operation; otherwise, update the FetchCompleted state to True, and then perform the fetch operation on the primary storage.

[0033] When fetching succeeds, the FetchCompleted status of CR is updated to True, and the Fetching status of CR is updated to False; when fetching fails, the Failed status of CR is updated to True, and the Fetching status of CR is updated to False; During the CR preparation process, there may be other training tasks that also depend on the CR with the same commit ID. In this case, these tasks will also enter the suspended state and wait for the CR to be ready. When the CR status changes to completed, based on the Argo Workflow task activation mechanism, the system automatically activates the suspended tasks and continues DAG scheduling and execution.

[0034] For the tasks in the suspended state that subscribe to the CR, the data corresponding to the CR is reused in sequence in the suspended queue order.

[0035] When the CR data preparation is completed in the above process, the training task calls the version data of the CR identified by the specified git repo path and commit id to complete the task training.

[0036] Provide the system with the expected state of the CR, that is, the specified commit ID, in the form of a YAML configuration file, and match the actual state of the CR with the expected state, so that the current actual state of the CR reaches the expected state.

[0037] The process of the actual CR status approaching the expected status is a conventional technical process based on the Kubernetes platform and will not be described in detail in this application.

[0038] In one possible approach, When multiple versions of data are in use at the same time, only one cross-cluster data transfer git fetch is required, and then multiple versions are distributed through secondary storage within the training cluster, avoiding the network load caused by multiple cross-cluster transfer operations.

[0039] After secondary storage according to data version, multiple tasks can share the same version of data, reducing duplicate data storage work.

[0040] In one possible implementation, multiple tasks share the same version data. Based on the required commit id, check whether the version data corresponding to the required commit id can be called directly. If the required data is called by other tasks, the current task that needs the version data will be suspended. When the previous task that uses the version data is completed, other suspended tasks that subscribe to the version data will be executed in sequence according to the preset execution order to complete data sharing, avoid multiple loading of the same version data when multiple tasks are executed, reduce the work of repeated data storage, and thus improve the efficiency of data storage.

[0041] Furthermore, according to another aspect of the present disclosure, a device for data management and distribution of a machine learning platform is provided. Figure 4The device for data management and distribution of the machine learning platform of the embodiment of the present disclosure includes a device configured to implement any of the aforementioned methods when executing executable instructions.

[0042] It should be pointed out here that the data management and distribution devices of the machine learning platform of the embodiment of the present disclosure can be connected through a bus or in other ways, which is not specifically limited here.

[0043] The memory 110 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the method for data management and distribution of the machine learning platform in the embodiment of the present disclosure. The processor 120 executes the various functional applications and data processing of the device for data management and distribution of the machine learning platform by running the software programs or modules stored in the memory.

[0044] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, any of the above-mentioned methods is implemented.

[0045] According to the data management and distribution method of the machine learning platform of the above embodiment of the present application, data can be managed and shared through the Git version control system. When facing the requirements of multi-availability zone training tasks, a two-level storage structure is set up in each availability zone. The first-level storage is used to mirror the git source station, and the second-level storage is used to store files according to the commit id, and Argo Workflow is used for task DAG scheduling. When multiple versions of the data are in use at the same time, only one cross-cluster data transmission git fetch is required, and then it is distributed into multiple versions through the second-level storage in the training cluster to avoid the network load caused by multiple cross-cluster transmission operations. After the second-level storage is performed according to the data version, multiple tasks can share the same version of data, reducing repeated data storage work. The present application effectively improves the efficiency of data storage and transmission by realizing storage space and distribution operation sharing among multiple tasks.

[0046] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data management and distribution method for a machine learning platform, characterized in that: Based on the Git version control data system to manage and share data, data storage includes multiple machine learning training cluster availability zones, each of which is set up with a two-level storage structure, where the first-level storage is used to mirror the git source station; The secondary storage is used to store files according to commit id, and each data version is stored in the secondary storage, and each version has a corresponding commit id identifier. Different training tasks share the call to the secondary storage.

2. The data management and distribution method of the machine learning platform according to claim 1, characterized in that: Use ArgoWorkflows to schedule training tasks; the steps include: When data needs to be called in a machine learning training, the commit id of the data to be called is determined; According to the commit id to be called, determine whether the data to be called is ready through the custom resources of Kubernetes; When the data to be called is prepared, the currently executed machine learning training uses the data to be called for training.

3. The data management and distribution method of the machine learning platform according to claim 2, characterized in that: According to the commit id identifier to be called, determining whether the data to be called is ready through the custom resource of Kubernetes also includes: When the data to be called is not ready, check whether the CR corresponding to the commit id to be called is being executed according to the commit id to be called, and obtain a judgment result; When the CR is being executed, suspend the CR to be executed; When the CR to be executed can be executed, the data to be called is ready.

4. The data management and distribution method of the machine learning platform according to claim 3, characterized in that: The judgment result of checking whether the CR corresponding to the commit id identifier to be called is obtained during execution also includes: When the CR is not being executed, the CR application is submitted and CR data preparation is performed.

5. The data management and distribution method of the machine learning platform according to claim 4, characterized in that: The CR data preparation includes: Determine whether the specified commit id is obtained from the primary storage; When a specified commit ID is obtained from the primary storage, based on the obtained specified commit ID, data corresponding to the obtained commit ID is used as the secondary storage; When the specified commit id is not obtained, a fetch operation of the primary storage is triggered to obtain the specified commit id again.

6. The data management and distribution method of the machine learning platform according to claim 5, characterized in that: When the specified commit id is not obtained, the fetch operation of the primary storage is triggered, including: When the specified commit id is not obtained, before triggering the fetch operation of the primary storage, check whether there is a fetch operation of the primary storage being executed; When there is a fetch operation of the primary storage being executed, the CR enters a waiting mode; When the fetch operation of the primary storage is completed, the CR that enters the waiting mode obtains the specified commit id again.

7. The data management and distribution method of the machine learning platform according to any one of claims 1 to 6, characterized in that: When data needs to be called during the machine learning training, Argo Workflow is used to schedule tasks and describe a training task as a DAG. When the DAG is executed to the data initialization node, check whether data preparation is completed. The data initialization node of the DAG is implemented based on CR.

8. The data management and distribution method of the machine learning platform according to any one of claims 1 to 6, characterized in that: The CR preparation completion status check is implemented by a custom resource controller; The state of the CR includes at least one of an initial state, first-level data preparation in progress, first-level data preparation completed, data preparation failed, and data preparation completed.

9. A data management and distribution device for a machine learning platform, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 8 when executing the executable instructions.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images