Multi-cluster based model training system, method, device and storage medium

Abstract

The embodiment of the present invention provides a model training system, method, device and storage medium based on multiple clusters, which receives data processing tasks related to model training sent by a client through a task processing unit, and sends operation instructions of the data processing tasks to the corresponding cluster according to the data processing tasks, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instructions, and stores the processing results in the storage unit and / or returns them to the client through the task processing unit. By applying a storage unit for storing data and / or processing results that need to be shared across clusters in a model training system based on multiple clusters, the storage unit is used to realize the problem of data sharing and synchronization of multiple clusters during the model training process, avoids the waste of resources caused by data migration and data storage backup, avoids the leakage of source data, solves the problem of slow synchronization when the data volume is large, and improves the efficiency of model training.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a multi-cluster based model training system, method, device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the demand for model training tasks is gradually increasing. Model training can be done through physical machines or clusters. Clusters can provide CPU, GPU, memory and other resources for model training through resource scheduling, which can improve machine utilization. Therefore, choosing clusters for model training is a development trend.

[0003] Currently, the clusters used for model training mainly include Yarn (Yet Another Resource Negotiator) clusters and Kubernetes (K8S for short, Google's open source container orchestration engine) clusters, etc. They can each support the training tasks of multiple engines and provide stable service support when deployed as training clusters alone. In the existing cluster-based training platform, user data is in a cluster or storage. Only one cluster is needed to meet the user's training needs and save the results to the cluster for easy user management. The reading and saving speed of user data in the same cluster is faster. From the perspective of efficiency, using one cluster in a single business can solve most of the user's needs.

[0004] However, with the increasing requirements for service diversification and training efficiency, a single cluster service can no longer meet users' more personalized needs. Users usually store data in different clusters or storage resources according to different business lines. Each cluster data has its own dedicated business and characteristics, and has a specific user group. When users need to switch clusters, they usually need to download and upload data from different clusters to complete data migration, which places great demands on the storage space of user devices and is inefficient. Alternatively, the platform administrator can copy data between clusters, but the copied data is stored in multiple copies, resulting in a waste of storage resources and the risk of privacy data leakage. Summary of the invention

[0005] Embodiments of the present invention provide a multi-cluster based model training system, method, device and storage medium to solve the problem of efficient and secure sharing of cluster data when performing model training based on multiple clusters, thereby improving the efficiency of model training.

[0006] In a first aspect, an embodiment of the present invention provides a multi-cluster based model training system, comprising: a client, a storage unit, at least two clusters, and a task processing unit corresponding to each of the clusters; each of the task processing units is communicatively connected to the corresponding cluster, the client, and the storage unit; each of the clusters is communicatively connected to the storage unit; the clusters include resources required for data processing tasks;

[0007] The client is used to send a data processing task to the task processing unit; wherein the data processing task includes a data acquisition task or a model training task;

[0008] The task processing unit is used for sending an operation instruction of the data processing task to the corresponding cluster according to the received data processing task;

[0009] The cluster is used to perform data processing on the locally stored data and / or the data obtained from the storage unit according to the operation instruction, so as to obtain the data and / or processing results that need to be shared across clusters for the data processing task, and store them in the storage unit and / or return them to the client through the task processing unit;

[0010] The storage unit is used to store the data that needs to be shared across clusters and / or the processing results of the data processing task.

[0011] In a second aspect, an embodiment of the present invention provides a model training method, which is applied to a task processing unit corresponding to each cluster in a multi-cluster based model training system, and the method includes:

[0012] Receiving a data processing task related to model training; wherein the data processing task includes a target data acquisition task or a model training task;

[0013] According to the data processing task, an operation instruction of the data processing task is sent to the corresponding cluster, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instruction, obtains the data and / or processing results of the data processing task that need to be shared across clusters, and stores them in the storage unit;

[0014] The storage unit is used to store the data and / or processing results of the data processing task that need to be shared across clusters.

[0015] In a third aspect, an embodiment of the present invention provides a task processing unit, which is applied to a task processing unit corresponding to each cluster in a multi-cluster based model training system, and the task processing unit includes:

[0016] An agent module, used to receive data processing tasks related to model training; wherein the data processing tasks include target data acquisition tasks or model training tasks;

[0017] The engine module is used to send an operation instruction of the data processing task to the corresponding cluster according to the data processing task, so that the cluster performs data processing on the locally stored data and / or the data obtained from the storage unit according to the operation instruction, obtains the data and / or processing results of the data processing task that need to be shared across clusters, and stores them in the storage unit;

[0018] The storage unit is used to store the data and / or processing results of the data processing task that need to be shared across clusters.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer device, including:

[0020] Memory for storing computer programs;

[0021] A processor is used to run the computer program stored in the memory to implement the method as described in the second aspect.

[0022] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the second aspect.

[0023] The multi-cluster-based model training system, method, device and storage medium provided by the embodiment of the present invention receive data processing tasks related to model training sent by the client through the task processing unit, and send the operation instructions of the data processing tasks to the corresponding cluster according to the data processing tasks, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instructions, and stores the processing results in the storage unit and / or returns them to the client through the task processing unit. By applying a storage unit for storing data and / or processing results that need to be shared across clusters in the multi-cluster-based model training system, the problem of data sharing and synchronization of multiple clusters during the model training process is realized through the storage unit, avoiding resource waste caused by data migration and data storage backup, avoiding source data leakage, solving the problem of slow synchronization when the data volume is large, and improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] Figure 1An architecture diagram of a multi-cluster based model training system provided by one embodiment of the present invention;

[0026] Figure 2 A flowchart of a model training method provided by an embodiment of the present invention;

[0027] Figure 3 A flowchart of a model training method provided by another embodiment of the present invention;

[0028] Figure 4 A flowchart of a model training method provided by another embodiment of the present invention;

[0029] Figure 5 A signaling diagram of a model training method provided by another embodiment of the present invention;

[0030] Figure 6 A structural diagram of a task processing unit provided by an embodiment of the present invention;

[0031] Figure 7 A structural diagram of a computer device provided in accordance with an embodiment of the present invention.

[0032] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] In the prior art, as the requirements for service diversification and training efficiency continue to increase, single cluster services have shown their own drawbacks. For example, commonly used machine learning frameworks such as Spark generally run in Yarn clusters, while deep learning frameworks such as TensorFlow, Caffe, PyTorch, etc. generally run in K8S clusters. Therefore, single cluster services can no longer meet users' more personalized needs. For example, when users want to use the comment data in the mall to train the BERT (Bidirectional Encoder Representations from Transformers) model, the BERT model runs most efficiently in the K8S cluster, while the mall data is stored in the data mart and needs to be extracted using Spark tasks. Although Spark tasks can also be run in the K8S cluster, the efficiency is not as good as in the Yarn cluster, mainly because the Yarn cluster focuses more on offline jobs and resource scheduling. Therefore, it is necessary to extract data in the Yarn cluster and train the BERT model in the K8S cluster, that is, it is necessary to implement the model training process based on multiple clusters.

[0035] In the prior art, users usually store data in different clusters according to different business lines, such as intermediate data or result data. Each cluster data has its own dedicated business and characteristics, and has a specific user group. When a user needs to switch clusters, it is usually necessary to download and upload data from different clusters to complete data migration, especially for some user data or original data used by user tasks, which has a large amount of data and places great demands on the storage space of the user's device and is inefficient. Alternatively, the platform administrator can copy data between clusters, but the copied data is stored in multiple copies, resulting in a waste of storage resources and the risk of privacy data leakage.

[0036] Taking the above example, the comment data in the mall extracted by the Yarn cluster is stored in the storage resources of the Yarn cluster, and the K8S cluster needs to use the comment data in the mall to perform the BERT model. The user can first download the comment data in the mall from the Yarn cluster and then upload it to the K8S cluster, which requires a large amount of storage space on the user's device, is time-consuming, labor-intensive, and inefficient. If the platform administrator copies the comment data in the mall in the Yarn cluster to the K8S cluster, the comment data in the mall is stored in the Yarn cluster and the K8S cluster respectively, resulting in a waste of storage resources.

[0037] In order to solve the above problems, a storage unit for storing data and / or processing results that need to be shared across clusters is introduced in an embodiment of the present invention, that is, when data needs to be used across clusters, the data is synchronized in the storage unit for use by other clusters. For example, in the above example, data extraction tasks can be performed in the Yarn cluster, and the extraction results can be synchronized to the storage unit. The K8S cluster can perform model training based on the data in the storage unit. The storage unit can realize the problem of data sharing and synchronization of multiple clusters during the model training process, avoid resource waste caused by data migration and data storage backup, avoid source data leakage, solve the problem of slow synchronization when the data volume is large, and improve efficiency.

[0038] More specifically, an embodiment of the present invention provides a model training system based on multiple clusters, such as Figure 1 As shown, the multi-cluster based model training system includes: a client 110, a storage unit 120, at least two clusters (taking a first cluster 130 and a second cluster 140 as an example), and a task processing unit corresponding to each cluster (a first task unit 150, a second task unit 160); each task processing unit is communicatively connected with its corresponding cluster, the client 110, and the storage unit 120; each cluster is communicatively connected with the storage unit 120; the cluster includes resources required for data processing tasks.

[0039] Among them, the client 110 is used to send the data processing task to the task processing unit; the task processing unit is used to send the operation instruction of the data processing task to its corresponding cluster according to the received data processing task; the cluster is used to process the locally stored data or the data obtained from the storage unit 120 according to the operation instruction, and store the processing result in the storage unit 120 and / or return it to the client 110 through the task processing unit; the storage unit 120 is used to store the data and / or processing results of the data processing task that need to be shared across clusters.

[0040] The data processing tasks in this embodiment are not limited to data acquisition tasks, model training tasks, etc. The client 110 can send the corresponding data processing tasks to the task processing unit of the cluster to be used according to the user's instructions. For example, in the above example, if the user needs the Yarn cluster to extract data, the data acquisition task is sent to the task processing unit of the Yarn cluster through the client, and if the user needs the K8S cluster to train the model, the model training task is sent to the task processing unit of the K8S cluster through the client. In this way, the respective advantages of different clusters can be brought into play.

[0041] In this embodiment, each cluster deploys a task processing unit separately, which can facilitate the expansion of the system. When a new cluster is added to the system, the new cluster can be connected to the system by simply deploying and configuring the task processing unit, which has better scalability.

[0042] In order to clearly introduce the specific process of applying the multi-cluster based model training system to perform model training, taking the at least two clusters including the first cluster 130 (which can correspond to the Yarn cluster in the above example) and the second cluster 140 (which can correspond to the K8S cluster in the above example) as an example, the first cluster 130 corresponds to the first task processing unit 150, and the second cluster 140 corresponds to the second task processing unit 160, then the specific process of applying the multi-cluster based model training system to perform model training can be: the client 110 sends the target data acquisition task to the first task processing unit 150; the first task processing unit 150 sends the target data acquisition task to the first cluster according to the acquisition task. Group 130 sends a first operation instruction for the acquisition task; the first cluster 130 acquires the target data as data that needs to be shared across clusters according to the first operation instruction, and stores it in the storage unit 120; the client 110 sends the model training task to the second task processing unit 160; the second task processing unit 160 sends a second operation instruction for the model training task to the second cluster 140 according to the model training task; the second cluster 140 acquires the target data from the storage unit 120 for model training according to the second operation instruction, and stores the training result in the storage unit 120 and / or returns it to the client 110 through the second task processing unit.

[0043] It should be noted that there can be multiple clients, facing different users, and each client is connected to each task processing unit. Figure 1 Only one client is taken as an example.

[0044] In addition, the above-mentioned first cluster and second cluster can be converted to each other without conflict. For example, for cluster A and cluster B, if the user wants to obtain data through cluster A and cluster B performs model training, cluster A is the first cluster and cluster B is the second cluster; if the user wants to obtain data through cluster B and cluster A performs model training, cluster B is the first cluster and cluster A is the second cluster.

[0045] In addition, the first cluster is not limited to one, that is, the data required for the model training process can come from different clusters. For example, data is obtained from cluster A1 and cluster A2 and stored in a storage unit. Cluster B performs model training based on the data of cluster A1 and cluster A2 in the storage unit. In addition, the required data in the model training process may also be stored locally in the second cluster, and this part of the data does not need to be obtained by the second cluster from the storage unit.

[0046] Optionally, the storage unit in this embodiment may be an object storage service (OSS), which is an Internet-oriented distributed storage service that is massive, secure, high-performance, highly reliable, and low-cost. It can more safely and reliably store data and / or processing results that need to be shared across clusters, has fast read and write speeds, is conducive to data sharing, and can also avoid the risk of privacy data leakage, ensuring data security. Of course, the storage unit may also be a file storage device, which will not be described here.

[0047] Based on the above embodiments, the task processing unit may specifically include an agent subunit (Agent) and an engine subunit (Engine), the first task processing unit 150 includes an agent subunit 151 and an engine subunit 152, the second task processing unit 160 includes an agent subunit 161 and an engine subunit 162, the agent subunit is used to connect to the client, and the engine subunit is connected to the corresponding cluster and is responsible for operating the cluster.

[0048] Specifically, the proxy subunit is used to receive the data processing task sent by the client; the engine subunit is used to obtain the operation instruction of the data processing task according to the data processing task, and send the operation instruction to its corresponding cluster. In this embodiment, the engine subunit can encapsulate the data processing task into an operation instruction that can be executed by the cluster.

[0049] In this embodiment, the proxy subunit and the engine subunit can be pre-configured in a standardized manner. After receiving the data processing task sent by the client, the proxy subunit can obtain the client identification, data mart information, and task information according to the standardized configuration; wherein the client identification is used to identify the client (or the client's application) to prepare for the subsequent access to multiple clients (or multiple applications); the data mart information is the relevant information of the data mart required when data needs to be extracted, and supports multi-mart operations; the task information is used to describe the task details, including the task type and task operation, whether data synchronization is required and data synchronization information, etc. The engine subunit can obtain the relevant information and operation details and operation statements of the cluster based on the above information of the proxy subunit, so as to obtain the operation instructions that the cluster can execute according to the preset operation instruction acquisition rules.

[0050] In addition, the engine subunit is also used to monitor the task status of the cluster, and send the task status to the client through the proxy subunit for display on the client, so that the user can understand the task progress and then perform the next operation when the task is completed, such as continuing the model training task when the data acquisition task is completed. In addition, the proxy subunit is also used to record the log of the data processing task and send it to the client so that the client can view the log and understand the specific execution process of the data processing task.

[0051] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0052] Figure 2 Flowchart of the model training method provided by an embodiment of the present invention. This embodiment provides a model training method, which is applied to the above-mentioned multi-cluster-based model training system, the multi-cluster-based model training system includes a client, a storage unit, at least two clusters, and a task processing unit corresponding to each of the clusters, the execution subject is the first task processing unit or the second task processing unit, and the specific steps of the method are as follows:

[0053] S201. The task processing unit receives a data processing task related to model training sent by a client.

[0054] In this embodiment, the client can send data processing tasks related to model training to the task processing unit, including but not limited to data acquisition tasks, model training tasks, etc., where different tasks can be sent to task processing units corresponding to different clusters, and the task processing unit can perform subsequent processing after receiving the data processing tasks. For example, if the data acquisition task is executed by the first cluster, the data acquisition task sent by the client will be received by the first task processing unit corresponding to it; and if the model training task is executed by the second cluster, the model training task sent by the client will be received by the second task processing unit corresponding to it.

[0055] S202. The task processing unit sends an operation instruction of the data processing task to its corresponding cluster according to the data processing task, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instruction, obtains the data and / or processing results of the data processing task that need to be shared across clusters, and stores them in the storage unit and / or returns them to the client through the task processing unit.

[0056] In this embodiment, the task processing unit encapsulates the data processing task into an operation instruction that can be executed by the corresponding cluster based on the received data processing task, and sends it to the cluster. Further, after receiving the operation instruction, the cluster performs the corresponding data processing process according to the operation instruction, wherein the source of the processed data can be the local storage data of the cluster and / or the data obtained from the storage unit, and the processing result can be stored in the storage unit for use by other clusters or for use by the client, or can be directly returned to the client through the task processing unit. The specific execution process of the task can vary according to the type of data processing task. For example, for data acquisition tasks, the cluster can extract target data, and for model training tasks, the cluster can perform model training based on local storage data and / or data obtained from the storage unit.

[0057] The model training method provided in this embodiment receives data processing tasks related to model training sent by the client through a task processing unit, and sends operation instructions of the data processing tasks to the corresponding cluster according to the data processing tasks, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instructions, and stores the processing results in the storage unit and / or returns them to the client through the task processing unit. By applying a storage unit for storing data and / or processing results that need to be shared across clusters in a multi-cluster based model training system, the problem of data sharing and synchronization of multiple clusters during the model training process is realized through the storage unit, avoiding resource waste caused by data migration and data storage backup, avoiding source data leakage, solving the problem of slow synchronization when the amount of data is large, and improving the efficiency of model training.

[0058] The following takes the first cluster performing data acquisition and the second cluster performing model training as an example to introduce the data acquisition process and model training process of the multi-cluster based model training system respectively.

[0059] Figure 3 Flowchart of the model training method provided by an embodiment of the present invention. This embodiment provides a model training method, which is applied to the above-mentioned multi-cluster-based model training system, and the execution subject is the first task processing unit corresponding to the first cluster. The specific steps of the method are as follows:

[0060] S301. The first task processing unit receives a target data acquisition task sent by a client.

[0061] In this embodiment, the user needs to know in advance that the target data is stored in the first cluster or that the first cluster can acquire the target data, and then send a target data acquisition task to the first task processing unit corresponding to the first cluster through the client.

[0062] S302. The first task processing unit sends the first operation instruction of the acquisition task to the first cluster according to the acquisition task, so that the first cluster acquires the target data according to the first operation instruction, and stores the target data in the storage unit as the data that needs to be shared across clusters, so that the second cluster can acquire the target data from the storage unit for model training.

[0063] In this embodiment, the first task processing unit encapsulates the acquisition task of the received target data into a first operation instruction that can be executed by the first cluster, and sends it to the first cluster. Furthermore, after receiving the first operation instruction, the first cluster acquires the target data and stores it in the storage unit for use by the second cluster when performing model training.

[0064] Further, the first task processing unit in S302 sends a first operation instruction for acquiring the task to the first cluster according to the acquired task, which may specifically include:

[0065] The first task processing unit acquires a first operation instruction of the acquisition task according to the acquisition task and a preset operation instruction acquisition rule of the first cluster, and sends the first operation instruction to the first cluster.

[0066] In this embodiment, the operation instruction acquisition rules can be pre-configured in the first task processing unit, that is, the rules on how to convert the received tasks into operation statements that can be executed by the first cluster. The rules can be specifically configured according to the first cluster and will not be repeated here. In this embodiment, the agent subunit of the first task processing unit can receive the target data acquisition task sent by the client, and the engine subunit can obtain the operation statement that can be executed by the first cluster according to the acquisition task and the preset operation instruction acquisition rules of the first cluster, and encapsulate it into the first operation instruction and send it to the first cluster.

[0067] On the basis of the above embodiment, after the first task processing unit sends the first operation instruction of acquiring the task to the first cluster, the method may further include:

[0068] The first task processing unit obtains the task status of the acquisition task; and sends the task status to the client, so that the client sends the model training task to the second task processing unit when determining that the acquisition task is completed.

[0069] In this embodiment, the first task processing unit can monitor the task status of the first cluster executing the acquisition task, including whether it is executed normally, the progress of the task, etc., and then the first task processing unit can send the task status to the client. Optionally, the task status can be sent to the client when the task status is abnormal or the task is completed, or when the client sends a task status query request to the first task processing unit. When the acquisition task is completed, the client can further send a model training task to the second task processing unit, thereby starting the training process of the second cluster, wherein the sending of the model training task by the client to the second task processing unit can be triggered by the user, or can be automatically triggered by the client when it is determined that the acquisition task is completed. In this embodiment, the engine subunit of the first task processing unit can obtain the task status of the acquisition task from the first cluster, and the agent subunit sends the task status to the client.

[0070] Based on any of the above embodiments, the method may further include:

[0071] The first task processing unit obtains storage location information of the target data in the storage unit, and sends the storage location information to the client, so that the client carries the storage location information when sending the model training task to the second task processing unit.

[0072] In this embodiment, the first task processing unit may specify the storage location of the target data in the storage unit in advance, so that the first cluster stores the target data in the storage location; or the first task processing unit may obtain the storage location of the target data in the storage unit after the first cluster stores the target data in the storage unit. After obtaining the storage location information of the target data in the storage unit, the storage location information is sent to the client, and then when the client sends the model training task to the second task processing unit, the storage location information can be carried in the model training task, so that the second cluster can obtain the target data from the storage unit according to the storage location information and perform model training.

[0073] The model training method provided in the above embodiment sends the target data acquisition task to the first task processing unit through the client; the first task processing unit sends the first operation instruction of the acquisition task to the first cluster according to the acquisition task; the first cluster acquires the target data according to the first operation instruction and stores it in the storage unit; the client sends the model training task to the second task processing unit; the second task processing unit sends the second operation instruction of the model training task to the second cluster according to the model training task; the second cluster acquires the target data from the storage unit according to the second operation instruction for model training, and stores the training result in the storage unit and / or returns it to the client through the second task processing unit. By applying a storage unit for storing data and / or processing results that need to be shared across clusters in a multi-cluster based model training system, the storage unit is used to realize the problem of data sharing and synchronization of multiple clusters during the model training process, thereby avoiding resource waste caused by data migration and data storage backup, avoiding source data leakage, solving the problem of slow synchronization when the data volume is large, and improving the efficiency of model training.

[0074] Figure 4 Flowchart of the model training method provided by an embodiment of the present invention. This embodiment provides a model training method, which is applied to the above-mentioned multi-cluster-based model training system, and the execution subject is the second task processing unit corresponding to the second cluster. The specific steps of the method are as follows:

[0075] S401. The second task processing unit receives a model training task sent by a client.

[0076] In this embodiment, when the first cluster completes the task of acquiring the target data and stores the target data in the storage unit, the model training process of the second cluster can be started at this time, and the client can send the model training task to the second task processing unit to start the training process of the second cluster, wherein the sending of the model training task by the client to the second task processing unit can be triggered by the user or automatically triggered by the client when it is determined that the acquisition task is completed. It should be noted that the model training task can carry the storage location information of the target data in the storage unit, so that the second cluster can obtain the target data from the storage unit according to the storage location information.

[0077] S402. The second task processing unit sends a second operation instruction of the model training task to the second cluster according to the model training task, so that the second cluster obtains the target data from the storage unit for model training according to the second operation instruction, and stores the training result in the storage unit and / or returns it to the client through the second task processing unit.

[0078] In this embodiment, the second task processing unit encapsulates the model training task into a second operation instruction executable by the second cluster based on the received model training task, and sends it to the second cluster. Furthermore, after receiving the second operation instruction, the second cluster obtains the target data from the storage unit for model training, and stores the training results in the storage unit after completing the model training. The user can view the training results from the storage unit. Of course, the second cluster can also send the training results directly to the client through the second task processing unit.

[0079] In this embodiment, the client triggers the model training task when determining that the acquisition task is completed, which can avoid interactions between clusters and interactions between task processing units, and can facilitate the expansion of the multi-cluster based model training system. When adding new clusters, it is not necessary to build communication connections between clusters and communication connections between task processing units.

[0080] Further, the second task processing unit in S402 sends a second operation instruction of the model training task to the second cluster according to the model training task, which may specifically include:

[0081] The second task processing unit obtains the second operation instruction of the model training task according to the model training task and the preset operation instruction acquisition rule of the second cluster, and sends the second operation instruction to the second cluster.

[0082] In this embodiment, the operation instruction acquisition rules can be pre-configured in the second task processing unit, that is, the rules on how to convert the received tasks into operation statements that can be executed by the second cluster. The rules can be specifically configured according to the second cluster and will not be repeated here. In this embodiment, the proxy subunit of the second task processing unit can receive the model training task sent by the client, and the engine subunit can obtain the operation statement that can be executed by the second cluster according to the model training task and the preset operation instruction acquisition rules of the second cluster, and encapsulate it into a second operation instruction and send it to the second cluster.

[0083] On the basis of the above embodiment, after the second task processing unit sends the second operation instruction of the model training task to the second cluster, the method may further include:

[0084] The second task processing unit obtains the task status of the model training task; and sends the task status to the client, so that when the training result is stored in the storage unit, the client obtains the training result from the storage unit when it determines that the model training task is completed.

[0085] In this embodiment, the second task processing unit can monitor the task status of the second cluster executing the model training task, including whether it is executed normally, the task progress, etc., and then the second task processing unit can send the task status to the client. Optionally, the task status can be sent to the client when the task status is abnormal or the task is completed, or when the client sends a task status query request to the second task processing unit. In this embodiment, the engine subunit of the second task processing unit can obtain the task status of the acquisition task from the second cluster, and the agent subunit can send the task status to the client.

[0086] Based on any of the above embodiments, the method may further include:

[0087] The second task processing unit obtains storage location information of the training result in the storage unit, and sends the storage location information to the client, so that the client obtains the training result from the storage unit according to the storage location information.

[0088] In this embodiment, when the second cluster stores the training results in the storage unit, the second task processing unit can obtain the storage location information of the training results in the storage unit and send it to the client, so that the client can obtain or view the training results from the storage unit according to the storage location information; in addition, the second task processing unit can specify the storage location of the training results in the storage unit in advance, so that the second cluster stores the training results in this location.

[0089] The model training method provided in the above embodiment sends the target data acquisition task to the first task processing unit through the client; the first task processing unit sends the first operation instruction of the acquisition task to the first cluster according to the acquisition task; the first cluster acquires the target data according to the first operation instruction and stores it in the storage unit; the client sends the model training task to the second task processing unit; the second task processing unit sends the second operation instruction of the model training task to the second cluster according to the model training task; the second cluster acquires the target data from the storage unit according to the second operation instruction for model training, and stores the training result in the storage unit and / or returns it to the client through the second task processing unit. By applying a storage unit for storing data and / or processing results that need to be shared across clusters in a multi-cluster based model training system, the storage unit is used to realize the problem of data sharing and synchronization of multiple clusters during the model training process, thereby avoiding resource waste caused by data migration and data storage backup, avoiding source data leakage, solving the problem of slow synchronization when the data volume is large, and improving the efficiency of model training.

[0090] Figure 5A signaling diagram of a model training method provided by an embodiment of the present invention. This embodiment provides a model training method, which is applied to the above-mentioned multi-cluster-based model training system, wherein at least two clusters include a first cluster and a second cluster, the first cluster corresponds to a first task processing unit, and the second cluster corresponds to a second task processing unit, and the specific steps of the method are as follows:

[0091] S501, the client sends a target data acquisition task to a first task processing unit;

[0092] S502: The first task processing unit sends a first operation instruction of the acquisition task to the first cluster according to the acquisition task;

[0093] S503: The first cluster obtains the target data according to the first operation instruction and stores it in a storage unit;

[0094] S504: The first task processing unit obtains the task status of the acquisition task and sends it to the client;

[0095] S505: When determining that the acquisition task is completed, the client sends the model training task to the second task processing unit;

[0096] S506. The second task processing unit sends a second operation instruction of the model training task to the second cluster according to the model training task;

[0097] S507, the second cluster obtains the target data from the storage unit;

[0098] S508. The second cluster performs model training according to the second operation instruction;

[0099] S509, the second cluster stores the training result in a storage unit;

[0100] Of course, S509 can also be returned to the client through the second task processing unit, which will not be repeated here.

[0101] S510, the second task processing unit obtains the task status of the model training task and sends it to the client;

[0102] S511. When determining that the model training task is completed, the client obtains the training result from the storage unit.

[0103] The principles and effects of the model training method provided in the embodiments of the present invention can be found in the above embodiments and will not be described in detail here.

[0104] Figure 6The task processing unit provided in this embodiment is the first task processing unit in the above embodiment, and can execute the processing flow provided in the model training method embodiment on the first task processing unit side, such as Figure 6 As shown, the task processing unit 600 includes: an agent module 601 and an engine module 602 .

[0105] The proxy module 601 is used to receive data processing tasks related to model training sent by the client; wherein the data processing tasks include target data acquisition tasks or model training tasks;

[0106] The engine module 602 is used to send an operation instruction of the data processing task to the corresponding cluster according to the data processing task, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instruction, obtains the data and / or processing results of the data processing task that need to be shared across clusters, and stores them in the storage unit;

[0107] The storage unit is used to store the data and / or processing results of the data processing task that need to be shared across clusters.

[0108] In a possible design, the data processing tasks related to the model training include a target data acquisition task;

[0109] The engine module 602 is used for:

[0110] According to the acquisition task, a first operation instruction of the acquisition task is sent to the corresponding cluster, so that the cluster acquires the target data according to the first operation instruction, and stores the target data in a storage unit as the data that needs to be shared across clusters, so that other clusters can acquire the target data from the storage unit for model training.

[0111] In a possible design, the data processing tasks related to model training include model training tasks;

[0112] The engine module 602 is used for:

[0113] According to the model training task, a second operation instruction of the model training task is sent to the corresponding cluster, so that the cluster obtains the target data from the storage unit for model training according to the second operation instruction, and stores the training result in the storage unit and / or returns it to the client through the task processing unit.

[0114] In a possible design, when the engine module 602 sends the first operation instruction of the acquisition task to the corresponding cluster according to the acquisition task, it is configured to:

[0115] According to the acquisition task and the preset operation instruction acquisition rule of the corresponding cluster, a first operation instruction of the acquisition task is acquired, and the first operation instruction is sent to the corresponding cluster.

[0116] In a possible design, the engine module 602 is further used to obtain the task status of the acquisition task;

[0117] The proxy module 601 is also used to send the task status to the client, so that the client sends the model training task to the task processing unit corresponding to the cluster used to train the model when determining that the acquisition task is completed.

[0118] In a possible design, the engine module 602 is further used to obtain storage location information of the target data in the storage unit;

[0119] The proxy module 601 is also used to send the storage location information to the client, so that the client carries the storage location information when sending the model training task.

[0120] In a possible design, when the engine module 602 sends the second operation instruction of the model training task to its corresponding cluster according to the model training task, it is used to:

[0121] According to the preset operation instruction acquisition rules of the model training task and its corresponding cluster, the second operation instruction of the model training task is obtained, and the second operation instruction is sent to its corresponding cluster.

[0122] In a possible design, the engine module 602 is further used to obtain the task status of the model training task;

[0123] The proxy module 601 is also used to send the task status to the client, so that when the training result is stored in the storage unit, the client obtains the training result from the storage unit when it determines that the model training task is completed.

[0124] In a possible design, the engine module 602 is further used to obtain storage location information of the training result in the storage unit;

[0125] The proxy module 601 is further used to send the storage location information to the client, so that the client obtains the training result from the storage unit according to the storage location information.

[0126] The task processing unit provided in the embodiment of the present invention can be specifically used to execute the above Figure 2-4The specific functions of the provided method embodiment will not be repeated here.

[0127] The embodiment of the present invention receives data processing tasks related to model training sent by the client through a task processing unit, and sends an operation instruction of the data processing task to the corresponding cluster according to the data processing task, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instruction, and stores the processing results in the storage unit and / or returns them to the client through the task processing unit. By applying a storage unit for storing data and / or processing results that need to be shared across clusters in a multi-cluster based model training system, the problem of data sharing and synchronization of multiple clusters during the model training process is realized through the storage unit, avoiding resource waste caused by data migration and data storage backup, avoiding source data leakage, solving the problem of slow synchronization when the data volume is large, and improving the efficiency of model training.

[0128] Figure 7 The computer device provided in the embodiment of the present invention is a structural diagram of the computer device provided in the embodiment of the present invention. The computer device provided in the embodiment of the present invention is a task processing unit in the above embodiment, and can execute the processing flow provided in the model training method embodiment on the first task processing unit side, such as Figure 7 As shown, the computer device 70 includes a memory 71, a processor 72, a computer program and a communication interface 73; wherein the computer program is stored in the memory 71 and is configured so that the processor 72 executes the model training method on the first task processing unit side described in the above embodiment.

[0129] Figure 7 The computer device of the illustrated embodiment can be used to execute the technical solutions of the above-mentioned model training method embodiments, including the model training method on the first task processing unit side or the technical solutions of the model training method embodiments on the first task processing unit side. The implementation principles and technical effects are similar and will not be repeated here.

[0130] In addition, this embodiment also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the model training method on the first task processing unit side described in the above embodiment.

[0131] In addition, this embodiment also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the model training method on the second task processing unit side described in the above embodiment.

[0132] The present application also provides a computer program, including a program code. When a computer runs the computer program, the program code executes the model training method on the first task processing unit side as described in the above embodiment.

[0133] The present application also provides a computer program, including a program code. When a computer runs the computer program, the program code executes the model training method on the second task processing unit side as described in the above embodiment.

[0134] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0138] It should be noted that the terms "first", "second", etc. in various embodiments of the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0139] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-cluster based model training system, characterized in that: include: A client, a storage unit, at least two clusters, and a task processing unit corresponding to each of the clusters; Each of the task processing units is communicatively connected with the corresponding cluster, the client, and the storage unit; Each of the clusters is in communication with the storage unit; the clusters include resources required for data processing tasks; The client is used to send a data processing task to the task processing unit; wherein the data processing task includes a data acquisition task or a model training task; The task processing unit is used for sending an operation instruction of the data processing task to the corresponding cluster according to the received data processing task; The cluster is used to perform data processing on the locally stored data and / or the data obtained from the storage unit according to the operation instruction, so as to obtain the data and / or processing results that need to be shared across clusters for the data processing task, and store them in the storage unit and / or return them to the client through the task processing unit; The storage unit is used to store the data that needs to be shared across clusters and / or the processing results of the data processing task.

2. The multi-cluster based model training system according to claim 1, characterized in that: The task processing unit includes an agent subunit and an engine subunit; The proxy subunit is used to receive the data processing task sent by the client; The engine subunit is used to obtain an operation instruction of the data processing task according to the data processing task, and send the operation instruction to the corresponding cluster.

3. The multi-cluster based model training system according to claim 2, characterized in that: The engine subunit is further used to monitor the task status of the cluster and send the task status to the client through the proxy subunit; The client is also used to display the task status.

4. The multi-cluster based model training system according to claim 2 or 3, characterized in that: The proxy subunit is also used to record the log of the data processing task running and send it to the client; The client is also used to display the log.

5. A model training method, characterized in that: Applied to a task processing unit corresponding to each cluster in a multi-cluster based model training system, the method comprises: Receiving a data processing task related to model training; wherein the data processing task includes a target data acquisition task or a model training task; According to the data processing task, an operation instruction of the data processing task is sent to the corresponding cluster, so that the cluster processes the locally stored data and / or the data obtained from the storage unit according to the operation instruction, obtains the data and / or processing results of the data processing task that need to be shared across clusters, and stores them in the storage unit; The storage unit is used to store the data and / or the processing result of the data processing task that need to be shared across clusters.

6. The method according to claim 5, characterized in that The data processing tasks related to the model training include the task of acquiring target data; The step of sending an operation instruction of the data processing task to a corresponding cluster according to the data processing task includes: According to the acquisition task, a first operation instruction of the acquisition task is sent to the corresponding cluster, so that the cluster acquires the target data according to the first operation instruction, and stores the target data in the storage unit as the data that needs to be shared across clusters, so that other clusters can acquire the target data from the storage unit for model training.

7. The method according to claim 5 or 6, characterized in that: The data processing tasks related to the model training include model training tasks; The step of sending an operation instruction of the data processing task to a corresponding cluster according to the data processing task includes: According to the model training task, a second operation instruction of the model training task is sent to the corresponding cluster, so that the cluster obtains the target data from the storage unit for model training according to the second operation instruction, and stores the training result in the storage unit and / or returns it to the client through the task processing unit.

8. The method according to claim 6, characterized in that The step of sending a first operation instruction for acquiring the task to a corresponding cluster according to the acquiring task includes: According to the acquisition task and the preset operation instruction acquisition rule of the corresponding cluster, a first operation instruction of the acquisition task is acquired, and the first operation instruction is sent to the corresponding cluster.

9. The method according to claim 6 or 8, characterized in that: After sending the first operation instruction of acquiring the task to the corresponding cluster, the method further includes: Obtaining the task status of the acquisition task; The task status is sent to the client, so that the client sends the model training task to the task processing unit corresponding to the cluster used to train the model when determining that the acquisition task is completed.

10. The method according to claim 9, characterized in that Also includes: The storage location information of the target data in the storage unit is obtained, and the storage location information is sent to the client, so that the client carries the storage location information when sending the model training task.

11. The method according to claim 7, characterized in that The sending, according to the model training task, a second operation instruction of the model training task to a corresponding cluster thereof, comprises: According to the preset operation instruction acquisition rules of the model training task and its corresponding cluster, the second operation instruction of the model training task is obtained, and the second operation instruction is sent to its corresponding cluster.

12. The method according to claim 7, characterized in that After sending the second operation instruction of the model training task to the corresponding cluster, the method further includes: Obtaining the task status of the model training task; The task status is sent to the client, so that when the training result is stored in the storage unit, the client obtains the training result from the storage unit when it is determined that the model training task is completed.

13. The method according to claim 12, characterized in that Also includes: The storage location information of the training result in the storage unit is obtained, and the storage location information is sent to the client, so that the client obtains the training result from the storage unit according to the storage location information.

14. A task processing unit, characterized in that: A task processing unit corresponding to each cluster in a multi-cluster based model training system, wherein the task processing unit comprises: An agent module, used to receive data processing tasks related to model training; wherein the data processing tasks include target data acquisition tasks or model training tasks; The engine module is used to send an operation instruction of the data processing task to the corresponding cluster according to the data processing task, so that the cluster performs data processing on the locally stored data and / or the data obtained from the storage unit according to the operation instruction, obtains the data and / or processing results of the data processing task that need to be shared across clusters, and stores them in the storage unit; The storage unit is used to store the data and / or processing results of the data processing task that need to be shared across clusters.

15. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to run the computer program stored in the memory to implement the method according to any one of claims 5 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 5 to 13 when executed by a processor.

Images