Model training method and device

Abstract

The present invention discloses a model training method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining training configuration information of a model training task, the training configuration information includes: role information, environment information and algorithm configuration information; creating a corresponding number of containers on a cluster of a container platform according to the role information; configuring the operating environment of the node corresponding to the container according to the environment information; using a model training framework, starting a training model corresponding to the algorithm configuration information on the node; using the container platform, scheduling the container so that the node performs the model training task. This implementation can improve the efficiency of model training and reduce the workload of staff.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method and device. Background Art

[0002] In the prior art, workers usually perform model training tasks on local physical machines or training platforms. The time cost of performing training tasks on local physical machines is high. When performing model training tasks on training platforms, since workers are not aware of the distributed training operating environment, they are usually required to manually deploy the training model to each working node, which is very complicated and time-consuming. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a model training method and device, which can automatically deploy the training model to each working node, thereby improving the deployment efficiency of the model training system and reducing the workload of the staff.

[0004] In a first aspect, an embodiment of the present invention provides a model training method, comprising:

[0005] Obtaining training configuration information for a model training task, wherein the training configuration information includes: role information, environment information, and algorithm configuration information;

[0006] According to the role information, a corresponding number of containers are created on the cluster of the container platform;

[0007] According to the environment information, configure the operating environment of the node corresponding to the container;

[0008] Using a model training framework, starting a training model corresponding to the algorithm configuration information on the node;

[0009] The container platform is used to schedule the container so that the node executes the model training task.

[0010] Optionally, the node executes the model training task, including:

[0011] The node sends a file processing request to the data service center through the container corresponding to the node;

[0012] In response to the file processing request, the data service center determines the next batch of file information of the node, and sends the next batch of file information to the node;

[0013] The node obtains the next batch of training files corresponding to the next batch of file information, and uses the next batch of training files to perform the model training task.

[0014] Optionally, the data service center is provided with a queue for each of the nodes, and the elements in the queue are used to represent the training files of the nodes that have not completed training;

[0015] The data service center determines the next batch of file information of the node, including:

[0016] According to the file processing request, the data service center determines the processing status of the node for the current batch of training files;

[0017] When the processing status is successful, the data service center deletes a target element from the queue corresponding to the node, the target element corresponding to the training file of the current batch;

[0018] The data service center determines the next batch of file information of the node from the queue corresponding to the node.

[0019] Optionally, after the data service center determines the processing status of the node for the current batch of training files according to the file processing request, the method further includes:

[0020] When the processing status is processing failure, fault tolerance processing is performed on the node according to the elements in the queue corresponding to the node in the data service center.

[0021] Optionally, the data service center is provided with a queue for each of the nodes, and the elements in the queue are used to represent the training files of the nodes that have not completed training;

[0022] After scheduling the container using the container platform, the method further includes:

[0023] In the event of a system failure, obtaining elements in a queue corresponding to each of the nodes in the data service center;

[0024] Fault-tolerance processing is performed on each of the nodes according to the elements in the queue corresponding to each of the nodes.

[0025] Optionally, the role information includes: a correspondence between roles and nodes, the role including at least one of the following: a master node, a working node, a parameter server, and a data center;

[0026] The step of creating a corresponding number of containers on a cluster of a container platform according to the role information includes:

[0027] According to the correspondence between the roles and the nodes, a corresponding container is created on each node in the cluster of the container platform.

[0028] Optionally, the container platform is built using kubernetes, and the model training framework is built using tensorflow.

[0029] In a second aspect, an embodiment of the present invention provides a model training device, comprising:

[0030] An information acquisition module is used to acquire training configuration information of a model training task, wherein the training configuration information includes: role information, environment information, and algorithm configuration information;

[0031] A container creation module, used to create a corresponding number of containers on a cluster of a container platform according to the role information;

[0032] An environment configuration module, used to configure the operating environment of the node corresponding to the container according to the environment information;

[0033] A model startup module, used to start the training model corresponding to the algorithm configuration information on the node using a model training framework;

[0034] The task execution module is used to utilize the container platform to schedule the container so that the node executes the model training task.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0036] one or more processors;

[0037] a storage device for storing one or more programs,

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0039] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the above embodiments.

[0040] One embodiment of the above invention has the following advantages or beneficial effects: according to the training configuration information, a corresponding number of containers are created on the cluster of the container platform. The container can isolate resources and the operating environment, and then configure the operating environment on the node corresponding to each container and start the training model using the model training framework. Finally, by scheduling each container, the node executes the model training task. The staff can pre-configure the training configuration information of the model training task according to the needs, without manually deploying the training model to each working node. Therefore, it can reduce the workload of the staff and improve the deployment efficiency of the distributed model training system.

[0041] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0043] Figure 1 It is a schematic diagram of a process of a model training method provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a process of another model training method provided by an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of the overall architecture of a model training system provided by an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of a processing flow in a data service provided by an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of a process of another model training method provided by an embodiment of the present invention;

[0048] Figure 6 is a structural schematic diagram of a model training device provided by an embodiment of the present invention;

[0049] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0051] Model training platforms often support model training in a variety of different business scenarios, such as sorting models, precision ranking models, advertising click-through rate prediction, reinforcement learning and other models. This type of model training has the following characteristics: the scale of machines required for training is huge; the time for a single experiment is long; the machine environment required for training is different. In a production environment, it is common for the entire training task to be interrupted due to environmental anomalies of various work nodes. In addition, algorithm personnel are unaware of the distributed training operating environment. They need to manually expand the model to a multi-machine distributed mode (including splitting the model and distributing training data to different work nodes), which is also very complex and time-consuming.

[0052] Based on this, an embodiment of the present invention designs a model training method. Figure 1 FIG. 1 is a schematic diagram of a process of a model training method provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] Step 101: Obtain training configuration information for a model training task, where the training configuration information includes role information, environment information, and algorithm configuration information.

[0054] The information that users can enter in the client of the training system includes: configuration files, algorithm configuration files, etc. The configuration files may include: various role cluster resources, image information, etc. Algorithm configuration information may include: model configuration information and training methods, which are used to start the training framework in the model training framework.

[0055] Step 102: Create a corresponding number of containers on the cluster of the container platform according to the role information.

[0056] The container platform can be a platform built based on the container scheduling framework. Containers can isolate resources and operating environments. Through containers, resource scheduling for training tasks, the operating environment of different nodes, and the configuration of model algorithms can be realized. The container platform can be built using Kubernetes, centralized scheduler Hadoop YARN, and two-layer scheduler Mesos.

[0057] You can create a corresponding number of containers on the container platform cluster in the following ways:

[0058] Mode 1. The role may include at least one of the following: master node, worker node, parameter server, and data center. The role information may include: the correspondence between the role and the node. According to the correspondence between the role and the node, a corresponding container is created on each node in the cluster of the container platform. The role information in the configuration file may be in the following form: node 1 worker; node 2 worker; node 3 ps. The container platform creates corresponding containers on node 1, node 2, and node 3.

[0059] Method 2. There are multiple nodes (physical machines) in the cluster of the container platform. The platform will record the resources available for each node, including: CPU, GPU, memory, etc. After receiving the user's configuration file, the container platform will start the corresponding containers on these available nodes according to the creation strategy. The creation strategy may include: the number of nodes corresponding to different roles, the resource requirements of different roles, etc. One node can correspond to one or more containers, and one container can also correspond to one or more nodes, which is determined by the scheduling strategy and the available resources of the machine.

[0060] Step 103: According to the environment information, configure the operating environment of the node corresponding to the container.

[0061] The environment information may include image information. The container platform will start the environment corresponding to the node according to the image information. Furthermore, the container platform will start the environment corresponding to at least one working node according to the image information. For different model training tasks, such as sorting model, refined ranking model, and advertising click-through rate estimation, different environments can be started on the working nodes corresponding to different training tasks.

[0062] Step 104: Using the model training framework, start the training model corresponding to the algorithm configuration information on the node.

[0063] The model training framework is a distributed model training framework, such as tensorflow, Pytorch, MXNet, PaddlePaddle, etc.

[0064] Step 105: Use the container platform to schedule the container so that the node executes the model training task.

[0065] Use the container platform to schedule containers, and then use the preset scheduling algorithm to adopt different scheduling strategies for working nodes.

[0066] In an embodiment of the present application, the system can create a corresponding number of containers on the cluster of the container platform according to the training configuration information. The container can isolate resources and the operating environment, and then configure the operating environment on the node corresponding to each container and start the training model. Finally, by scheduling each container, the node executes the model training task. The staff can pre-configure the training configuration information of the model training task according to the needs, without manually deploying the training model to each working node. Therefore, it can reduce the workload of the staff and improve the deployment efficiency of the distributed model training system.

[0067] The framework layer of model training is responsible for sensing and running distributed computing models such as deep learning, but it cannot intervene in the automatic restart after the training task fails. Therefore, it is very helpful to improve the scheduling layer and framework layer of the machine learning training platform in an effective way to achieve automatic fault-tolerant processing of large-scale distributed training in the production environment.

[0068] There is a common phenomenon in the production environment: since the training process generally relies on many third-party libraries (HDFS, CFS shared storage, hadoop, kafka, etc.), and their stability cannot be guaranteed, many tasks that have been trained for a long time will occasionally fail due to fluctuations. Since it is impossible to control the environment of other dependent clusters, effective fault-tolerant methods can only be found from the platform side. In addition, it is very important to note that in a distributed environment, working nodes are required to load data in parallel for training. How to ensure the consistency of training tasks after failure and restart is also a very important issue. Therefore, an embodiment of the present invention ensures consistency between working nodes (workers) by creating a data center component to manage data information in training.

[0069] In the existing technology, each business party encapsulates a layer of custom training framework according to their business scenario requirements, and the distributed training operation modes are inconsistent. Based on businesses with different distributed training frameworks, algorithm engineers cannot perceive the various working nodes of each training task. Fault tolerance processing depends on the implementation of the business party framework, and since the platform scheduling layer cannot be controlled, fully automatic fault tolerance cannot be achieved.

[0070] Based on this, an embodiment of the present invention designs a model training method, which controls the distribution of training data through a data service center and records the training status of each working node to achieve system fault tolerance. Figure 2 FIG. 1 is a schematic diagram of the process of another model training method provided by an embodiment of the present invention. Figure 2 As shown, the method includes:

[0071] Step 201: Obtain training configuration information for the model training task, where the training configuration information includes: role information, environment information, and algorithm configuration information.

[0072] Step 202: Create a corresponding number of containers on the cluster of the container platform according to the role information.

[0073] Step 203: According to the environment information, configure the operating environment of the node corresponding to the container.

[0074] Step 204: Using the model training framework, start the training model corresponding to the algorithm configuration information on the node.

[0075] Step 205: Use the container platform to schedule the container so that the node executes the model training task.

[0076] Step 206: The node sends a file processing request to the data service center through its corresponding container.

[0077] The data service center may be provided with a queue for each node, and the elements in the queue are used to represent the training files of the node that have not completed training. The elements in the queue may be the file identifier, file storage path, file name, etc. of the training file.

[0078] A status flag may also be set for each node in the data center. The status flag indicates the training status of the node for the current batch of training files. The status flag may include: unprocessed, processing, successful processing, failed processing, etc.

[0079] Step 207: According to the file processing request, the data service center determines the processing status of the node for the current batch of training files.

[0080] If the processing status is successful, execute step 208. If the processing status is failed, execute step 211.

[0081] Step 208: The data service center deletes the target element from the queue corresponding to the node, where the target element corresponds to the training file of the current batch.

[0082] Step 209: The data service center determines the next batch of file information of the node from the queue corresponding to the node, and sends the next batch of file information to the node.

[0083] The next batch of file information may be a file identifier, a file storage path, a file name, etc. of the next batch of files. Through the next batch of file information, the node can obtain the next batch of files, and then use the next batch of files for machine learning training.

[0084] After the data service center sends the next batch of file information to the node, the status indicator corresponding to the node can be modified to being processed.

[0085] Step 210: The node obtains the next batch of training files corresponding to the next batch of file information, and uses the next batch of training files to perform the model training task.

[0086] Step 211: Perform fault tolerance processing on the node according to the elements in the queue corresponding to the node in the data service center.

[0087] In the event of a single worker node failure, fault tolerance is performed based on the queue of that worker node. The elements in the queue are saved as checkpoints, and each worker node will read the previously saved checkpoint file and obtain the previous state of the worker node, allowing the cluster to resume synchronization and then continue training.

[0088] In one embodiment of the present invention, after scheduling the container using the container platform, it also includes: in the event of a system failure, obtaining elements in the queue corresponding to each node in the data service center; and performing fault-tolerant processing on each node based on the elements in the queue corresponding to each node.

[0089] In the event of a system failure, such as a master node failure or a parameter server node failure, all worker nodes need to be processed. If a worker node other than the master node fails, it is not a big problem, just restart a worker node. If a parameter server node fails, the master node will suspend the jobs of all worker nodes and restore the tasks of all parameter server nodes at the last checkpoint. If the master node fails, similar to the parameter server node failure, all worker node jobs will be suspended, and after the failure is repaired, the master node tasks will be restored to the state of the last checkpoint.

[0090] In an embodiment of the present invention, the data service center is used to uniformly control the distribution of training data, so that the current training status of each working node can be determined, and the probability of model training failure due to machine environment abnormalities or network abnormalities can be reduced.

[0091] The entire process needs to be uniformly distributed through the training platform. The distributed training is deployed to the computing cluster. The use of the data center service component to uniformly distribute the training data can reduce the probability of model training failure due to abnormal machine environment or network abnormalities. This realizes the effective use of the computing cluster, improves the utilization efficiency, and can also improve the algorithm iteration efficiency of algorithm personnel, bringing about the improvement of online model effects. In addition, the data center service component can also record the data currently used for training of each working node in the training, which also ensures that the working node can load the data before the failure to continue training after the failure restart, ensuring the consistency between the working nodes.

[0092] In order to make the method of the embodiment of the present invention easier to understand, the following container platform is built using kubernetes, and the model training framework is tensorflow, which is explained as a specific embodiment. Figure 3 FIG. 1 is a schematic diagram of the overall architecture of a model training system provided by an embodiment of the present invention. Figure 3 As shown in the figure, the system construction mainly includes the following parts:

[0093] (1) Data storage: mainly consists of two parts: HDFS data storage and shared data storage.

[0094] HDFS data storage. A shared training model log information storage based on distributed file storage. HDFS data storage mainly refers to HDFS storage, which stores training data.

[0095] Shared data storage. It mainly implements persistent storage of the models and logs generated by each task training, ensuring that multiple different working nodes can read files together during distributed training; the main function of the training code storage is to use each working node in the task to have a public file access path to achieve code file sharing and persistence, ensuring that the training code file can be obtained after the training working node is abnormally restarted. Shared data storage: mainly refers to cfs (file storage, Cloud File Storage), which is a network disk mounted in the training container, which stores model code and training logs.

[0096] (2) Training process launcher: The main function of this part is to launch distributed training tasks and cluster deployment. It can be divided into three parts:

[0097] Task distribution and deployment, training task status monitoring and fault-tolerant restart process; task distribution and deployment are done through the client of the machine learning platform, and distributed tasks are created with one click according to the user's configuration information. Pods can be created on the kubernetes cluster, which simplifies the scheduling of distributed training tasks and solves the problem of unified scheduling of large-scale machine learning.

[0098] Training task status monitoring. Since the main task of algorithm engineers is to write distributed training code, when the number of training nodes is large, it is very cumbersome to monitor whether the training process is running normally. Therefore, it is very difficult to obtain the status of the training task, and it is impossible to perceive whether the training task has an error and needs to be restarted. The platform implements the real-time reporting of the status of each worker node in the pod and the monitoring and alarm of real-time tasks. After obtaining the training status of the task, it is passed to the scheduler of the corresponding working node, and then rescheduling and restarting are realized. The fault-tolerant restart process specifically uses the shared file and kubernetes mechanism to realize that after the working node is rescheduled, it can be restored to the previous software environment and re-pull the training process.

[0099] (3) Kubernetes cluster manager: The underlying hardware of this solution is mainly built on the kubernetes container cluster management system. It will configure the basic environment required for the task based on the configuration information written by the user, and distribute the task to the corresponding physical nodes of the cluster through the control center. Through docker technology, the platform realizes multi-task environment and resource isolation. kubernetes is a new leading distributed architecture solution based on container technology, and is also a complete distributed system support platform with high reliability and scalability.

[0100] (4) Data Center Server: Most distributed machine learning tasks are divided into two modes: model parallelism and data parallelism. JD.com's advertising and recommendation-related scenarios mainly use data-parallel distributed training. The main contribution of this patent is for data-parallel distributed training scenarios. The main function of the data center server is to record training files based on user configuration information and distribute training files based on the number of workers. This function can also record the status of file consumption based on the worker training the file, and record file processing information through multiple queues. In addition, the data center server can also perform error detection on the training file and perform fault tolerance for abnormal access to HDFS to keep the training task going.

[0101] (5) Distributed training framework: Most distributed machine learning training tasks are business codes written based on the deep learning training framework built by algorithm personnel themselves. Therefore, the fault tolerance of the deep learning framework itself is also an indispensable part of this solution. Based on the framework layer, since all workers need to maintain consistency during the training cycle, if a worker rejoins the cluster, other workers will also restart. Now, each worker will read the previously saved checkpoint file and obtain its previous state, so that the cluster can be restored to synchronization and then continue training. If a non-master node fails, it is not a big problem. Just restart a worker node. If a parameter server node fails, the master node will suspend the jobs of all worker nodes and restore all parameter server tasks at the last checkpoint. If the egress node fails, similar to the parameter server node, everything will be suspended. After the fault is repaired, the master node will be restored to the state of the last checkpoint.

[0102] Figure 4 FIG. 1 is a schematic diagram of a processing flow in a data service provided by an embodiment of the present invention. Figure 4As shown, the solution of the embodiment of the present invention deploys a separate data center service (data-service) in the distributed cluster. The main functions of the data-service service are: first, the corresponding file name needs to be obtained from the distributed storage of the training data, and it is instantiated as a GeneratorDataset through the Dataset module in tensorflow. Then, based on the dataset, a Dataset responsible for data distribution and data-checkpoint based on the grpc service is implemented.

[0103] The difference from tensorflow in loading data is that the worker nodes in the training process communicate with the data center service to obtain the data set file for training. Based on the customized tensorflow Dataset, by creating a grpc service and sending it to the data center server, the corresponding training data file is obtained, and the corresponding file record is left in the data center. This greatly reduces the probability of multiple workers making mistakes in obtaining training data, and realizes the perception of training files and the fault tolerance of file-level training data during training.

[0104] Figure 5 FIG. 1 is a schematic diagram of a process of another model training method provided by an embodiment of the present invention. Figure 5 As shown in the figure, the overall process of the overall platform architecture is to submit distributed training, estimation or verification tasks to the kubernetes cluster through the client. When the task starts, the data service center will obtain a list of files containing all file names and load them into the memory of the pod (not belonging to one of the workers). Subsequently, the worker node will request the data set file from the data center at runtime, report the status after processing, and then obtain the next file. The method includes the following steps:

[0105] Step S01: The user uses the client command line tool to first determine whether the user is logged in. The user registration information is created through the Django-based user registration system. After successful login, the corresponding distributed fault-tolerant permissions are obtained, so that the distributed fault-tolerant training tasks can be submitted to the cluster.

[0106] Step S02: Verify the legitimacy of the user-defined training framework code according to the algorithm file and configuration path specified by the user. And through the information in this configuration file, the various roles in the distributed training process, including the mirror environment information of the data center service, can also be obtained.

[0107] Step S03: Since the machine learning framework version developed within the department will be constantly updated and the user algorithm environment is different, this solution provides an algorithm task submission module that supports multiple versions. Specifically, through the perfect combination of docker container image and kubernetes, the user's needs for different training environments with multiple versions and multiple training nodes are realized.

[0108] Step S04: The core function of the distributed machine learning platform is to automatically allocate computing resources and create pods for multiple role machines in the tensorflow distributed cluster, including the master node (chief), worker node (worker), parameter server (ps), and data center (data-service) through cloud-native kubernetes. The fault tolerance function can avoid model training failures caused by hardware problems or communication problems. Specifically, it includes the following stages:

[0109] Phase 1. Distributed training creation phase. This phase mainly parses the configuration file information of the algorithm directory provided by the user, and creates a corresponding series of distributed training resources through the client, including CPU & GPU resources. Specifically, for each distributed training, it is tasked for easy management. By setting a separate training ID for a single task and then parsing the configuration file, the number of all machine roles during the task startup process is known, and the pod is allocated. And through the kubernetes deployment, multiple copies can be updated and managed in a timely manner, and different scheduling strategies can be adopted for working nodes through custom scheduling algorithms.

[0110] Phase 2: Task activation phase, training tasks fail due to machine or network problems. In the task activation phase, the user algorithm files are packaged and synchronized to the distributed shared storage file system, and then the algorithm framework is synchronized to the machine's disk during actual training. The training task is then started through the script file to start distributed training. Both data service nodes and distributed training nodes can go through this step to reschedule the pod activation task after a single node fails.

[0111] Phase 3. Training error restart phase. The main function of this phase is to set up a data service center in the data center service node. In distributed training, the worker node obtains a single batch of training files by requesting the data service node, and finally consumes all the files. Specifically, the worker end of the training node reads the training sample in the form of an iterator of file names through the iterator of the Dataset based on the grpc service, and then performs corresponding conversion operations through tf.data.Dataset and uses it for subsequent calculations. The worker will use checkpoint to save the above-obtained file name to implement a fault-tolerant mechanism. Specifically, each training worker node will have a queue. If the file is processed, the data service center will remove the data file; if the file is in the process of being processed, this indicates the processing status; if the file is wrong during the processing, such as abnormal access to the HDFS machine and obtaining an incorrect file name, then the corresponding processing is performed to make it fault-tolerant.

[0112] In the embodiments of the present invention, a more efficient and flexible distributed machine learning training computing resource and environment can be built; the utilization rate of the machine can be improved to provide a stable basic environment for large-scale machine learning; the training frameworks developed by different business parties based on tensorflow are fully abstracted and combined together, and a common data center module is provided for large-scale distributed training tasks, and data is distributed uniformly by the data center to reduce the probability of training task failure caused by data pull timeout due to network problems of the worker. In addition, the data center module will also record the data used by the worker in training to ensure consistency after the worker is restarted; at the same time, the solution will realize the overall process of distributed training through the combination of kubernetes distributed task scheduling and distributed machine learning framework, and through the microservice architecture, support hot restart of training tasks, and realize fault tolerance.

[0113] The key point of the solution of the embodiment of the present invention is the combination of kubernetes and machine learning distributed training. Its innovation lies in the use of kubernetes' distributed architecture solution based on container technology, which can well run the distributed machine learning framework on the kubernetes cluster management, and then obtain the status of the working nodes participating in the distributed training in kubernetes through code-side control, avoiding machine problems and network problems that cause training interruptions when training machines on a large scale, and realizing fault-tolerant processing of tasks. Similarly, since kubernetes is built on the basis of containers, the machine learning platform in the embodiment of the present invention can realize the requirements of multiple versions of development environment, and through the containerized environment solution, it realizes one-click creation of large-scale distributed tasks. At the same time, the requirements of different machine learning frameworks can also be compatible with the core content of the data center module to achieve rapid compatibility, thereby very simply realizing fault tolerance of distributed training tasks.

[0114] Figure 6 FIG. 1 is a schematic diagram of a model training device provided by an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0115] The information acquisition module 601 is used to acquire the training configuration information of the model training task, wherein the training configuration information includes: role information, environment information and algorithm configuration information;

[0116] A container creation module 602, configured to create a corresponding number of containers on a cluster of a container platform according to the role information;

[0117] An environment configuration module 603 is used to configure the operating environment of the node corresponding to the container according to the environment information;

[0118] A model startup module 604 is used to start the training model corresponding to the algorithm configuration information on the node using a model training framework;

[0119] The task execution module 605 is used to utilize the container platform to schedule the container so that the node executes the model training task.

[0120] Optionally, the task execution module 605 is further used to:

[0121] The node sends a file processing request to the data service center through the container corresponding to the node;

[0122] In response to the file processing request, the data service center determines the next batch of file information of the node, and sends the next batch of file information to the node;

[0123] The node obtains the next batch of training files corresponding to the next batch of file information, and uses the next batch of training files to perform the model training task.

[0124] Optionally, the data service center is provided with a queue for each of the nodes, and the elements in the queue are used to represent the training files of the nodes that have not completed training;

[0125] The task execution module 605 is also used to: according to the file processing request, the data service center determines the processing status of the node for the current batch of training files;

[0126] When the processing status is successful, the data service center deletes a target element from the queue corresponding to the node, the target element corresponding to the training file of the current batch;

[0127] The data service center determines the next batch of file information of the node from the queue corresponding to the node.

[0128] Optionally, the task execution module 605 is further used to: when the processing status is processing failure, perform fault-tolerant processing on the node according to the elements in the queue corresponding to the node in the data service center.

[0129] Optionally, the data service center is provided with a queue for each of the nodes, and the elements in the queue are used to represent the training files of the nodes that have not completed training;

[0130] The task execution module 605 is also used to: in the event of a system failure, obtain the elements in the queue corresponding to each of the nodes in the data service center;

[0131] Fault-tolerance processing is performed on each of the nodes according to the elements in the queue corresponding to each of the nodes.

[0132] Optionally, the role information includes: a correspondence between roles and nodes, the role including at least one of the following: a master node, a working node, a parameter server, and a data center;

[0133] The container creation module 602 is specifically used to create a corresponding container on each node in the cluster of the container platform according to the correspondence between the roles and the nodes.

[0134] Optionally, the container platform is built using kubernetes, and the model training framework is built using tensorflow.

[0135] An embodiment of the present invention provides an electronic device, including:

[0136] one or more processors;

[0137] a storage device for storing one or more programs,

[0138] When one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0139] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0141] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0142] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0143] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, they may be described as: an information acquisition module, a container creation module, an environment configuration module, a model startup module, and a task execution module. Among them, the names of these modules do not constitute a limitation on the modules themselves under certain circumstances. For example, the information acquisition module may also be described as a "module for obtaining training configuration information for model training tasks."

[0146] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0147] Obtaining training configuration information for a model training task, wherein the training configuration information includes: role information, environment information, and algorithm configuration information;

[0148] According to the role information, a corresponding number of containers are created on the cluster of the container platform;

[0149] According to the environment information, configure the operating environment of the node corresponding to the container;

[0150] Using a model training framework, starting a training model corresponding to the algorithm configuration information on the node;

[0151] The container platform is used to schedule the container so that the node executes the model training task.

[0152] According to the technical solution of the embodiment of the present invention, the system can create a corresponding number of containers on the cluster of the container platform according to the training configuration information. The container can isolate resources and the operating environment, and then configure the operating environment on the node corresponding to each container and start the training model. Finally, by scheduling each container, the node executes the model training task. The staff can pre-configure the training configuration information of the model training task according to the needs, without manually deploying the training model to each working node. Therefore, it can reduce the workload of the staff and improve the efficiency of model training.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: include: Obtaining training configuration information for a model training task, wherein the training configuration information includes: role information, environment information, and algorithm configuration information; According to the role information, a corresponding number of containers are created on the cluster of the container platform; According to the environment information, configure the operating environment of the node corresponding to the container; Using a model training framework, starting a training model corresponding to the algorithm configuration information on the node; Using the container platform, scheduling the container so that the node executes the model training task; The node performs the model training task, including: The node sends a file processing request to the data service center through the container corresponding to the node; In response to the file processing request, the data service center determines the next batch of file information of the node, and sends the next batch of file information to the node; The node obtains the next batch of training files corresponding to the next batch of file information, and uses the next batch of training files to perform the model training task.

2. The method according to claim 1, characterized in that The data service center is provided with a queue for each of the nodes, and the elements in the queue are used to represent the training files of the nodes that have not completed training; The data service center determines the next batch of file information of the node, including: According to the file processing request, the data service center determines the processing status of the node for the current batch of training files; When the processing status is successful, the data service center deletes a target element from the queue corresponding to the node, the target element corresponding to the training file of the current batch; The data service center determines the next batch of file information of the node from the queue corresponding to the node.

3. The method according to claim 2, characterized in that After the data service center determines the processing status of the node for the current batch of training files according to the file processing request, the method further includes: When the processing status is processing failure, fault tolerance processing is performed on the node according to the elements in the queue corresponding to the node in the data service center.

4. The method according to claim 1, characterized in that: The data service center is provided with a queue for each of the nodes, and the elements in the queue are used to represent the training files of the nodes that have not completed training; After scheduling the container using the container platform, the method further includes: In the event of a system failure, obtaining elements in a queue corresponding to each of the nodes in the data service center; Fault-tolerance processing is performed on each of the nodes according to the elements in the queue corresponding to each of the nodes.

5. The method according to claim 1, characterized in that The role information includes: a correspondence between roles and nodes, and the roles include at least one of the following: a master node, a working node, a parameter server, and a data center; The step of creating a corresponding number of containers on a cluster of a container platform according to the role information includes: According to the correspondence between the roles and the nodes, a corresponding container is created on each node in the cluster of the container platform.

6. The method according to claim 1, characterized in that The container platform is built using kubernetes, and the model training framework is built using tensorflow.

7. A model training device, characterized in that: include: An information acquisition module is used to acquire training configuration information of a model training task, wherein the training configuration information includes: role information, environment information, and algorithm configuration information; A container creation module, used to create a corresponding number of containers on a cluster of a container platform according to the role information; An environment configuration module, used to configure the operating environment of the node corresponding to the container according to the environment information; A model startup module, used to start the training model corresponding to the algorithm configuration information on the node using a model training framework; A task execution module, used to schedule the container using the container platform so that the node executes the model training task; The task execution module is specifically used for: the node sends a file processing request to the data service center through the container corresponding to the node; in response to the file processing request, the data service center determines the next batch of file information of the node, and sends the next batch of file information to the node; the node obtains the next batch of training files corresponding to the next batch of file information, and uses the next batch of training files to execute the model training task.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images