Training task management method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a training task management method and device, electronic equipment and storage medium. By receiving a training task submitted by a user, the training task contains target container orchestration cluster identification information and configuration information of the training task, based on the target container orchestration cluster identification information and the configuration information of the training task, the training task is submitted to the target container orchestration cluster in the plurality of container orchestration clusters, so as to run the training task through the target container orchestration cluster, and based on the state monitor corresponding to the target container orchestration cluster, the running state of the training task is listened. Compared with the prior art, the embodiment of the present disclosure can realize cross-cluster task scheduling, improve resource utilization and task execution efficiency, can realize real-time state monitoring, and facilitate timely discovery and processing of task exceptions through unified management of multiple clusters.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training task management method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, the demand for computing resources for deep learning training tasks is increasing day by day.

[0003] Traditional training task management methods have several limitations: deployment on a single physical machine is limited by node resource bottlenecks; deployment on multiple physical machines requires manual maintenance of environment consistency, resulting in high operational complexity; while solutions based on a single Kubernetes cluster achieve resource scheduling and management of multiple nodes at the underlying level, when dealing with training task management across multiple clusters, each Kubernetes cluster needs to be deployed separately, leading to training task isolation, redundant deployment, low utilization of computing resources, and low operational efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a training task management method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide a training task management method applied to a management server, the management server being used to manage multiple container orchestration clusters, the method comprising: Receive training tasks submitted by users, wherein the training tasks include target container orchestration cluster identification information and configuration information of the training tasks; Based on the target container orchestration cluster identification information and the training task configuration information, the training task is submitted to the target container orchestration cluster among the multiple container orchestration clusters so that the training task can be run through the target container orchestration cluster. Based on the status monitor corresponding to the target container orchestration cluster, the running status of the training task is monitored.

[0006] In some embodiments, prior to receiving the user-submitted training task, the method further includes: The system receives a container orchestration cluster registration request, registers multiple container orchestration clusters to the management server based on the request, and stores the metadata and node information of each container orchestration cluster in the database.

[0007] In some embodiments, the configuration information for the training task includes at least one of the following: The training task's framework type, the number of containers included in the training task, the resource requirements of each container, the startup command of the training task, the container image of the training task, the training dataset of the training task, and the storage mount data of the training task.

[0008] In some embodiments, submitting the training task to the target container orchestration cluster among the plurality of container orchestration clusters based on the target container orchestration cluster identifier information and the configuration information of the training task includes: The training task is sent to a message queue, the pending message of the training task is sent through the message queue, and the configuration information of the training task is written into the database. Based on the message to be processed, the training task is obtained from the message queue; Based on the framework type of the training task, the number of containers included in the training task, and the resource requirements of each container, a task description file corresponding to the training task is generated. Based on the target container orchestration cluster identifier information, the interface of the target container orchestration cluster is called to submit the task description file to the target container orchestration cluster.

[0009] In some embodiments, monitoring the running status of the training task based on the status monitor corresponding to the target container orchestration cluster includes: Create a corresponding status monitor for each container orchestration cluster; Based on the status monitor corresponding to the target container orchestration cluster, listen for changes in the running status of the training task and changes in the running status of each container included in the training task. The status information of the training task is stored in the database and displayed on the user interface. When a change in the running status of the training task is detected, the status information of the training task is updated; When the training task finishes running, the exit status code and exception information of each container included in the training task are obtained through the status monitor corresponding to the target container orchestration cluster.

[0010] In some embodiments, running the training task through the target container orchestration cluster includes: If the current idle resource data of the target container orchestration cluster is greater than or equal to the sum of the resource requirements of each container included in the training task, then each container included in the training task will be scheduled to run on an idle node of the target container orchestration cluster; or, If the current idle resource data is less than the sum of the resource requirements of each container, the training task is added to the waiting queue and waited until the current idle resource data is greater than or equal to the sum of the resource requirements of each container, at which point the training task is run.

[0011] In some embodiments, the method further includes: Multiple first instances are deployed using a load balancing method, and these multiple first instances receive training tasks submitted by users; or, Multiple second instances are deployed using a multi-instance approach, and the training tasks in the message queue are processed in parallel by these multiple second instances; or, The status listener is deployed in a primary / backup mode. When the primary status listener fails, the target status listener is determined from the backup status listeners, and monitoring is performed based on the target status listener.

[0012] Secondly, embodiments of this disclosure provide a training task management device applied to a management server, the management server being used to manage multiple container orchestration clusters, the device comprising: The receiving module is used to receive training tasks submitted by users, wherein the training tasks include target container orchestration cluster identification information and configuration information of the training tasks. The submission module is used to submit the training task to the target container orchestration cluster among the multiple container orchestration clusters based on the target container orchestration cluster identification information and the configuration information of the training task, so as to run the training task through the target container orchestration cluster. The monitoring module is used to monitor the running status of the training task based on the status monitor corresponding to the target container orchestration cluster.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: Memory; Processor; and Computer programs; The computer program is stored in memory and configured to be executed by a processor to implement the method as described in the first aspect.

[0014] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method as described in the first aspect.

[0015] Fifthly, embodiments of this disclosure also provide a computer program product comprising a computer program or instructions that, when executed by a processor, implement the method as described in the first aspect.

[0016] The training task management method, apparatus, electronic device, and storage medium provided in this disclosure receive training tasks submitted by users. These training tasks include target container orchestration cluster identification information and configuration information. Based on the target container orchestration cluster identification information and the configuration information, the training task is submitted to a target container orchestration cluster among multiple container orchestration clusters for execution. The execution status of the training task is monitored using a status monitor corresponding to the target container orchestration cluster. Compared to existing technologies, this disclosure, through unified management of multiple clusters, enables cross-cluster task scheduling, improves resource utilization and task execution efficiency, and allows for real-time status monitoring, facilitating timely detection and handling of task anomalies. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a training task management method provided in this embodiment of the disclosure; Figure 2 A flowchart of a training task management method provided in another embodiment of this disclosure; Figure 3 A flowchart of a training task management method provided in another embodiment of this disclosure; Figure 4 This is a schematic diagram of the training task management system architecture provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of the training task submission process provided in this embodiment of the disclosure; Figure 6 This is a schematic diagram of the structure of the training task management device provided in an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0022] With the development of artificial intelligence technology, the demand for computing resources for deep learning training tasks is increasing day by day.

[0023] Traditional training task management methods have several limitations: deployment on a single physical machine is limited by node resource bottlenecks; deployment on multiple physical machines requires manual maintenance of environment consistency, resulting in high operational complexity; while solutions based on a single Kubernetes cluster achieve resource scheduling and management of multiple nodes at the underlying level, when dealing with training task management across multiple clusters, each Kubernetes cluster needs to be deployed separately, leading to training task isolation, redundant deployment, low utilization of computing resources, and low operational efficiency.

[0024] To address this issue, this disclosure provides a training task management method, which will be described below with reference to specific embodiments.

[0025] Figure 1 This is a flowchart illustrating a training task management method provided in an embodiment of this disclosure. The method is executed by an electronic device, which can specifically be a management server. The management server can be a single server, a server cluster, a distributed cluster, or a centralized cluster. This method can be applied to scenarios involving the management of training tasks.

[0026] It is understood that the training task management method provided in this disclosure can also be applied to other scenarios.

[0027] The following is about Figure 1 The training task management method shown is introduced below. This method can be applied to a management server, which is used to manage multiple container orchestration clusters. The specific steps of this method are as follows: S101. Receive the training task submitted by the user. The training task includes the target container orchestration cluster identification information and the configuration information of the training task.

[0028] In this step, the user submits a training task, and the management server receives the submitted training task. Optionally, the training task includes the target container orchestration cluster identifier information and the training task configuration information.

[0029] like Figure 4As shown, the management server can include three units: the interface service unit ApiServer, which receives requests from users from the front-end page or OpenAPI; the task orchestration unit Launcher, which submits training tasks to the corresponding container orchestration clusters for execution according to the container orchestration clusters specified by the user; and the status monitoring unit Tracker, which monitors the status of training tasks in each managed container orchestration cluster in real time.

[0030] Specifically, this step can be executed by the "ApiServer" module of the management server. Users can initiate training tasks through the web frontend or the OpenAPI interface. This training task is a structured data packet that must explicitly specify the specific Kubernetes cluster in which the training task is expected to run, i.e., carrying the target container orchestration cluster identifier, such as cluster-id: "prod-gpu-01", and the configuration information of the training task. A unified entry point for task submission is implemented, simplifying user operations. For example, by selecting the target container orchestration cluster, such as cluster 1, through the user interface, and filling in the task image training:v1.2, the startup command python main.py, the required number of GPUs (4), and clicking submit, ApiServer will receive the training task.

[0031] In some embodiments, the configuration information of the training task includes at least one of the following: the framework type of the training task, the number of containers included in the training task, the resource requirements of each container, the start command of the training task, the container image of the training task, the training dataset of the training task, and the storage mount data of the training task.

[0032] Optionally, the configuration information for a training task may include any one of the following: the framework type of the training task, the number of containers included in the training task, the resource requirements of each container, the startup command of the training task, the container image of the training task, the training dataset of the training task, and the storage mount data of the training task. It may also include any combination of several of these, or other configuration information, without specific limitations. Flexible task configuration methods can be provided to meet the needs of different training scenarios.

[0033] Framework type: Specifies the framework used for distributed training, such as... Figure 5 As shown, it includes the PyTorch framework, the TensorFlow framework, and may also include other frameworks, which are not limited here. The system generates the corresponding startup script and inter-process communication configuration based on this.

[0034] Number of containers: Specifies the number of Pod replicas that need to be started for the training task, determining the parallel scale of the task. For example... Figure 5 As shown, for example, in the PyTorch framework, the number of containers is one Master and three Workers; in the TensorFlow framework, the number of containers is eight Workers, one parameter server (PS), and one evaluator (Evaluator).

[0035] Resource requirements for each container: Define the required computing resources for each container (e.g., CPU: "4", memory: "16Gi", GPU: "1"), which is the basis for the cluster scheduler to match and bind resources. Figure 5 As shown, for example, in the PyTorch framework, the resource requirements of each container are as follows: the Master's resource requirements are A800: 8 cards, CPU: 10 cores, mem: 500Gib, and each Worker's resource requirements are A800: 8 cards, CPU: 10 cores, mem: 500Gib.

[0036] Startup command: The entry command executed after the container starts, such as "python / app / train.py --epochs=50".

[0037] Container image: A Docker image containing training code and runtime environment, such as registry.company.com / ai / torch:1.12-cuda11.3.

[0038] Training dataset: Specifies the source path of the training data, which may be a path to network storage (NFS) or a URI of a cloud storage bucket (S3).

[0039] Storage mount data: Defines which path inside the container the external storage volume (such as a PVC) will be mounted to for persistent logs or models.

[0040] S102. Based on the target container orchestration cluster identifier information and the training task configuration information, the training task is submitted to the target container orchestration cluster among multiple container orchestration clusters so that the training task can be run through the target container orchestration cluster.

[0041] In this step, the management server locates the target container orchestration Kubernetes cluster based on the target container orchestration cluster identifier information, and submits the training task to the target container orchestration cluster according to the training task configuration information, thus realizing cross-cluster training task scheduling. The target container orchestration cluster is used to run the training task.

[0042] S103. Based on the status monitor corresponding to the target container orchestration cluster, monitor the running status of the training task.

[0043] In this step, the management server deploys an independent status monitor for each managed Kubernetes cluster. By monitoring the status monitor corresponding to the target container orchestration cluster, the running status of the training task is tracked. The running status can include committing, committed, queued, deploying, running, completed, and failed.

[0044] This embodiment of the disclosure receives training tasks submitted by users. These training tasks include target container orchestration cluster identification information and training task configuration information. Based on this information, the training task is submitted to a target container orchestration cluster among multiple container orchestration clusters for execution. The execution status of the training task is monitored using a status monitor corresponding to the target container orchestration cluster. Compared to existing technologies, this embodiment of the disclosure achieves cross-cluster task scheduling through unified management of multiple clusters, improving resource utilization and task execution efficiency. It also enables real-time status monitoring, facilitating timely detection and handling of task anomalies.

[0045] Based on the above embodiments, the method further includes steps A, B, and C: Step A: Deploy multiple first instances using a load balancing method, and receive training tasks submitted by users through multiple first instances; In this step, multiple identical initial instances, namely ApiServer instances (e.g., 3), are deployed, and a load balancer (such as Nginx or Cloud ELB) is configured in the front end. User-submitted training tasks first reach the load balancer, which then distributes them to any healthy ApiServer instance in the backend. This improves the throughput and availability of request access. The failure of a single instance does not affect the service; the load balancer will automatically switch traffic to other instances.

[0046] Step B: Deploy multiple second instances using a multi-instance approach, and process the training tasks in the message queue in parallel through multiple second instances; In this step, multiple second instances, or Launcher instances, are deployed. These instances connect to the same message queue and listen for messages within it. The message queue can employ a competing consumer pattern, ensuring that a task message is consumed and processed by only one Launcher instance. By deploying multiple instances, the potential for sudden surges in message queue traffic is distributed, preventing service unavailability due to single-instance failures and improving task processing capacity and reliability. Multiple instances process tasks in the queue in parallel, and if one instance fails, any unfinished messages will be retrieved and processed by other instances.

[0047] Step C: Deploy the status listener in a primary / backup mode. When the primary status listener fails, determine the target status listener from the backup status listeners and start listening based on the target status listener.

[0048] In this step, a master-slave mode is used to deploy the status listener. When the Tracker instance starts, it selects a master node to create a listener for the container orchestration cluster, while other nodes act as backup status listeners. If the master node releases its lock or crashes, a new master node is immediately selected from the backup nodes to listen to the training tasks of the container orchestration cluster. This ensures that the critical status monitoring link is not interrupted. Even if the physical node running the Tracker crashes, failover can be completed within seconds, achieving high availability of the listener.

[0049] It is understandable that steps A, B, and C are parallel.

[0050] This embodiment of the disclosure uses multiple first instances to receive training tasks submitted by users, thereby improving the training task processing capability and avoiding single points of failure. Multiple second instances process training tasks in the message queue in parallel, thereby improving task processing efficiency and ensuring high availability of the system. The status listener is deployed in a master-slave mode to ensure the continuity of status monitoring and improve system reliability.

[0051] Figure 2 A flowchart of a training task management method provided in another embodiment of this disclosure is shown below. Figure 2 As shown, the method includes the following steps: S201. Receive container orchestration cluster registration request, register multiple container orchestration clusters to the management server based on the container orchestration cluster registration request, and store the metadata and node information of each container orchestration cluster in the database.

[0052] In this step, a cluster registration interface is pre-set. Users or administrators will initiate a container orchestration cluster registration request, which includes the access endpoint and authentication information of the Kubernetes cluster to be managed. The management server receives the container orchestration cluster registration request and further registers multiple container orchestration clusters to the management server based on the registration request. The metadata and node information of each container orchestration cluster in the multiple container orchestration clusters are stored in the database, which can realize unified management of multiple clusters and simplify the complexity of cluster management.

[0053] Specifically, the management server verifies the validity of the information carried in the registration request (such as network connectivity and permissions). Once verification is successful, the container orchestration cluster is recorded as an available resource pool, and a control connection is established. After successful registration, the management server actively queries the metadata of the container orchestration cluster (such as Kubernetes version and total number of nodes) and the detailed information of each worker node, i.e., the server node (such as IP address, CPU / GPU model and quantity, memory, operating system, etc.), and persistently stores the metadata and node information in a relational database (such as MySQL), providing a data foundation for resource querying, scheduling decisions, and operation and maintenance management.

[0054] S202. Receive the training task submitted by the user. The training task includes the target container orchestration cluster identification information and the configuration information of the training task.

[0055] Specifically, the implementation process and principle of S202 and S101 are the same, and will not be repeated here.

[0056] S203. Send the training task to the message queue, send the pending message of the training task through the message queue, and write the configuration information of the training task into the database.

[0057] In this step, after receiving the training task, the training task is sent to a message queue. For example, the training task is encapsulated into a message to be processed (such as JSON format) and delivered to a message queue (such as Kafka / RabbitMQ). The message queue is then used to send the message to be processed of the training task to the management server. At the same time, the initial state of the training task (such as SUBMITTED) and configuration information are persisted to the database. This achieves asynchronous requests and peak shaving, improving API response speed and system concurrency resistance.

[0058] S204. Based on the messages to be processed, retrieve the training task from the message queue.

[0059] In this step, the management server receives the message to be processed and retrieves the training task from the message queue based on the message. Specifically, as follows... Figure 4 As shown, the Launcher continuously listens to the message queue. When a new message arrives, it is retrieved and deserialized to obtain the training task. For example, multiple Launcher instances can be deployed to consume the queue simultaneously, speeding up task delivery.

[0060] S205. Based on the framework type of the training task, the number of containers included in the training task, and the resource requirements of each container, generate the task description file corresponding to the training task.

[0061] In this step, a task description file can be generated based on the framework type, number of containers, and resource requirements. Specifically, the Launcher will dynamically generate a task description YAML file conforming to the Kubernetes specification based on the configuration information of the training task.

[0062] S206. Based on the target container orchestration cluster identification information, call the target container orchestration cluster's interface and submit the task description file to the target container orchestration cluster.

[0063] In this step, after obtaining the task description file, the management server will call the target container orchestration cluster's interface based on the target container orchestration cluster's identification information to submit the task description file to the target container orchestration cluster. Specifically, as follows... Figure 4 As shown, the Launcher uses the target container orchestration cluster's client library (such as client-go) to submit the generated task description file to the specified target container orchestration cluster by calling the Kubernetes API Server's Create Job interface.

[0064] Asynchronous processing is achieved through message queues, which improve system throughput and ensure reliable transmission and processing of tasks. Task description files enable automated deployment and execution of tasks.

[0065] In some embodiments, Figure 2 Steps S203, S204, S205, and S206 shown can be used as Figure 1 One specific implementation of step S102 shown.

[0066] S207. If the current idle resource data of the target container orchestration cluster is greater than or equal to the sum of the resource requirements of each container included in the training task, schedule each container included in the training task to run on an idle node of the target container orchestration cluster.

[0067] In this step, if the current idle resources in the target container orchestration cluster are greater than or equal to the total resources required by all container pods of the current training task, the task can be scheduled and run. For example, using the Volcano scheduler as the scheduler and the Gang strategy: scheduling only occurs when all pod resources are met; otherwise, pods are not scheduled. When the total resources (GPU, CPU, memory, etc.) required by each role's corresponding pod (e.g., the pod corresponding to the master, the pod corresponding to the worker) in the training task are less than or equal to the current idle resources in the cluster, then each pod of the task will be scheduled by the scheduler to run on an idle node, and the entire task will begin execution.

[0068] S208. If the current idle resource data is less than the sum of the resource requirements of each container, add the training task to the waiting queue and wait until the current idle resource data is greater than or equal to the sum of the resource requirements of each container, then run the training task.

[0069] In this step, if the current available resources in the target container orchestration cluster are less than the total resources required by all pods in the current training task, the task needs to wait in a queue until the current available resources are greater than or equal to the total resource requirements of all containers before the training task can run. If the total resources (GPU, CPU, memory, etc.) required by each role's corresponding pod in the training task exceed the current available resources in the cluster, the task cannot be executed and must wait in a queue until the allocated resources in the target container orchestration cluster are released, and the current available resources are greater than or equal to the total resource requirements of all containers before the training task can be executed.

[0070] This embodiment ensures that tasks are only scheduled when resources are sufficient, avoiding performance degradation caused by resource contention, achieving reasonable resource allocation and sequential task execution, and improving system stability.

[0071] S209. Based on the status monitor corresponding to the target container orchestration cluster, monitor the running status of the training task.

[0072] Specifically, the implementation process and principle of S209 and S103 are the same, and will not be repeated here.

[0073] This embodiment of the disclosure receives a container orchestration cluster registration request, registers multiple container orchestration clusters to a management server based on the request, and stores the metadata and node information of each container orchestration cluster in a database. It then receives training tasks submitted by users, each containing a target container orchestration cluster identifier and configuration information. The training task is then sent to a message queue, which sends a pending message for the training task. The configuration information of the training task is written to the database, and the training task is retrieved from the message queue based on the pending message. Further, based on the framework type of the training task, the number of containers included in the training task, and the resource requirements of each container, a task description file corresponding to the training task is generated. The target container orchestration cluster's interface is called based on the target container orchestration cluster identifier to submit the task description file to the target container orchestration cluster. Next, if the current idle resource data of the target container orchestration cluster is greater than or equal to the sum of the resource requirements of each container included in the training task, the containers included in the training task are scheduled to run on idle nodes of the target container orchestration cluster; or, if the current idle resource data is less than the sum of the resource requirements of each container, the training task is added to a waiting queue and waits until the current idle resource data is greater than or equal to the sum of the resource requirements of each container, at which point the training task is run. Furthermore, the running status of the training task is monitored based on the status monitor corresponding to the target container orchestration cluster. Through this method, embodiments of this disclosure can achieve unified management of multiple clusters, simplify cluster management complexity, achieve asynchronous processing through message queues to improve system throughput, achieve reliable task transmission and processing, achieve automated task deployment and execution through task description files, and enable real-time status monitoring for timely detection and handling of task anomalies.

[0074] Figure 3 A flowchart of a training task management method provided in another embodiment of this disclosure is shown below. Figure 3 As shown, the method includes the following steps: S301. Receive the training task submitted by the user. The training task includes the target container orchestration cluster identification information and the configuration information of the training task.

[0075] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0076] S302. Based on the target container orchestration cluster identification information and the training task configuration information, the training task is submitted to the target container orchestration cluster among multiple container orchestration clusters so that the training task can be run through the target container orchestration cluster.

[0077] Specifically, the implementation process and principle of S302 and S102 are the same, and will not be repeated here.

[0078] S303. Create corresponding status monitors for each container orchestration cluster.

[0079] In this step, the management server creates a corresponding status monitor for each container orchestration cluster. The status monitor is used to monitor the status of Jobs and Pods within the container orchestration cluster in real time. Specifically, the management server instantiates a status monitor Tracker for each container orchestration cluster. Each Tracker establishes an independent connection and listening session with its corresponding container orchestration cluster.

[0080] S304. Based on the status monitor corresponding to the target container orchestration cluster, listen to the change information of the running status of the training task and the change information of the running status of each container included in the training task.

[0081] In this step, the state monitor creates a monitoring and notification Informer mechanism for Job and Pod resources in the target container orchestration cluster. These Informers listen for events on all relevant resources and filter out state changes belonging to training tasks and container objects.

[0082] S305. Store the status information of the training task in the database and display the status information of the training task on the user interface.

[0083] In this step, the management server updates the corresponding training task record in the database in real time with the monitored status information. Simultaneously, the front-end interface retrieves the latest task status from the database or API via polling or WebSocket and displays it to the user.

[0084] S306. When a change in the running status of the training task is detected, update the status information of the training task.

[0085] In this step, if a change in the running status of a training task is detected, the status information of the training task will be updated in real time based on the detected status information. Simultaneously, the task status in the corresponding Job and Pod tables in the database will be updated. This is a continuous monitoring process to ensure that the task status in the database remains consistent with the actual status in the target container orchestration cluster.

[0086] S307. When the training task finishes running, obtain the exit status code and exception information of each container included in the training task through the status monitor corresponding to the target container orchestration cluster.

[0087] When a training task finishes running, i.e., when the job status is detected as either "Failed" or "Succeeded," the status monitor will further query the termination details of all container Pods under that job, including exit status codes and exception information, and record them in the database, providing accurate fault diagnosis basis. When the current training task finishes running, the Tracker will also obtain the exit status code (ExitCode) and exception information (Message) for each Pod under the job and write them to the database.

[0088] This embodiment of the disclosure receives a training task submitted by a user. The training task includes a target container orchestration cluster identifier and configuration information. Based on these identifiers, the training task is submitted to a target container orchestration cluster among multiple container orchestration clusters for execution. Further, a corresponding status monitor is created for each container orchestration cluster. Based on the status monitor corresponding to the target container orchestration cluster, changes in the running status of the training task and the running status of each container included in the training task are monitored. The status information of the training task is stored in a database and displayed on the user interface. When a change in the running status of the training task is detected, the status information is updated. When the training task finishes running, the exit status code and exception information of each container included in the training task are obtained through the status monitor corresponding to the target container orchestration cluster. Compared to existing technologies, the embodiments of this disclosure provide independent status monitoring for each cluster, ensuring the accuracy of monitoring, real-time monitoring of task and container status changes, timely detection of problems, providing persistent and visual display of status for easy viewing by users, ensuring the real-time and accuracy of status information, and providing detailed operation result information to facilitate problem analysis and optimization.

[0089] The solutions in the various embodiments of this disclosure can be used individually or in combination without conflict. For example, S201, receiving a container orchestration cluster registration request, registering multiple container orchestration clusters to the management server based on the registration request, and storing the metadata and node information of each container orchestration cluster in the database, and S303, creating corresponding status monitors for each container orchestration cluster, can be used in combination.

[0090] Figure 6 This is a schematic diagram of the structure of a training task management device provided in an embodiment of this disclosure. The training task management device can be a management server as described in the above embodiment, or it can be a component or part of the management server. The training task management device provided in this embodiment can execute the processing flow provided in the training task management method embodiment, such as... Figure 6 As shown, the training task management device 50 is applied to a management server, which manages multiple container orchestration clusters. The device includes a receiving module 51, a submission module 52, and a listening module 53. The receiving module 51 receives training tasks submitted by users, which include target container orchestration cluster identification information and training task configuration information. The submission module 52 submits the training task to the target container orchestration cluster among the multiple container orchestration clusters based on the target container orchestration cluster identification information and the training task configuration information, so that the training task can be run through the target container orchestration cluster. The listening module 53 monitors the running status of the training task based on the status monitor corresponding to the target container orchestration cluster.

[0091] Optionally, before receiving the training task submitted by the user, the device 50 further includes: a registration module 54; the registration module 54 is used to receive a container orchestration cluster registration request, register multiple container orchestration clusters to the management server based on the container orchestration cluster registration request, and store the metadata and node information of each container orchestration cluster in the database.

[0092] Optionally, the configuration information of the training task includes at least one of the following: the framework type of the training task, the number of containers included in the training task, the resource requirements of each container, the start command of the training task, the container image of the training task, the training dataset of the training task, and the storage mount data of the training task.

[0093] Optionally, when the submission module 52 submits the training task to the target container orchestration cluster among multiple container orchestration clusters based on the target container orchestration cluster identifier information and the training task configuration information, it is specifically used for: sending the training task to a message queue, sending the training task's pending message through the message queue, and writing the training task's configuration information into the database; retrieving the training task from the message queue based on the pending message; generating a task description file corresponding to the training task according to the training task's framework type, the number of containers included in the training task, and the resource requirement data of each container; and calling the target container orchestration cluster's interface based on the target container orchestration cluster identifier information to submit the task description file to the target container orchestration cluster.

[0094] Optionally, when the monitoring module 53 monitors the running status of the training task based on the status monitor corresponding to the target container orchestration cluster, it specifically performs the following: creating corresponding status monitors for each container orchestration cluster; monitoring changes in the running status of the training task and the running status of each container included in the training task based on the status monitor corresponding to the target container orchestration cluster; storing the status information of the training task in the database and displaying the status information of the training task on the user interface; updating the status information of the training task when a change in the running status of the training task is detected; and obtaining the exit status code and exception information of each container included in the training task through the status monitor corresponding to the target container orchestration cluster when the training task finishes running.

[0095] Optionally, when running a training task through the target container orchestration cluster, the specific methods are as follows: if the current idle resource data of the target container orchestration cluster is greater than or equal to the sum of the resource requirements of each container included in the training task, schedule each container included in the training task to run on an idle node of the target container orchestration cluster; or, if the current idle resource data is less than the sum of the resource requirements of each container, add the training task to a waiting queue and wait until the current idle resource data is greater than or equal to the sum of the resource requirements of each container, then run the training task.

[0096] Optionally, the device 50 further includes: a deployment module 55; the deployment module 55 is used to deploy multiple first instances in a load balancing manner, and receive training tasks submitted by users through multiple first instances; or, to deploy multiple second instances in a multi-instance manner, and process training tasks in the message queue in parallel through multiple second instances; or, to deploy a status listener in a master-slave mode, and when the master status listener fails, to determine the target status listener from the standby status listeners and to listen based on the target status listener.

[0097] Figure 6 The training task management device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0098] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 7 It shows a schematic diagram of a structure suitable for implementing the electronic device 600 in the embodiments of this disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0099] like Figure 7As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603 to implement the training task management method as described in the embodiments of this disclosure. Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0100] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0101] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the training task management method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of embodiments of this disclosure.

[0102] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0103] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0104] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0105] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: Receive training tasks submitted by users, wherein the training tasks include target container orchestration cluster identification information and configuration information of the training tasks; Based on the target container orchestration cluster identification information and the training task configuration information, the training task is submitted to the target container orchestration cluster among the multiple container orchestration clusters so that the training task can be run through the target container orchestration cluster. Based on the status monitor corresponding to the target container orchestration cluster, the running status of the training task is monitored.

[0106] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also execute other steps of the above embodiments.

[0107] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0110] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0111] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0113] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0114] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A training task management method, characterized in that, Applied to a management server, the management server being used to manage multiple container orchestration clusters, the method includes: Receive training tasks submitted by users, wherein the training tasks include target container orchestration cluster identification information and configuration information of the training tasks; Based on the target container orchestration cluster identification information and the training task configuration information, the training task is submitted to the target container orchestration cluster among the multiple container orchestration clusters so that the training task can be run through the target container orchestration cluster. Based on the status monitor corresponding to the target container orchestration cluster, the running status of the training task is monitored.

2. The method according to claim 1, characterized in that, Before receiving the training task submitted by the user, the method further includes: The system receives a container orchestration cluster registration request, registers multiple container orchestration clusters to the management server based on the request, and stores the metadata and node information of each container orchestration cluster in the database.

3. The method according to claim 1, characterized in that, The configuration information for the training task includes at least one of the following: The training task's framework type, the number of containers included in the training task, the resource requirements of each container, the startup command of the training task, the container image of the training task, the training dataset of the training task, and the storage mount data of the training task.

4. The method according to claim 3, characterized in that, The step of submitting the training task to the target container orchestration cluster among the multiple container orchestration clusters based on the target container orchestration cluster identifier information and the training task configuration information includes: The training task is sent to a message queue, the pending message of the training task is sent through the message queue, and the configuration information of the training task is written into the database. Based on the message to be processed, the training task is obtained from the message queue; Based on the framework type of the training task, the number of containers included in the training task, and the resource requirements of each container, a task description file corresponding to the training task is generated. Based on the target container orchestration cluster identifier information, the interface of the target container orchestration cluster is called to submit the task description file to the target container orchestration cluster.

5. The method according to claim 1, characterized in that, The monitoring of the running status of the training task based on the status monitor corresponding to the target container orchestration cluster includes: Create a corresponding status monitor for each container orchestration cluster; Based on the status monitor corresponding to the target container orchestration cluster, listen for changes in the running status of the training task and changes in the running status of each container included in the training task. The status information of the training task is stored in the database and displayed on the user interface. When a change in the running status of the training task is detected, the status information of the training task is updated; When the training task finishes running, the exit status code and exception information of each container included in the training task are obtained through the status monitor corresponding to the target container orchestration cluster.

6. The method according to claim 1, characterized in that, Running the training task through the target container orchestration cluster includes: If the current idle resource data of the target container orchestration cluster is greater than or equal to the sum of the resource requirements of each container included in the training task, then each container included in the training task will be scheduled to run on an idle node of the target container orchestration cluster; or, If the current idle resource data is less than the sum of the resource requirements of each container, the training task is added to the waiting queue and waited until the current idle resource data is greater than or equal to the sum of the resource requirements of each container, at which point the training task is run.

7. The method according to claim 1, characterized in that, The method further includes: Multiple first instances are deployed using a load balancing method, and these multiple first instances receive training tasks submitted by users; or, Multiple second instances are deployed using a multi-instance approach, and the training tasks in the message queue are processed in parallel by these multiple second instances; or, The status listener is deployed in a primary / backup mode. When the primary status listener fails, the target status listener is determined from the backup status listeners, and monitoring is performed based on the target status listener.

8. A training task management device, characterized in that, Applied to a management server for managing multiple container orchestration clusters, the apparatus includes: The receiving module is used to receive training tasks submitted by users, wherein the training tasks include target container orchestration cluster identification information and configuration information of the training tasks. The submission module is used to submit the training task to the target container orchestration cluster among the multiple container orchestration clusters based on the target container orchestration cluster identification information and the configuration information of the training task, so as to run the training task through the target container orchestration cluster. The monitoring module is used to monitor the running status of the training task based on the status monitor corresponding to the target container orchestration cluster.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images