Distributed task state monitoring method and device, electronic equipment and storage medium
By creating custom resource objects and minimum scheduling units in a distributed system and utilizing the custom resource types of the Kubernetes system, the problem of difficult unified monitoring of distributed task status is solved, and efficient and robust task lifecycle management and resource scheduling are achieved.
Patent Information
- Application Number
- CN202510624553.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2025-09-19
AI Technical Summary
In a distributed system, the operating status of each container is different, and unified monitoring and management cannot be performed, which makes the lifecycle management of distributed tasks difficult.
By creating custom resource objects and minimum scheduling units, based on the custom resource types of the Kubernetes system, unified lifecycle management of distributed tasks is achieved, the status of the minimum scheduling unit and custom resource objects is monitored in real time, and the monitoring status of distributed tasks is updated.
It achieves precise, efficient and robust lifecycle management of distributed tasks, improves task execution efficiency and success rate, avoids resource scheduling conflicts, and improves the fault tolerance of the system.
Smart Images
Figure CN120670248A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed processing technology, and in particular to a distributed task status monitoring method, device, electronic device and storage medium. Background Art
[0002] When performing distributed processing, creating containers on multiple nodes in a distributed system to execute distributed tasks such as distributed training tasks has become an important way to run distributed tasks.
[0003] At this time, although the lifecycle management of distributed tasks can be performed by monitoring the running status of containers, the running status of each container is different, and unified monitoring and management cannot be performed. It also makes it difficult to rely on the lifecycle management of distributed tasks to optimize the running mode of distributed tasks. Summary of the Invention
[0004] The present invention provides a distributed task status monitoring method, device, electronic device and storage medium to address the defect in the prior art that the execution status of distributed tasks is difficult to uniformly monitor and control, and to achieve precise, efficient and robust unified lifecycle management of distributed tasks.
[0005] The present invention provides a distributed task status monitoring method, comprising: In the case where available resources of available nodes of the distributed system meet the resource requirements of the distributed task to be executed, a custom resource object and a minimum scheduling unit are created, and the distributed task to be executed is distributed to a target node; the target node is used to execute the distributed task to be executed; the resource requirements of the distributed task to be executed are determined based on the distributed task whose monitoring state is submitted; the minimum scheduling unit is used to manage and monitor the target node; and the custom resource object is used to manage and monitor the minimum scheduling unit. Based on the monitoring status of the minimum scheduling unit, updating the monitoring status of the custom resource object; Based on the monitoring status of the custom resource object, the monitoring status of the to-be-executed distributed task is updated.
[0006] A distributed task status monitoring method provided by the present invention further includes: updating the monitoring status of the distributed task to be executed to a waiting state when the available resources of the distributed system do not meet the resource requirements of the distributed task to be executed; Determine whether the available resources of the distributed system meet the resource requirements of the waiting task according to a preset frequency, until the available resources of the distributed system meet the resource requirements of the waiting task, and then jump to the step of creating a custom resource object and a minimum scheduling unit; The waiting task is a distributed task to be executed whose monitoring state is the waiting state.
[0007] According to a distributed task status monitoring method provided by the present invention, updating the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit includes: When the monitoring state of any of the minimum scheduling units is an unschedulable state, updating the monitoring state of the custom resource object to a canceled state; the unschedulable state is determined when the target node corresponding to the minimum scheduling unit changes from an available state to an unavailable state; The updating of the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object includes: When the monitoring state of the custom resource object is a cancel state, updating the monitoring state of the to-be-executed distributed task to a waiting state; After updating the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object, the method further includes: The minimum scheduling unit and the custom resource object are deleted, and the process jumps to the step of determining whether the available resources of the distributed system meet the resource requirements of the waiting task according to the preset frequency.
[0008] According to a distributed task status monitoring method provided by the present invention, updating the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit includes: When the monitoring status of any of the minimum scheduling units is a startup failure state or an execution error state, updating the monitoring status of the custom resource object to an error state; The updating of the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object includes: When the monitoring status of the custom resource object is an error status, the monitoring status of the to-be-executed distributed task is updated to an execution failure status.
[0009] According to a distributed task status monitoring method provided by the present invention, the creation of a custom resource object and a minimum scheduling unit includes: Based on the resource requirements of the distributed task to be executed, create the custom resource object corresponding to the distributed task to be executed; A specified number of the minimum scheduling units are created on multiple target nodes; the specified number is determined based on the minimum scheduling unit number in the resource requirement; and the multiple target nodes are determined from the available nodes based on the node selection rule in the resource requirement.
[0010] According to a distributed task status monitoring method provided by the present invention, after updating the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object, the method further includes: When the monitoring status of the distributed task to be executed is an execution success status or an execution failure status, the minimum scheduling unit and the custom resource object are deleted.
[0011] The present invention also provides a distributed task status monitoring device, comprising: A creation module is configured to create a custom resource object and a minimum scheduling unit when available resources of available nodes of a distributed system meet resource requirements of a distributed task to be executed, and distribute the distributed task to be executed to a target node; the target node is configured to execute the distributed task to be executed; the resource requirements of the distributed task to be executed are determined based on a distributed task whose monitoring state is a submitted state; the minimum scheduling unit is configured to manage and monitor the target node; and the custom resource object is configured to manage and monitor the minimum scheduling unit. A first status updating module, configured to update the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit; The second status updating module is configured to update the monitoring status of the to-be-executed distributed task based on the monitoring status of the custom resource object.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-described distributed task status monitoring methods is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described distributed task status monitoring methods.
[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the distributed task status monitoring method as described above is implemented.
[0015] The distributed task status monitoring method, device, electronic device and storage medium provided by the present invention customize custom resource objects corresponding to distributed tasks, and the custom resource objects uniformly manage and monitor the resources of each node during the execution of distributed tasks by managing and monitoring the minimum scheduling unit. The custom resource objects are monitored by real-time monitoring of the minimum scheduling unit, and the status of distributed tasks is updated by monitoring the custom resource objects. This can clearly and efficiently realize the life cycle management of tasks and the conversion between task states, optimize the management method of distributed tasks such as complex distributed training tasks, thereby improving the efficiency and success rate of distributed tasks, and realizing precise, efficient and robust task life cycle management of distributed task execution; at the same time, by judging whether the resources of the distributed system meet the resource requirements of the distributed task each time a distributed task is executed, resource scheduling conflicts can be avoided and resource scheduling efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0017] Figure 1 This is one of the flow charts of the distributed task status monitoring method provided by the present invention.
[0018] Figure 2 This is the second flow chart of the distributed task status monitoring method provided by the present invention.
[0019] Figure 3 It is a structural diagram of the distributed task status monitoring device provided by the present invention.
[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0021] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0022] It should be noted that, in the description of the present invention, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed. For ordinary technicians in this field, the specific meaning of the above terms in the present invention can be understood according to the specific circumstances.
[0023] The terms "first," "second," and the like in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that embodiments of the present invention can be implemented in orders other than those illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and the like generally refer to a class of objects, and do not limit the number of objects. For example, the first object may be one or more.
[0024] The following combination Figures 1-4 The present invention describes a distributed task status monitoring method, device, electronic device, and storage medium.
[0025] Figure 1 This is one of the flow charts of the distributed task status monitoring method provided by the present invention, such as Figure 1 As shown, the distributed task status monitoring method includes but is not limited to steps 101 to 103.
[0026] It should be noted that the executor of the distributed task status monitoring method provided by the present invention is the corresponding distributed task status monitoring device, which can specifically be a server or computer equipment, such as a mobile phone, tablet computer, laptop computer, PDA, vehicle-mounted electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc.
[0027] Step 101: When available resources of available nodes of a distributed system meet resource requirements of a distributed task to be executed, a custom resource object and a minimum scheduling unit are created, and the distributed task to be executed is distributed to a target node.
[0028] The target node is used to execute the distributed task to be executed; the resource requirement of the distributed task to be executed is determined based on the distributed task whose monitoring status is the submitted status; the minimum scheduling unit is used to manage and monitor the target node; the custom resource object is used to manage and monitor the minimum scheduling unit.
[0029] Among them, the distributed tasks to be executed include but are not limited to distributed training tasks, distributed data processing tasks and other distributed tasks.
[0030] A distributed system consists of multiple nodes that process distributed tasks.
[0031] Taking distributed training tasks as an example, the distributed training tasks are scheduled to multiple nodes in the distributed system for training. Each training node calculates its own iteration results, and each node communicates through the network to calculate a new model.
[0032] A custom resource object (Group) is an object defined based on a custom resource type (CustomResource Definition, CRD) in the Kubernetes system. The minimum scheduling unit (Pod) is a built-in resource in the Kubernetes system.
[0033] Kubernetes is a system for automating the deployment, scaling, and management of containerized applications. A pod is the smallest scheduling unit in the Kubernetes system. A pod encapsulates one or more containers, and the containers in a pod share storage, networking, and other functions.
[0034] The Kubernetes system allows the use of CRDs to extend the Application Programming Interface (API) and create custom resource objects by defining customized resource structures. The Kubernetes system can manage custom resource objects in the same way as it manages built-in resources such as Pods.
[0035] Specifically, a user creates a pending distributed task (job) and submits it to the database. At this point, the monitoring status of the pending distributed task is submitted. The resource requirements required to execute the pending distributed task with the submitted monitoring status are determined, and the available resources of the available nodes in the distributed system are determined to meet the resource requirements of the pending distributed task.
[0036] When the available resources of the available nodes in the distributed system meet the resource requirements of the distributed task to be executed, multiple available nodes in the distributed system with sufficient remaining available resources to execute the distributed task to be executed are identified as target nodes, custom resource objects and minimum scheduling units are created on the target nodes, and the distributed task to be executed is assigned to the multiple target nodes, so that the multiple target nodes jointly execute the distributed task to be executed. Among them, one custom resource object corresponds to one distributed task to be executed, and one target node corresponds to one or more minimum scheduling units. The number of minimum scheduling units is determined based on a user-specified method or an analysis of the distributed task to be executed.
[0037] In the process of target nodes jointly executing the distributed tasks to be executed, the minimum scheduling unit can manage and monitor the target nodes, and the custom resource object manages and monitors the minimum scheduling unit.
[0038] It can be understood that the custom resource object realizes the unified management of the target node by managing the minimum scheduling unit, that is, realizes the centralized management of all resources for running distributed tasks.
[0039] After creating a custom resource object and a minimum scheduling unit, the monitoring status of the custom resource object is unknown, and the monitoring status of the minimum scheduling unit is initialized.
[0040] Optionally, basic information of the minimum scheduling unit is stored in a custom resource object; the basic information of the minimum scheduling unit includes but is not limited to IP address, start time, run time, quantity, port number, etc.
[0041] Step 102: Based on the monitoring status of the minimum scheduling unit, update the monitoring status of the custom resource object.
[0042] Step 103: Based on the monitoring status of the custom resource object, update the monitoring status of the distributed task to be executed.
[0043] Specifically, the minimum scheduling unit updates its own monitoring status by monitoring the status of the target node executing the distributed task to be executed, the custom resource object updates its own monitoring status by monitoring the monitoring status of the minimum scheduling unit, and the distributed task to be executed updates its own monitoring status by monitoring the monitoring status of the custom resource object.
[0044] Figure 2 This is the second flow chart of the distributed task status monitoring method provided by the present invention, such as Figure 2As shown in the figure, five monitoring states are predefined for distributed tasks, including: Job Submit, Job Running, Job Success, Job Failed, and Job Waiting. Seven monitoring states are predefined for custom resource objects, including: Group Unknown, Group Starting, Group Initing, Group Running, Group Cancel, Group Stopped, and Group Error. Seven monitoring states are predefined for the minimum scheduling unit, including: Pod Initing, Pod Starting, Pod Running, Pod Failed, Pod Finished, Pod Error, and Pod Unschedulable.
[0045] After a user creates and submits a distributed task to the database, its initial monitoring status is Job Submit. If the available resources of the distributed system's available nodes meet the resource requirements of the distributed task, a custom resource object is created and a minimum scheduling unit is created on the target node. At this point, the custom resource object's initial monitoring status is Group Unknown, and each minimum scheduling unit's initial monitoring status is Pod Initing.
[0046] After the custom resource object and minimum scheduling unit are created, the minimum scheduling unit begins to initialize its operation. The monitoring status of each minimum scheduling unit is updated to Pod Starting, and the monitoring status of the custom resource object is updated to Group Starting.
[0047] During the process of running initialization of the minimum scheduling unit, if the minimum scheduling unit initialization fails, the monitoring status of the minimum scheduling unit is updated to Pod Failed; if the minimum scheduling unit initialization succeeds, the monitoring status of the minimum scheduling unit is updated to Pod Running.
[0048] If a custom resource object detects that all minimum scheduling units are in the Pod Running state, the custom resource object's monitoring state is updated to Group Initing, and the custom resource object enters the initialization phase. During the initialization phase, the custom resource object calls the engine scheduling unit (Engine Pod) to initialize the custom resource object based on the number of minimum scheduling units, IP addresses, and port numbers. After the custom resource object is initialized, its monitoring state is updated to Group Running, and the engine scheduling unit triggers each minimum scheduling unit to begin executing the pending distributed tasks.
[0049] When the monitoring status of the custom resource object is Group Running, the monitoring status of the distributed task to be executed is updated to Job Running.
[0050] When the target node monitored by the minimum scheduling unit completes the distributed task assigned to it, the monitoring status of the minimum scheduling unit is updated to Pod Finished. If the monitoring status of all minimum scheduling units monitored by the custom resource object is Pod Finished, the monitoring status of the custom resource object is updated to Group Stopped. If the monitoring status of the custom resource object is Group Stopped, the monitoring status of the distributed task to be executed is updated to Job Success, so that users can obtain feedback on the successful execution of the distributed task to be executed.
[0051] Generally speaking, when implementing distributed task lifecycle management, if the monitoring status of the distributed task is updated directly based on the monitoring status of the minimum scheduling unit, users are often required to manually trigger the operation of traversing the monitoring status of the minimum scheduling unit, and then respond to user operations to implement distributed task lifecycle management. However, by leveraging the custom resource types provided by the Kubernetes system to create custom resource objects to monitor and manage the status of distributed tasks, users no longer need to pay attention to the specific details of each minimum scheduling unit and each target node executing the distributed task. Instead, the custom resource object automatically updates its own monitoring status based on the monitoring status of the minimum scheduling unit, and further automatically updates the monitoring status of the distributed task and feedbacks it to the user, achieving unified, efficient, clear, and robust task management.
[0052] The distributed task status monitoring method provided by the present invention customizes a custom resource object corresponding to the distributed task, and the custom resource object uniformly manages and monitors the resources of each node during the execution of the distributed task by managing and monitoring the minimum scheduling unit, monitors the custom resource object by real-time monitoring of the minimum scheduling unit, and updates the status of the distributed task by monitoring the custom resource object. It can clearly and efficiently realize the life cycle management of the task and the conversion between task states, optimize the management method of distributed tasks such as complex distributed training tasks, thereby improving the efficiency and success rate of distributed tasks, and realizing precise, efficient and robust task life cycle management of the distributed task execution; at the same time, by judging whether the resources of the distributed system meet the resource requirements of the distributed task each time a distributed task is executed, resource scheduling conflicts can be avoided and resource scheduling efficiency can be improved.
[0053] Based on the above embodiment, as an optional embodiment, the distributed task status monitoring method further includes: when the available resources of the distributed system do not meet the resource requirements of the distributed task to be executed, updating the monitoring status of the distributed task to be executed to a waiting state; Determine whether the available resources of the distributed system meet the resource requirements of the waiting task according to a preset frequency, until the available resources of the distributed system meet the resource requirements of the waiting task, and then jump to the step of creating a custom resource object and a minimum scheduling unit; The waiting task is a distributed task to be executed whose monitoring state is the waiting state.
[0054] Specifically, combined Figure 2 As shown in the figure, when judging whether the available resources of the available nodes in the distributed system meet the resource requirements of the distributed task to be executed, if the available resources of the available nodes in the distributed system do not meet the resource requirements of the distributed task to be executed, the custom resource object and the minimum scheduling unit are not created for the time being, but the monitoring state of the distributed task to be executed is updated from the submission state Job Submit to the waiting state Job Waiting.
[0055] For the distributed tasks to be executed whose monitoring status is waiting status, that is, waiting tasks, periodic retries are performed according to the preset frequency to determine whether the available resources of the distributed system meet the resource requirements of the waiting tasks. Until it is determined that the available resources of the distributed system meet the resource requirements of the waiting tasks, the process jumps to the step of creating a custom resource object and a minimum scheduling unit, creates a custom resource object and a minimum scheduling unit corresponding to the waiting task, and distributes the waiting task to the target node for executing the waiting task and whose available resources meet the requirements.
[0056] Optionally, a task data table is pre-built to store distributed tasks to be executed and their monitoring status; the task data table is scanned regularly, and waiting tasks whose monitoring status in the task data table is waiting are retried to determine whether there are sufficient available resources. If so, a custom resource object and the minimum scheduling unit corresponding to the waiting task are created.
[0057] The distributed task status monitoring method provided by the present invention improves the fault tolerance of distributed task processing by first querying and determining whether the distributed system has sufficient available resources before running the distributed task, monitoring whether the distributed system resources are normal, and designing a task rescheduling mechanism for the situation where the distributed system resources are insufficient.
[0058] Based on the above embodiment, as an optional embodiment, updating the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit includes: When the monitoring state of any of the minimum scheduling units is an unschedulable state, updating the monitoring state of the custom resource object to a canceled state; the unschedulable state is determined when the target node corresponding to the minimum scheduling unit changes from an available state to an unavailable state; The updating of the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object includes: When the monitoring state of the custom resource object is a cancel state, updating the monitoring state of the to-be-executed distributed task to a waiting state; After updating the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object, the method further includes: The minimum scheduling unit and the custom resource object are deleted, and the process jumps to the step of determining whether the available resources of the distributed system meet the resource requirements of the waiting task according to the preset frequency.
[0059] Specifically, combined Figure 2 As shown in the figure, after the available resources of the distributed system meet the resource requirements of the distributed task to be executed and the custom resource object and minimum scheduling unit are created, the distributed task to be executed is distributed to the target node, which then runs the distributed task to be executed. If the target node corresponding to any minimum scheduling unit experiences an unexpected failure or routine maintenance, and changes from an available state (capable of running the distributed task to be executed) to an unavailable state (unable to run the distributed task to be executed), the monitoring status of the minimum scheduling unit corresponding to the target node is updated from the running state (PodRunning) to the unschedulable state (PodUnschedulable).
[0060] Traverse all minimum scheduling units. If the monitoring status of any minimum scheduling unit managed and monitored by a custom resource object is Pod Unschedulable, the monitoring status of the custom resource object is updated from Group Running to Group Cancel. The monitoring status of the distributed task to be executed is updated from Job Running to Job Waiting. The minimum scheduling unit and custom resource object created for the distributed task to be executed are deleted.
[0061] Jump to the step of determining whether the available resources of the distributed system meet the resource requirements of the waiting task according to the preset frequency, periodically retry the waiting tasks whose monitoring status is waiting in the task data table, and create the minimum scheduling unit and custom resource object corresponding to the waiting task again until the available resources meet the resource requirements of the waiting task.
[0062] The distributed task status monitoring method provided by the present invention uses the minimum scheduling unit to monitor the status of a node in real time. When the node becomes unavailable due to failure or maintenance after starting to execute a task, a rescheduling mechanism is designed to place the to-be-executed distributed task into the waiting task, automatically and regularly retry the waiting task, and reschedule the failed scheduling task to an available node, thereby improving the success rate of the distributed task and the system fault tolerance.
[0063] Based on the above embodiment, as an optional embodiment, updating the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit includes: When the monitoring status of any of the minimum scheduling units is a startup failure state or an execution error state, updating the monitoring status of the custom resource object to an error state; The updating of the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object includes: When the monitoring status of the custom resource object is an error status, the monitoring status of the to-be-executed distributed task is updated to an execution failure status.
[0064] Specifically, combined Figure 2 As shown in the figure, after creating the custom resource object and the minimum scheduling unit, the minimum scheduling unit begins initialization. The monitoring status of each minimum scheduling unit is updated from Pod Initing to Pod Starting. During the minimum scheduling unit initialization process, if the minimum scheduling unit fails to initialize due to a system error, the monitoring status of the minimum scheduling unit is updated from Pod Starting to Pod Failed.
[0065] If the minimum scheduling unit is initialized successfully, its monitoring status is updated from Pod Starting to Pod Running, and it begins monitoring the target node's status as it executes the pending distributed task. If system errors, execution errors, or other errors occur in the minimum scheduling unit or target node while the target node is executing the pending distributed task, the monitoring status of the minimum scheduling unit is updated from Pod Running to Pod Error.
[0066] Traverse all minimum scheduling units. If the monitoring status of any minimum scheduling unit monitored by the custom resource object management is updated to the startup failure status Pod Failed or the execution error status Pod Error, update the monitoring status of the custom resource object from the startup status Group Starting or the running status Group Running to the error status Group Error. Then, update the monitoring status of the distributed task to be executed from the submission status Job Submit or the execution status Job Running to the execution failure status Job Failed, and terminate the execution of the distributed task to be executed, so that the user can obtain feedback on the failure of the distributed task to be executed.
[0067] It can be understood that the monitoring state of the minimum scheduling unit is determined based on the running state of the minimum scheduling unit itself and the running state of the target node corresponding to the minimum scheduling unit executing the distributed task.
[0068] The distributed task status monitoring method provided by the present invention updates the monitoring status of the custom resource object to an error status when a system error, execution error, etc. occurs in the minimum scheduling unit and the target node, so as to determine that the monitoring status of the distributed task to be executed is updated to an execution failure status. It can clearly and efficiently realize the task lifecycle management and the conversion between task states, optimize the management method of distributed tasks such as complex distributed training tasks, and thus realize precise, efficient and robust task lifecycle management of distributed task execution.
[0069] Based on the above embodiment, as an optional embodiment, the step of creating a custom resource object and a minimum scheduling unit includes: Based on the resource requirements of the distributed task to be executed, create the custom resource object corresponding to the distributed task to be executed; A specified number of the minimum scheduling units are created on multiple target nodes; the specified number is determined based on the minimum scheduling unit number in the resource requirement; and the multiple target nodes are determined from the available nodes based on the node selection rule in the resource requirement.
[0070] Specifically, when the available resources of the available nodes of the distributed system meet the resource requirements of the distributed task to be executed, when creating the custom resource object and the minimum scheduling unit, the custom resource object corresponding to the distributed task to be executed is created according to the resource requirements of the distributed task to be executed.
[0071] Furthermore, according to the node selection rules pre-set in the resource requirements, a node selection tool is used to select and determine multiple target nodes from the available nodes in the distributed system, and then one or more minimum scheduling units are constructed on each target node, and the total number of minimum scheduling units constructed on multiple target nodes is the specified number pre-set in the resource requirements, thereby completing the creation of custom resource objects and minimum scheduling units.
[0072] It should be noted that the resource requirements of the distributed tasks to be executed include but are not limited to several types of demand information such as the task code, namespace, resource usage, user directory, and K8s parameters determined when the user creates and submits the distributed tasks to be executed.
[0073] Optionally, taking distributed training as an example of a distributed task to be executed, the resource usage in the resource requirements of the distributed task to be executed includes but is not limited to a deep learning framework, a specified number of minimum scheduling resources (Pods), the number of central processing units (CPUs) used by each minimum scheduling resource Pod, the number of graphics processing units (GPUs) used by each minimum scheduling resource Pod, and the memory used by each minimum scheduling resource Pod, etc. K8s parameters include but are not limited to node selection rules, etc.
[0074] For example, if a user creates a distributed task to be executed and selects a two-machine, four-GPU distributed training task, a custom resource object Group is created to manage all resources used by the distributed training task. Two Pods are created in the Group, each managing four GPU resources. The GPU resources managed by each Pod belong to the same target node, and different Pods in the Group are scheduled by Kubernetes based on available resources. They can be scheduled to different target nodes or the same target node.
[0075] For example, if the node selection rule in a resource requirement specifies that distributed tasks to be executed should only be placed on nodes that support distributed training, then when creating a custom resource object, use the nodeSelector tool in Kubernetes to schedule the custom resource object only on target nodes labeled "distributed_learning," facilitating categorized management of node resources.
[0076] Optionally, the definition format of the resource usage situation in the resource requirements of the distributed task to be executed is in JSON format.
[0077] The distributed task status monitoring method provided by the present invention customizes a custom resource object corresponding to the distributed task, and the custom resource object uniformly manages and monitors the resources of each node during the execution of the distributed task by managing and monitoring the minimum scheduling unit. The custom resource object is monitored by real-time monitoring of the minimum scheduling unit, and the status of the distributed task is updated by monitoring the custom resource object. This method can clearly and efficiently realize the life cycle management of the task and the conversion between task states, optimize the management method of distributed tasks such as complex distributed training tasks, thereby improving the efficiency and success rate of distributed tasks, and realizing precise, efficient and robust task life cycle management of the distributed task execution.
[0078] Based on the above embodiment, as an optional embodiment, after updating the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object, the method further includes: When the monitoring status of the to-be-executed distributed task is an execution success status or an execution failure status, the minimum scheduling unit and the custom resource object are deleted.
[0079] Specifically, when the monitoring status of the distributed task to be executed is Job Success or Job Failed, the execution of the distributed task to be executed is terminated, the custom resource object corresponding to the distributed task to be executed is deleted, and all the minimum scheduling units managed and monitored by the custom resource object are automatically deleted.
[0080] The distributed task status monitoring method provided by the present invention terminates task execution when the monitoring status of the distributed task to be executed is an execution success status or an execution failure status, and deletes the custom resource object and minimum scheduling unit corresponding to the distributed task to be executed, thereby promptly releasing the resources occupied by the distributed task to be executed in the distributed system, thereby improving the processing capability of the distributed system.
[0081] Overall, the distributed task status monitoring method provided by the present invention utilizes Kubernetes' CRD technology to customize custom resource objects Group to correspond one-to-one to distributed task Jobs, and uses custom resource objects Group to uniformly manage the various sub-resource Pods running the distributed tasks, monitor the status of Pods and Groups in real time, and update the status of tasks according to the status of Pods and Groups, thereby realizing efficient and precise management of tasks; and, before creating Groups and Pods, first query whether the distributed system has sufficient available device resources, and implement a task rescheduling mechanism for resource failures and insufficient resources, thereby improving the fault tolerance of the system.
[0082] Figure 3 This is a schematic diagram of the structure of the distributed task status monitoring device provided by the present invention. Figure 3 As shown, the distributed task status monitoring device includes but is not limited to a creation module 301 , a first status update module 302 and a second status update module 303 .
[0083] A creation module 301 is used to create a custom resource object and a minimum scheduling unit when the available resources of the available nodes of the distributed system meet the resource requirements of the distributed task to be executed, and distribute the distributed task to be executed to the target node; the target node is used to execute the distributed task to be executed; the resource requirement of the distributed task to be executed is determined based on the distributed task whose monitoring status is the submitted status; the minimum scheduling unit is used to manage and monitor the target node; and the custom resource object is used to manage and monitor the minimum scheduling unit.
[0084] The first status updating module 302 is configured to update the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit.
[0085] The second status updating module 303 is configured to update the monitoring status of the to-be-executed distributed task based on the monitoring status of the custom resource object.
[0086] It should be noted that the distributed task status monitoring device provided by the present invention can execute the distributed task status monitoring method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.
[0087] The distributed task status monitoring device provided by the present invention customizes a custom resource object corresponding to the distributed task, and the custom resource object manages and monitors the resources of each node during the execution of the distributed task by managing and monitoring the minimum scheduling unit. The custom resource object is monitored by real-time monitoring of the minimum scheduling unit, and the status of the distributed task is updated by monitoring the custom resource object. It can clearly and efficiently realize the life cycle management of the task and the conversion between task states, optimize the management method of distributed tasks such as complex distributed training tasks, thereby improving the efficiency and success rate of distributed tasks, and realizing precise, efficient and robust task life cycle management of distributed task execution; at the same time, by judging whether the resources of the distributed system meet the resource requirements of the distributed task each time a distributed task is executed, resource scheduling conflicts can be avoided and resource scheduling efficiency can be improved.
[0088] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to execute the distributed task status monitoring method provided in any of the above embodiments, the distributed task status monitoring method including but not limited to the following steps: when the available resources of the available nodes of the distributed system meet the resource requirements of the distributed task to be executed, creating a custom resource object and a minimum scheduling unit, and distributing the distributed task to be executed to a target node; the target node is used to execute the distributed task to be executed; the resource requirements of the distributed task to be executed are determined based on the distributed task whose monitoring status is submitted; the minimum scheduling unit is used to manage and monitor the target node; the custom resource object is used to manage and monitor the minimum scheduling unit; based on the monitoring status of the minimum scheduling unit, updating the monitoring status of the custom resource object; based on the monitoring status of the custom resource object, updating the monitoring status of the distributed task to be executed.
[0089] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the distributed task status monitoring method provided by any of the above embodiments, and the distributed task status monitoring method includes but is not limited to the following steps: when the available resources of the available nodes of the distributed system meet the resource requirements of the distributed task to be executed, create a custom resource object and a minimum scheduling unit, and distribute the distributed task to be executed to the target node; the target node is used to execute the distributed task to be executed; the resource requirement of the distributed task to be executed is determined based on the distributed task whose monitoring status is the submission status; the minimum scheduling unit is used to manage and monitor the target node; the custom resource object is used to manage and monitor the minimum scheduling unit; based on the monitoring status of the minimum scheduling unit, update the monitoring status of the custom resource object; based on the monitoring status of the custom resource object, update the monitoring status of the distributed task to be executed.
[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed task status monitoring method provided by any of the above embodiments, the distributed task status monitoring method including but not limited to the following steps: when the available resources of the available nodes of the distributed system meet the resource requirements of the distributed task to be executed, creating a custom resource object and a minimum scheduling unit, and distributing the distributed task to be executed to a target node; the target node is used to execute the distributed task to be executed; the resource requirement of the distributed task to be executed is determined based on the distributed task whose monitoring status is the submitted status; the minimum scheduling unit is used to manage and monitor the target node; the custom resource object is used to manage and monitor the minimum scheduling unit; based on the monitoring status of the minimum scheduling unit, the monitoring status of the custom resource object is updated; based on the monitoring status of the custom resource object, the monitoring status of the distributed task to be executed is updated.
[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.
[0093] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.
[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A distributed task status monitoring method, characterized in that: include: In the case where available resources of available nodes of the distributed system meet the resource requirements of the distributed task to be executed, a custom resource object and a minimum scheduling unit are created, and the distributed task to be executed is distributed to a target node; the target node is used to execute the distributed task to be executed; the resource requirements of the distributed task to be executed are determined based on the distributed task whose monitoring state is submitted; the minimum scheduling unit is used to manage and monitor the target node; and the custom resource object is used to manage and monitor the minimum scheduling unit. Based on the monitoring status of the minimum scheduling unit, updating the monitoring status of the custom resource object; Based on the monitoring status of the custom resource object, the monitoring status of the to-be-executed distributed task is updated.
2. The distributed task status monitoring method according to claim 1, characterized in that: Also includes: When the available resources of the distributed system do not meet the resource requirements of the distributed task to be executed, updating the monitoring state of the distributed task to be executed to a waiting state; Determine whether the available resources of the distributed system meet the resource requirements of the waiting task according to a preset frequency, until the available resources of the distributed system meet the resource requirements of the waiting task, and then jump to the step of creating a custom resource object and a minimum scheduling unit; The waiting task is a distributed task to be executed whose monitoring state is the waiting state.
3. The distributed task status monitoring method according to claim 2, characterized in that: The updating of the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit includes: When the monitoring state of any of the minimum scheduling units is an unschedulable state, updating the monitoring state of the custom resource object to a canceled state; the unschedulable state is determined when the target node corresponding to the minimum scheduling unit changes from an available state to an unavailable state; The updating of the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object includes: When the monitoring state of the custom resource object is a cancel state, updating the monitoring state of the to-be-executed distributed task to a waiting state; After updating the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object, the method further includes: The minimum scheduling unit and the custom resource object are deleted, and the process jumps to the step of determining whether the available resources of the distributed system meet the resource requirements of the waiting task according to the preset frequency.
4. The distributed task status monitoring method according to claim 1, characterized in that: The updating of the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit includes: When the monitoring status of any of the minimum scheduling units is a startup failure state or an execution error state, updating the monitoring status of the custom resource object to an error state; The updating of the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object includes: When the monitoring status of the custom resource object is an error status, the monitoring status of the to-be-executed distributed task is updated to an execution failure status.
5. The distributed task status monitoring method according to claim 1, characterized in that: The creation of the custom resource object and the minimum scheduling unit includes: Based on the resource requirements of the distributed task to be executed, create the custom resource object corresponding to the distributed task to be executed; A specified number of the minimum scheduling units are created on multiple target nodes; the specified number is determined based on the minimum scheduling unit number in the resource requirement; and the multiple target nodes are determined from the available nodes based on the node selection rule in the resource requirement.
6. The distributed task status monitoring method according to claim 1, characterized in that: After updating the monitoring status of the distributed task to be executed based on the monitoring status of the custom resource object, the method further includes: When the monitoring status of the distributed task to be executed is an execution success status or an execution failure status, the minimum scheduling unit and the custom resource object are deleted.
7. A distributed task status monitoring device, characterized in that: include: A creation module is configured to create a custom resource object and a minimum scheduling unit when available resources of available nodes of a distributed system meet resource requirements of a distributed task to be executed, and distribute the distributed task to be executed to a target node; the target node is configured to execute the distributed task to be executed; the resource requirements of the distributed task to be executed are determined based on a distributed task whose monitoring state is a submitted state; the minimum scheduling unit is configured to manage and monitor the target node; and the custom resource object is configured to manage and monitor the minimum scheduling unit. A first status updating module, configured to update the monitoring status of the custom resource object based on the monitoring status of the minimum scheduling unit; The second status updating module is configured to update the monitoring status of the to-be-executed distributed task based on the monitoring status of the custom resource object.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the distributed task status monitoring method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed task status monitoring method according to any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the distributed task status monitoring method according to any one of claims 1 to 6 is implemented.