Task resource elastic scaling method, system and equipment
By deploying shims at nodes, performing hijacking operations and saving checkpoints, the problem of data loss when the training task resource is elastically scaling is solved, and the elastic scaling process without data loss is achieved.
Patent Information
- Application Number
- CN202411524753.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-07
AI Technical Summary
The prior art causes data loss when the training task restarts during the elastic scaling process of training task resources.
Avoid data loss by deploying shims at each node, listening to hijacking records, performing hijacking operations, saving checkpoints for training tasks, and reloading checkpoints after resource scaling.
It realizes that when the training task resources are elastically scalable, there is no loss of data, and there is no intrusion into the training task and no perception of the user.
Smart Images
Figure CN119396587A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of elastic scaling of task resources, and in particular to a method, system and device for elastic scaling of task resources. Background Art
[0002] With the rapid development of artificial intelligence and deep learning technologies, large-scale model training tasks have become a common requirement in research and industrial applications. However, these training tasks usually take a long time and require a lot of computing resources. In order to improve resource utilization and be able to cope with scheduling scenarios including offline, priority preemption, and adaptation, training tasks need to be able to have elastic scaling capabilities.
[0003] Currently, most frameworks used for elastic scaling of task resources for training tasks, such as Pytorch, only focus on the elastic scaling of the algorithm layer, not the elastic scaling of computing power. It is often necessary to manually add computing nodes before starting the training task. When adding and reducing computing nodes, all training tasks will be restarted due to consistency requirements. The elastic orchestration framework provided by k8s generally only considers computing power scaling, and does not consider the problem of training task restart caused by computing power scaling, resulting in data loss of training tasks.
[0004] Therefore, the elastic scaling of task resources in the prior art has the problem of training task data loss when the training task is restarted. Summary of the invention
[0005] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a method, system and device for elastic scaling of task resources, which can solve the problem of elastic scaling of task resources in the prior art that training task data may be lost when the training task is restarted.
[0006] In order to solve the above problems, the present invention is implemented according to the following scheme:
[0007] A task resource elastic scaling method is provided, which is applied to perform task resource scaling on a cluster including multiple nodes, each node being deployed with a shim, and the elastic scaling method comprises:
[0008] Determine the elastic scaling operation based on the elastic scaling request sent by the user;
[0009] When the elastic scaling request is a capacity expansion request, the elastic scaling operation is a capacity expansion operation, including: the elastic controller creates a target container group and generates a hijacking record; the hijacker records the hijacking operation performed by the shim; after the hijacking operation is completed, the shim triggers the task container to restart;
[0010] When the elastic scaling request is a shrinking request, the elastic scaling operation is a shrinking operation, including: the elastic controller marks the state of the target container group and generates a hijacking record; the hijacker records the hijacking operation performed by the shim; after the hijacking operation is completed, the shim deletes the marked target container group and triggers the task container to restart;
[0011] The hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the hijacking record monitored; the task container includes the task resources of the node executing the task.
[0012] Compared with the prior art, the present invention provides a method for elastically scaling task resources with the following beneficial effects: through the hijacking operation of the shim on the training task during the expansion operation and the reduction operation, it is achieved that when the training task is restarted, the training task data will not be lost, and there is no intrusion on the training task, and the user is unaware of the restart of the training task; at the same time, in the reduction operation, the shim deletes and destroys the target container group only after executing the hijacking operation, thereby avoiding the loss of training task data.
[0013] Optionally, the capacity expansion operation further includes: after the elastic controller creates the target container group, the scheduler schedules the target container group to a target node, where the target node is at least one of the multiple nodes in the cluster;
[0014] It is determined whether the target container group is successfully scheduled to the target node. If yes, the elastic controller generates a hijacking record. If no, the capacity expansion operation is terminated.
[0015] Optionally, the target container group includes multiple task replicas, and determining whether the target container group is successfully scheduled to the target node includes determining whether at least one container replica of the target container group is scheduled to the target node.
[0016] Optionally, the hijacking record includes:
[0017] training tasks processed by the cluster;
[0018] The operation name and timestamp of the elastic controller, where the operation name includes the name of the target container group and the name of the state of the target container group, and the timestamp includes the time when the target container group was created and the time when the state of the target container group was marked;
[0019] The status of the hijacking record includes a hijacking start status and a hijacking completion status. The hijacking start status is used to indicate that the shim starts to execute the hijacking operation; the hijacking completion status is used to indicate that the shim ends to execute the hijacking operation.
[0020] Optionally, the hijacking operation includes:
[0021] The shim monitors the hijacking record and determines the training task of the hijacking record;
[0022] The shim calls the save checkpoint method of the training task through a preset sock socket channel to trigger the save checkpoint of the training task;
[0023] Updating the state of the hijacking record to the hijacking start state;
[0024] After the checkpoint is saved, the state of the hijacking record is updated to the hijacking end state.
[0025] Optionally, in the capacity expansion operation, the shim triggers the task container to restart, including:
[0026] The shim of each node triggers the restart of the training task;
[0027] Reloads the checkpoint saved during the hijack operation.
[0028] Optionally, in the scaling-down operation, the shim deletes the target container group in the marked state and triggers the task container to restart, including:
[0029] Determine, according to the target container group, a target shim for performing a deletion operation on the target container;
[0030] The shim of each node triggers the restart of the training task;
[0031] Reloads the checkpoint saved during the hijack operation.
[0032] Optionally, determining, according to the target container group, a target shim for performing a deletion operation on the target container includes:
[0033] Determine the node where the target container group is located as the node to be deleted;
[0034] The gasket of the node to be deleted is determined as a target gasket, and the target gasket deletes the task container of the node to be deleted.
[0035] A task resource elastic scaling system is also provided, which is applied to the above-mentioned task resource elastic scaling method, and includes:
[0036] A shim is deployed in each node and is used to perform a hijacking operation, trigger a task container restart in the expansion operation, and delete a target container group in a marked state and trigger a task container restart in the reduction operation;
[0037] A scheduler, deployed outside the node, for scheduling the target container group to the target node in the capacity expansion operation, where the target node is at least one of the multiple nodes in the cluster;
[0038] Hijacker, deployed outside the node, to record the hijacking operations performed by the shim;
[0039] An elastic controller, deployed outside the node, is used to create a target container group and generate a hijacking record in the capacity expansion operation, and to mark the state of the target container group and generate a hijacking record in the capacity reduction operation;
[0040] The hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the hijacking record monitored; the task container includes the task resources of the node executing the task.
[0041] An electronic device is also provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the above-mentioned task resource elastic scaling method. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a flow chart of the elastic scaling method of the present invention;
[0043] Figure 2 A schematic diagram of the capacity expansion operation performed by the elastic scaling system of the present invention;
[0044] Figure 3 The figure is a schematic diagram of the elastic expansion and contraction system of the present invention performing a capacity reduction operation. DETAILED DESCRIPTION
[0045] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0046] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.
[0047] See also Figure 1As shown, the present invention provides a task resource elastic scaling method, which is applied to perform task resource scaling on a cluster including multiple nodes, each node is deployed with a gasket, and the elastic scaling method includes:
[0048] S1: Determine an elastic scaling operation based on an elastic scaling request sent by a user; wherein the elastic scaling request includes an expansion request and a reduction request; when the elastic scaling request is an expansion request, the elastic scaling operation is an expansion operation, and the expansion operation is used to increase the task resources of the cluster to perform training tasks; when the elastic scaling request is a reduction request, the elastic scaling operation is a reduction operation, and the reduction operation is used to reduce the task resources of the cluster to perform training tasks.
[0049] S21: When the elastic scaling request is a capacity expansion request, the elastic scaling operation performed is a capacity expansion operation, which includes:
[0050] S211: The elastic controller creates a target container group and generates a hijacking record; after the elastic controller creates the target container group, the scheduler needs to schedule the target container group and schedule the target container group to a target node, which is at least one of multiple nodes in the cluster; then it is necessary to determine whether the target container group is successfully scheduled to the target node, the target container group includes multiple container copies, if yes, it means that at least one container copy of the target container group is scheduled to the target node, and the elastic controller generates a hijacking record at this time; if no, it means that the resources of the target node are insufficient to support any container copy in the target container group to be scheduled to the target node, which means that there are no excess resources in the cluster for expansion operation at this time, and the expansion operation is terminated at this time; therefore, determining whether the target container group is successfully scheduled to the target node includes determining whether at least one container copy of the target container group is scheduled to the target node by the scheduler.
[0051] S212: The hijacker records the hijacking operation performed by the shim; the hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the monitored hijacking record.
[0052] S213: After the hijacking operation is completed, the shim triggers the task container to restart; the task container group includes the task resources of each node in the cluster to execute the training task.
[0053] S22: When the elastic scaling request is a shrinking request, the elastic scaling operation performed is a shrinking operation, which includes:
[0054] S221: The elastic controller marks the status of the target container group and generates a hijacking record; the elastic controller marks the status of the target container group, including determining the container group that needs to be deleted, and marking the status of the target container group as being deleted, which is used to indicate that the task resources included in the target container group need to be deleted to reduce the task resources for the cluster to perform training tasks.
[0055] S222: The hijacker records the hijacking operation performed by the shim; the hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the monitored hijacking record.
[0056] S223: After the hijacking operation is completed, the shim deletes the target container group in the marked state and triggers the restart of the task container; the task container group includes the task resources of each node in the cluster to perform the training task; when the shim deletes the target container in the marked state, first, the node where the target container group is located is determined as the node to be deleted, and the node to be deleted is at least one of the multiple nodes in the cluster; then the shim of the node to be deleted is determined as the target shim, and finally the task container of the node to be deleted is deleted through the target shim.
[0057] In step S211 and step S221, the hijacking record generated by the elastic controller includes: the training task processed by the cluster, the operation name and timestamp of the elastic controller, and the status of the hijacking record. The operation name includes the name of the target container group and the name of the status of the target container group. The timestamp includes the time of creating the target container group and the time of marking the status of the target container group. The status of the hijacking record includes the hijacking start status and the hijacking completion status. Whether the hijacking operation is being executed is determined by the status of the hijacking record; wherein, the hijacking start status is used to indicate that the shim starts to execute the hijacking operation, and the hijacking completion status is used to indicate that the shim ends to execute the hijacking operation. In this state, the shim has completed the hijacking operation, that is, when the status of the hijacking record changes from the hijacking start status to the hijacking completion status, it indicates that the shim has ended the process of executing the hijacking operation.
[0058] In step S212 and step S222, the hijacker records the hijacking operation performed by the gasket, wherein the hijacking operation includes: first, the gasket monitors the hijacking record generated by the elastic controller to determine the training task of the hijacking record, which is a training task for cluster processing; then the gasket calls the save (Checkpoint) checkpoint method of the training task through the preset (sock) socket channel to trigger the training task to save the (Checkpoint) checkpoint; and updates the status of the hijacking record to the hijacking start state to indicate that the gasket starts to execute the hijacking operation; when the (Checkpoint) checkpoint is saved, updates the status of the hijacking record to the hijacking end state to indicate that the gasket ends the hijacking operation; usually in the training framework of the training task, the training task represents the Master control node, and the Master is the management node of the distributed training task. Checkpoint is the checkpoint of the distributed training task and is a commonly used task saving technology.
[0059] In both step S213 and step S223, the task container group restart is triggered by the shim to reload the checkpoint saved by the shim in the hijacking operation, that is, to reload the training data of the training task executed by the cluster before the elastic scaling method is performed, thereby avoiding the loss of training data of the training task after the task resources are elastically scaled; during the restart process, specifically, the shim deployed in each node restarts the training task for its corresponding node respectively, and finally the training task data processed by the cluster will not be lost after the task resources are elastically scaled.
[0060] The present invention also provides a task resource elastic scaling system, which is applied to the above elastic scaling method, and includes:
[0061] The shim is deployed in each host node and provides an interface that complies with the CRI standard. It hijacks the entire CRI interface declaration cycle call process and interacts with the training task. When creating a target container group, the shim deploys a sock communication channel and mounts it to the task container to communicate with the training task inside the task container. The shim is used to perform hijack operations, trigger task container restarts during expansion operations, and delete the target container group in the marked state and trigger task container restarts during reduction operations.
[0062] The scheduler is deployed outside the node and is used to schedule the target container group to the target node during the expansion operation. The target node is at least one of the multiple nodes in the cluster.
[0063] Hijacker, deployed outside the node, to record the hijacking operations performed by the shim;
[0064] The elastic controller is deployed outside the node and is used to control the number of target container groups, create target container groups and generate hijacking records in the expansion operation, and mark the status of the target container group and generate hijacking records in the reduction operation. Controlling the number of target container groups includes increasing the number of container groups (expansion operation) and reducing the number of container groups (reduction operation).
[0065] The hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the monitored hijacking record; the task container includes the task resources of the node executing the task.
[0066] Next, combine Figure 2 , the process of performing capacity expansion operations on the elastic scaling system is described in detail:
[0067] First, the elastic controller creates a target container group (Pod in the figure), which includes multiple container replicas. Then the scheduler schedules the target container group (multiple container replicas) created by the elastic controller to schedule the target container group (multiple container replicas) to the target node. Figure 2In the figure, the target node corresponds to the node where shim 2 is located; the cluster includes multiple nodes, and a shim deployed on each node constitutes a shim cluster, which is used to perform hijacking operations and trigger task container restarts. Each node also includes a task container, which includes the task resources of its corresponding node to perform training tasks. The task containers included in each node constitute a training task cluster, that is, a task resource cluster that performs training tasks.
[0068] The scheduler schedules the target container group (multiple container copies) to the target node, that is, Figure 2 In the figure, if the scheduling is successful, the successfully scheduled task copies constitute the task container 2 in the figure. If the resources of the target node are insufficient to support the scheduling of at least one container copy in the target container group to the target node, the expansion operation ends, and the task resources for increasing the cluster to execute the training task fail.
[0069] When the scheduler successfully schedules at least one container copy in the target container group to the target node to form task container 2, the elastic controller generates a hijacking record. The hijacker records the hijacking record generated by the elastic controller and also records the hijacking operation performed by the shim.
[0070] When the shim performs the hijacking operation, each shim is connected to the task container of its corresponding node through a preset sock socket channel to save the task resources of the training task included in the task container, that is, to save the training data of the training task; finally, after the hijacking operation is completed, the training data of the saved training task is reloaded into all task containers in the cluster including the created target container group, thereby avoiding the loss of training data of the training task after the expansion is completed.
[0071] ETCD belongs to the database of the K8s cluster, and API-Server belongs to the API service of the k8s cluster. They are only used as auxiliary expressions in this invention. These two concepts belong to the standard concepts and standard components of native k8s and will not be described in detail here.
[0072] Next, combine Figure 3 , the process of performing a scaling operation on the elastic scaling system is described in detail:
[0073] First, the elastic controller marks the status of the target container group (Pod in the figure), which includes multiple container copies; the cluster includes multiple nodes, and a shim deployed on each node constitutes a shim cluster, which is used to perform hijacking operations and trigger task container restarts. Each node also includes a task container, which includes the task resources of its corresponding node to perform training tasks. The task containers included in each node constitute a training task cluster, that is, a task resource cluster that performs training tasks.
[0074] The target container group corresponds to Figure 3 In task container 2, after the elastic controller marks the status of the target container group, the elastic controller generates a hijacking record. The hijacker records the hijacking record generated by the elastic controller and also records the hijacking operation performed by the shim.
[0075] When the shim performs the hijacking operation, each shim is connected to the task container of its corresponding node through a preset sock socket channel to save the task resources of the training task included in the task container, that is, to save the training data of the training task; finally, after the hijacking operation is completed, the node where the target container group is located is determined, and the node is the node to be deleted. Figure 3 The node where task container 2 is located is located, and then shim shim 2 deletes task container 2 to reduce the task resources of the cluster for executing training tasks. Then, the training data of the saved training tasks is reloaded into the remaining task containers in the cluster except for the deleted task container 2, thereby avoiding the loss of training data of the training tasks after the reduction in capacity is completed.
[0076] ETCD belongs to the database of the K8s cluster, and API-Server belongs to the API service of the k8s cluster. They are only used as auxiliary expressions in this invention. These two concepts belong to the standard concepts and standard components of native k8s and will not be described in detail here.
[0077] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-mentioned elastic scaling method.
[0078] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0079] The memory can be used to store the computer program or module, and the processor implements various functions of the elastic scaling method by running or executing the computer program or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0080] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A task resource elastic scaling method, characterized in that: Applied to scaling task resources for a cluster including multiple nodes, each node is deployed with a shim, and the elastic scaling method includes: Determine the elastic scaling operation based on the elastic scaling request sent by the user; When the elastic scaling request is a capacity expansion request, the elastic scaling operation is a capacity expansion operation, including: the elastic controller creates a target container group and generates a hijacking record; the hijacker records the hijacking operation performed by the shim; after the hijacking operation is completed, the shim triggers the task container to restart; When the elastic scaling request is a shrinking request, the elastic scaling operation is a shrinking operation, including: the elastic controller marks the state of the target container group and generates a hijacking record; the hijacker records the hijacking operation performed by the shim; after the hijacking operation is completed, the shim deletes the marked target container group and triggers the task container to restart; The hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the hijacking record monitored; the task container group includes the task resources of the node executing the training task.
2. A task elastic scaling method according to claim 1, characterized in that: The expansion operation further includes: after the elastic controller creates the target container group, the scheduler schedules the target container group to a target node, where the target node is at least one of the multiple nodes in the cluster; It is determined whether the target container group is successfully scheduled to the target node. If yes, the elastic controller generates a hijacking record. If no, the capacity expansion operation is terminated.
3. A task elastic scaling method according to claim 2, characterized in that: The target container group includes multiple container replicas, and determining whether the target container group is successfully scheduled to the target node includes determining whether at least one container replica of the target container group is scheduled to the target node.
4. A task elastic scaling method according to claim 1, characterized in that: The hijacking records include: training tasks processed by the cluster; The operation name and timestamp of the elastic controller, where the operation name includes the name of the target container group and the name of the state of the target container group, and the timestamp includes the time when the target container group was created and the time when the state of the target container group was marked; The status of the hijacking record includes a hijacking start status and a hijacking completion status. The hijacking start status is used to indicate that the shim starts to execute the hijacking operation; the hijacking completion status is used to indicate that the shim ends to execute the hijacking operation.
5. A task elastic scaling method according to claim 4, characterized in that: The hijacking operation includes: The shim monitors the hijacking record and determines the training task of the hijacking record; The shim calls the save checkpoint method of the training task through a preset socket channel to trigger the save checkpoint of the training task; Updating the state of the hijacking record to the hijacking start state; After the checkpoint is saved, the state of the hijacking record is updated to the hijacking end state.
6. A task elastic scaling method according to claim 5, characterized in that: In the capacity expansion operation, the shim triggers the task container to restart, including: The shim of each node triggers the restart of the training task; Reloads the checkpoint saved during the hijack operation.
7. A task elastic scaling method according to claim 5, characterized in that: In the scaling-down operation, the shim deletes the target container group in the marked state and triggers the task container to restart, including: Determine, according to the target container group, a target shim for performing a deletion operation on the target container; The shim of each node triggers the restart of the training task; Reloads the checkpoint saved during the hijack operation.
8. A task elastic scaling method according to claim 7, characterized in that: The step of determining, according to the target container group, a target shim for performing a deletion operation on the target container includes: Determine the node where the target container group is located as the node to be deleted; The gasket of the node to be deleted is determined as a target gasket, and the target gasket deletes the task container of the node to be deleted.
9. A task resource elastic scaling system, characterized in that: A task resource elastic scaling method applied to the above claims 1-8 comprises: A shim is deployed in each node and is used to perform a hijacking operation, trigger a task container restart in the expansion operation, and delete a target container group in a marked state and trigger a task container restart in the reduction operation; A scheduler, deployed outside the node, for scheduling the target container group to the target node in the capacity expansion operation, where the target node is at least one of the multiple nodes in the cluster; Hijacker, deployed outside the node, to record the hijacking operations performed by the shim; An elastic controller, deployed outside the node, is used to create a target container group and generate a hijacking record in the capacity expansion operation, and to mark the state of the target container group and generate a hijacking record in the capacity reduction operation; The hijacking operation includes the operation performed by the shim on the training task processed by the cluster according to the hijacking record monitored; the task container includes the task resources of the node executing the task.
10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement a task resource elastic scaling method as described in claims 1-8.