A data center computing power maintenance scheduling method based on task interruption sensitivity

Abstract

This invention discloses a data center computing power maintenance scheduling method based on task interruption sensitivity, relating to the field of data center computing power operation and maintenance. The method acquires maintenance information, computing resource status, and task execution information of the computing nodes to be maintained; determines the set of affected tasks based on node occupancy relationships; and determines the task interruption sensitivity and single-node interruption impact value. For nodes with shared affected tasks or task dependencies, it determines the directional joint interruption impact value, calculates the directional interruption coupling amount, and constructs a maintenance coupling graph. Based on the task execution status recovery, it determines the minimum recovery interval; combines the maintenance duration and maintenance time limit to determine the maintenance sequence and start time; and sends maintenance control commands to the node management interface. This invention can reduce the impact of task interruptions caused by continuous maintenance of associated nodes, improving the continuity of data center computing power services and the reliability of maintenance scheduling.

Description

Technical Field

[0001] This invention relates to the field of data center computing power operation and maintenance, and in particular to a data center computing power maintenance scheduling method based on task interruption sensitivity. Background Technology

[0002] With the development of cloud computing, artificial intelligence training, and high-performance computing services, data centers typically deploy a large number of servers, CPU computing nodes, GPU computing nodes, and other computing resources, and use cluster management systems to schedule multiple computing tasks in parallel. To ensure the long-term stable operation of computing nodes, maintenance operations such as shutdown for inspection, restart maintenance, or component replacement are required for some computing nodes based on equipment operating status, maintenance cycle, or fault handling needs.

[0003] In data centers that continuously provide computing services, compute nodes awaiting maintenance often carry running computing tasks. When a compute node enters maintenance mode, its corresponding computing resources will be temporarily unavailable, potentially leading to computation rollback, restoration of running status, a decrease in available computing resources, or task execution delays. For distributed computing, parallel training, or tasks with replication and communication synchronization relationships, a running task may simultaneously occupy multiple compute nodes; therefore, maintenance operations on different compute nodes may affect each other.

[0004] Existing data center maintenance scheduling methods typically determine the maintenance timing or order for each computing node based on its own resource load, number of tasks, task priority, or the impact of a single node going out of service. However, for multiple computing nodes that share the same task or have computational collaboration, communication synchronization, replication, and execution dependencies, when one node enters maintenance mode, related tasks may be undergoing resource degradation, state recovery, or resynchronization. If another related computing node enters maintenance mode before this impact has subsided, the combined interruption impact may be significantly higher than the sum of the impacts of the two computing nodes undergoing maintenance separately, and may even cause running tasks to fail to meet the minimum runnable conditions.

[0005] Therefore, prioritizing maintenance based solely on the independent impact of each computing node's maintenance is insufficient to accurately reflect the directional combined effects of different computing nodes entering maintenance at varying sequences and time intervals. It also makes it difficult to reasonably determine the recovery waiting times required between related computing nodes. Furthermore, when data center tasks, resource allocation, or computing node status change, the pre-determined maintenance plan may not match the actual operating conditions, leading to unnecessary computing power interruptions, increased task recovery overhead, and decreased computing service continuity.

[0006] To address this, a data center computing power maintenance scheduling method based on task interruption sensitivity is proposed. Summary of the Invention

[0007] The main objective of this invention is to provide a data center computing power maintenance and scheduling method based on task interruption sensitivity, which can effectively solve the problems in the background art.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A data center computing power maintenance scheduling method based on task interruption sensitivity includes the following steps:

[0010] S1. Obtain maintenance information, computing resource status, and task execution information of multiple computing nodes to be maintained in the data center; wherein, the maintenance information includes at least the maintenance duration and maintenance time limit, and the task execution information includes at least the node occupancy relationship between the task and the computing node, the task execution status, and the minimum runnable conditions of the task.

[0011] S2. Determine the set of affected tasks corresponding to each computing node to be repaired based on the node occupancy relationship; for each computing node to be repaired, set the computing resources corresponding to the computing node to be repaired to an unavailable state, determine the corresponding task interruption sensitivity based on at least one of the technical state changes in computing rollback, state recovery, decrease in available computing resources and task execution delay caused by each affected task, and determine the single node interruption impact value of the computing node to be repaired based on the task interruption sensitivity of each affected task.

[0012] S3. For two computing nodes to be inspected that have common affected tasks or task dependencies, one of the computing nodes to be inspected is designated as the preceding inspection node and the other as the following inspection node. For different candidate time intervals τ, if the task impact generated by the preceding inspection node has not yet been recovered, the computing resources corresponding to the following inspection node are set to an unavailable state to obtain the corresponding directional joint interruption impact value.

[0013] S4. Calculate the directional interruption coupling amount based on the combined directional interruption impact value, the single-node interruption impact value of the preceding maintenance node, and the reference interruption impact value corresponding to the subsequent maintenance node entering maintenance state alone under the same time interval:

[0014]

[0015] in, Indicates the computing node to be inspected. First enter maintenance mode and wait for the time interval. Subsequent maintenance calculation nodes Directional interruption coupling amount when entering maintenance mode This indicates the corresponding directional joint interruption impact value. Indicates the computing node to be inspected. The impact value of a single-node interrupt. Indicates the computing node to be inspected. Under reference conditions where maintenance has not yet commenced, after a time interval Subsequent maintenance calculation nodes Reference interrupt impact value corresponding to entering maintenance state independently;

[0016] S5. When the directional interruption coupling amount exceeds the preset coupling threshold, or when two computing nodes to be repaired enter the repair state in sequence and there are affected tasks that do not meet the minimum operable conditions, it is determined that there is a directional repair coupling relationship between the two computing nodes to be repaired; a repair coupling graph is constructed with the computing nodes to be repaired as nodes and the directional repair coupling relationship as directed connection edges.

[0017] S6. For each directional maintenance coupling relationship in the maintenance coupling diagram, based on the recovery status of the operation status of the affected tasks after the preceding maintenance node enters the maintenance state, determine the minimum recovery interval that allows the subsequent maintenance node to enter the maintenance state.

[0018] S7. Based on the maintenance coupling diagram, the maintenance duration and maintenance time limit of each computing node to be maintained, and the minimum recovery interval, determine the maintenance sequence and maintenance start time of each computing node to be maintained, and send maintenance control instructions to the node management interface of the corresponding computing node to be maintained according to the determined maintenance sequence and maintenance start time, so that the corresponding computing node to be maintained enters the maintenance state.

[0019] Furthermore, the node occupancy relationship is used to characterize one or more computing nodes actually occupied by the running task; the task dependency association includes at least one of the following: computing collaboration relationship, communication synchronization relationship, replication relationship, or task execution dependency relationship between computing nodes corresponding to the same running task.

[0020] Furthermore, the determination of the task interruption sensitivity includes:

[0021] Determine at least one of the following interruption impact indicators when the computing node to be repaired enters an unavailable state: computing rollback amount, state recovery data amount, decrease in available computing resources, and task execution delay amount.

[0022] The interruption impact index is processed to unify the dimensions, and the task interruption sensitivity of the corresponding affected task is determined based on the processed interruption impact index.

[0023] The single-node interruption impact value of the computing node to be repaired is determined based on the task interruption sensitivity of each affected task corresponding to the computing node to be repaired.

[0024] Furthermore, obtaining the directional joint interruption impact value includes:

[0025] Set the computing resources of the preceding maintenance node to an unavailable state, and determine the task running status of the corresponding affected task over time based on the remaining available computing resources;

[0026] For candidate time interval Furthermore, the computing resources of subsequent maintenance nodes are set to an unavailable state, and the running tasks that are simultaneously affected by two computing nodes to be maintained, as well as the running tasks that are indirectly affected through task dependency associations, are identified.

[0027] Based on at least one of the following: remaining available computing resources, communication connection status, replica availability, and task execution status, determine whether the corresponding running task meets the minimum runnable conditions, and determine the directional joint interruption impact value. .

[0028] Furthermore, when the directional interruption coupling amount is greater than the preset coupling threshold, or when at least one affected task does not meet the minimum operable conditions after the preceding and subsequent maintenance nodes enter the maintenance state in sequence, the corresponding directed connection edge is determined as a strong coupling edge, and maintenance interval constraints are set for the two computing nodes to be maintained corresponding to the strong coupling edge.

[0029] Furthermore, regarding the computing nodes awaiting maintenance... Point to the computing node to be inspected Strongly coupled edges, in the candidate time interval set The minimum recovery interval is determined according to the following formula:

[0030]

[0031] in, Indicates the computing node to be inspected. Entering maintenance status until the computing node is allowed to be maintained. Minimum recovery interval before entering maintenance status. This represents the preset coupling threshold. Indicates the time interval between candidates The runtime state vector of the subsequently affected task. This represents the stable operating state domain determined by the minimum operability conditions of the affected task.

[0032] Furthermore, a maintenance sequence is generated based on the maintenance coupling diagram. And the candidate maintenance plans at the start time of maintenance for each node to be maintained, and the maintenance impact value of the candidate maintenance plans is determined according to the following formula:

[0033]

[0034] in, This indicates the maintenance impact value corresponding to the candidate maintenance plan. This represents the set of maintenance start times for each computational node to be inspected. Indicates the calculation node to be inspected. At the start of maintenance The impact value of a single-node interrupt is determined by the corresponding task's running status. This represents the set of computational nodes to be maintained in the candidate maintenance plan whose maintenance impact cycles are adjacent or overlap. This indicates the preset coupling influence coefficient;

[0035] Under the condition that the maintenance time limit of each computing node to be maintained and the time interval corresponding to the strongly coupled edge is not less than the corresponding minimum recovery interval are met, the target maintenance plan is determined according to the maintenance impact value.

[0036] Furthermore, during the maintenance process, when the node occupancy relationship, task execution status, computing resource status, or maintenance status of the computing node to be maintained changes, the nodes and directed connection edges affected by the changes in the maintenance coupling graph are determined, the corresponding directional interruption coupling amount and minimum recovery interval are recalculated, and the maintenance sequence or maintenance start time of the computing nodes to be maintained that have not yet been executed is adjusted according to the updated maintenance coupling graph.

[0037] A data center computing power maintenance and scheduling system based on task interruption sensitivity includes a cluster management controller, a task status acquisition interface, and a node management interface connected to multiple computing nodes. The cluster management controller is communicatively connected to the task status acquisition interface and the node management interface.

[0038] The task status acquisition interface is used to obtain the task running information of the currently running task and the computing resource status of each computing node;

[0039] The cluster management controller is used to determine the task interruption sensitivity and single-node interruption impact value of each computing node to be repaired based on the node occupancy relationship between tasks and computing nodes. For computing nodes to be repaired that have common affected tasks or task dependencies, it determines the directional joint interruption impact value under different repair sequence relationships and time intervals. Based on the directional joint interruption impact value, it determines the directional interruption coupling amount and constructs a repair coupling graph. Based on the repair coupling graph, it determines the minimum recovery interval, repair sequence, and repair start time among the computing nodes to be repaired.

[0040] The node management interface is used to receive maintenance control commands sent by the cluster management controller and control the corresponding computing node to be maintained to enter the maintenance state.

[0041] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This invention determines the task interruption sensitivity based on the changes in technical status of affected tasks after a computing node enters an unavailable state, such as computation rollback, state recovery, decrease in available computing resources, and task execution delay. This enables the maintenance impact assessment to reflect the technical impact of computing node shutdown on the actual computing power task execution process. Compared with maintenance scheduling based solely on node load or task priority, this invention improves the accuracy of maintenance impact assessment.

[0044] 2. This invention targets computing nodes that have common affected tasks or task dependencies. It determines the directional joint interruption impact by combining different maintenance sequence relationships and time intervals, and further determines the directional interruption coupling relationship. It can identify computing node combinations that have a small impact from individual maintenance but may cause insufficient task resources, communication interruption or unstable operation after continuous maintenance, thereby reducing the nonlinear task interruption impact generated during the maintenance of multiple computing nodes.

[0045] 3. This invention determines the minimum recovery interval between associated computing nodes based on the directional interruption coupling relationship and the recovery status of the affected tasks, and determines the maintenance sequence and start time of the computing nodes accordingly. This ensures that subsequent computing nodes enter the maintenance state only after the impact of the preceding maintenance has been reduced to an acceptable range and the related tasks have been restored to meet the minimum runnable conditions. This helps to reduce repeated task recovery and computation rollback, and maintain the continuity of available computing power and computing services in the data center.

[0046] 4. During the maintenance process, this invention can update the maintenance coupling relationship and minimum recovery interval based on changes in task node occupancy, task execution status, computing resource status, and node maintenance status, and adjust maintenance plans that have not yet been executed, so that the maintenance scheduling results can adapt to the dynamic operating status of the data center and improve the reliability and resource utilization efficiency of the continuous maintenance process of multiple computing nodes. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0048] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] Example 1

[0051] like Figure 1-2 As shown, a data center computing power maintenance and scheduling method based on task interruption sensitivity is applied to a data center including a cluster management controller, a task status acquisition interface, a node management interface, and multiple computing nodes. The computing nodes can be physical servers configured with CPUs, GPUs, NPUs, or other computing processors, or they can be computing units formed by partitioning physical servers and capable of independent resource management.

[0052] The cluster management controller acquires task execution information of currently running tasks and the computing resource status of each computing node through the task status acquisition interface, and sends maintenance control commands to the computing node to be maintained through the node management interface. The node management interface can be implemented using a server management interface, an out-of-band management interface, or a node control interface provided by the cluster management platform.

[0053] The method in this embodiment includes the following steps.

[0054] S1. Obtain maintenance information and task execution information of the computing node to be inspected.

[0055] First, the cluster management controller obtains multiple compute nodes that need to be maintained during the current maintenance cycle and forms a set of compute nodes to be maintained:

[0056]

[0057] in, Indicates the first One computing node awaiting maintenance. This indicates the number of calculation nodes to be inspected.

[0058] For each computing node to be inspected, obtain the corresponding inspection information, including the inspection duration, the allowed inspection time range, and the inspection time limit.

[0059] For example, for computing nodes that need to be inspected Its maintenance information can be represented as:

[0060]

[0061] in, Represents a computing node The estimated duration of the maintenance, This indicates the earliest time when maintenance is permitted to begin. This indicates the required timeframe for completing the maintenance.

[0062] At the same time, the set of tasks currently running in the data center is obtained through the task status acquisition interface:

[0063]

[0064] For any running task It can obtain the computing nodes it currently occupies, the stage of task execution, the amount of computing completed, the amount of computing to be completed, the currently available computing resources, and the minimum conditions for it to run.

[0065] The minimum runnable condition is used to represent the minimum resource and state requirements that a task needs to meet to continue to execute normally.

[0066] For example, the minimum runnable conditions for a distributed computing task may include: at least 4 GPU computing units; at least 2 task replicas; the number of valid worker nodes in the communication group is not less than a preset number; key parameter replicas are in an accessible state; and the current task stage has a recoverable running state.

[0067] Different minimum runnable conditions can be used for different types of tasks, and it is not required that all running tasks use the exact same conditions.

[0068] S2. Determine the task interruption sensitivity and the impact of a single-node interruption.

[0069] Establish node occupancy relationships based on the actual computing nodes occupied by the currently running task.

[0070] For example: task Occupying nodes ;Task Occupying nodes ;Task Occupying nodes .

[0071] When it is necessary to analyze the computing nodes to be inspected When a single node is affected by maintenance, the node will be managed in the cluster management controller. The corresponding computing resources are marked as simulated unavailable, without immediately powering down the actual nodes.

[0072] Subsequently, the affected node is determined based on the node occupancy relationship. The set of tasks affected:

[0073]

[0074] in, Indicates running task The set of computing nodes currently in use.

[0075] For any affected task The calculation rollback amount, state recovery data amount, decrease in available computing resources, and task execution delay amount generated after a node exits operation are obtained respectively.

[0076] Specifically, the computation rollback amount can be determined based on the difference between the current execution position of the task and the most recent recoverable state. For example, if a training task has been executed up to the 10,500th computation batch, and the most recent effective recovery state corresponds to the 10,000th computation batch, then when a node exits and the task needs to be re-executed from the most recent recovery state, the computational workload corresponding to 500 batches can be used as the computation rollback amount.

[0077] The amount of state recovery data can be determined based on the amount of memory state, video memory state, intermediate task data, model parameters, or computational state data that needs to be reloaded or retransmitted as required.

[0078] The decrease in available computing resources can be determined based on the changes in the number of processors, GPUs, computing throughput, or actual available computing power of a task before and after a node exits.

[0079] The task execution delay can be determined based on the increase in the remaining execution time of the task after the node exits.

[0080] Since the units of the above-mentioned indicators are different, their dimensions can be unified before comprehensive processing. For example, for any indicator... The following can be adopted:

[0081]

[0082] Normalization was performed, where, and It can be determined based on historical operational data of similar tasks within a predetermined statistical period, or it can be determined based on the upper and lower limits allowed by the corresponding indicators.

[0083] In one specific implementation, the task Subject to node The task interruption sensitivity when exiting an effect can be expressed as:

[0084]

[0085] in, Indicates task Relative to node Task interruption sensitivity; This represents the normalized calculation backoff amount; This represents the amount of data recovered after normalization. This represents the decrease in available computing resources after normalization. This represents the normalized task execution latency. These represent the weights of the corresponding interruption impact indicators.

[0086] The above weights can be pre-configured according to the data center operation strategy and must meet the following requirements:

[0087]

[0088] It should be noted that the above weighted model is only one implementation of task interruption sensitivity. Piecewise functions, lookup tables, or different interruption sensitivity calculation rules can also be used according to the task type.

[0089] If a node exits and causes a task to fail to meet the minimum runnable conditions, a failure penalty can be configured on top of the sensitivity mentioned above, making the interrupt sensitivity of this type of task significantly higher than that of a task that only experiences performance degradation.

[0090] Furthermore, based on the nodes The corresponding task interruption sensitivity determination node for each affected task Single-node interruption impact value .

[0091] For example, the following can be used:

[0092]

[0093] in, For the task The task impact coefficient is used to reflect the different technical requirements of different tasks for computing power continuity.

[0094] In other implementations, the impact value of a single-node interrupt can be determined by combining the maximum interrupt sensitivity with the cumulative interrupt sensitivity.

[0095] S3. Determine the impact value of directional joint interruption.

[0096] After obtaining the single-node interruption impact value of the computing node to be inspected, we further analyze the combined impact of multiple computing nodes entering the maintenance state consecutively.

[0097] To reduce unnecessary pairwise calculations, we can prioritize analyzing computation node pairs that share common affected tasks or have task dependencies.

[0098] Task dependency relationships can include: computational collaboration relationships between nodes corresponding to the same task; communication synchronization relationships between distributed computing nodes; primary or multiple replica relationships; and execution dependencies between preceding and subsequent tasks.

[0099] For example, nodes and Simultaneously participate in the task If both are affected by the same task, then they share a common impact. For example, a node... Run task ,node Run task Although the two did not run the same task, the task Task required The intermediate calculation results show that there is a task execution dependency between the two.

[0100] For nodes and First, set the nodes As a preliminary maintenance node.

[0101] In the simulation environment, nodes The corresponding computing resources are set to an unavailable state, and the running status of the affected tasks is updated according to the task recovery process.

[0102] Set nodes At any moment Entering maintenance mode, for candidate time intervals:

[0103]

[0104] At that moment:

[0105]

[0106] Further nodes The corresponding computing resources are set to an unavailable state.

[0107] At this point, the remaining available computing resources, communication connection status, replica availability, task execution status, and whether the minimum runnable conditions are met for each running task are recalculated.

[0108] The combined impact of the two nodes entering an unavailable state according to the above time relationship is denoted as:

[0109]

[0110] That is, the impact value of the directional joint interruption.

[0111] It should be noted that, Not only considering simultaneous occupation of nodes and For tasks that are indirectly affected by task dependencies, it is also possible to consider running tasks that are indirectly affected by task dependencies.

[0112] For example, if node Exiting the task The output delay, and the task Although not running on the node However, it depends on the task. The output of the task The waiting time and performance degradation can also be included in the impact of directional joint interruption.

[0113] S4. Calculate the directional interruption coupling amount

[0114] If only:

[0115]

[0116] As a coupling quantity, when the running task state changes naturally over time, the effects of this time change may be mistakenly interpreted as the coupling effect between the two nodes.

[0117] Therefore, this embodiment uses nodes under the same candidate time interval. The impact of entering maintenance mode alone is used as a reference baseline.

[0118] Specifically, at the node Under reference conditions where no maintenance has commenced, the time interval from the same starting point... Then, the node Set it to an unavailable state separately and determine the corresponding reference interrupt impact value:

[0119]

[0120] Then calculate the nodes Pointing to node Directional interruption coupling amount:

[0121]

[0122] This directional interruption coupling is used to represent: when already affected by a node In the event of maintenance disruptions, the node The additional disruption caused by re-entering the maintenance state cannot be simply explained by the maintenance effects of the two nodes independently.

[0123] when:

[0124]

[0125] When, it indicates that there is a positive additional interruption effect between the two computing nodes in the corresponding order and time interval.

[0126] It should be noted that, generally:

[0127]

[0128] For example, nodes After exiting, it takes a considerable amount of time to rebuild the replica, and the node... If the task can be quickly resumed after exiting, then perform maintenance first. Re-inspection Prior to inspection Re-inspection The resulting combined effects are different. Therefore, this embodiment uses a directed interruption coupling relationship instead of undirected node dependence.

[0129] S5. Construct a maintenance coupling diagram

[0130] The directional interruption coupling between computing nodes to be inspected is determined.

[0131] When at least one of the following conditions is met: First, the directional interruption coupling exceeds a preset coupling threshold; Second, after two computing nodes enter maintenance state in sequence, at least one of the affected tasks does not meet the minimum runnable conditions; then it is considered that there is a directional maintenance coupling relationship between the two corresponding nodes.

[0132] Therefore, a maintenance coupling diagram is established:

[0133]

[0134] in, This represents the set of nodes that need to be inspected and repaired. This represents the set of directed edges that form the directional maintenance coupling relationships.

[0135] For example, in the maintenance coupling diagram:

[0136]

[0137] Indicates at node If the node enters maintenance mode first, Maintenance cannot be initiated at any time; key points must be considered. Has the impact of the task been recovered?

[0138] When the directional interruption coupling exceeds the preset coupling threshold, or when a task does not meet the minimum runnable conditions, the corresponding edge can be further marked as a strongly coupled edge.

[0139] S6. Determine the minimum recovery interval.

[0140] For nodes Pointing to node Strongly coupled edges, obtain nodes The recovery status of affected tasks over time after entering maintenance status.

[0141] The running state vector can be represented as:

[0142]

[0143] Each state component can represent the current remaining available computing power, the proportion of normally operating replicas, the proportion of effective nodes in the communication group, the proportion of tasks that have been recovered, or other indicators that can reflect the operating status of affected tasks.

[0144] Establish a stable operating state domain based on the minimum operability requirements of each affected task:

[0145]

[0146] when:

[0147]

[0148] When this occurs, it indicates that the affected task has been restored to a stable operating state that allows subsequent maintenance operations to be performed.

[0149] Therefore, in the candidate time interval set In this process, the minimum recovery interval is determined as follows:

[0150]

[0151] in, This is the preset coupling threshold.

[0152] As can be seen, this embodiment does not simply set fixed manual maintenance intervals such as 30 minutes or 1 hour, but automatically determines the earliest time when subsequent computing nodes are allowed to enter maintenance state based on two conditions: whether the impact of additional interruption coupling has decreased to an allowable range and whether the relevant running tasks have recovered to a stable running state.

[0153] S7. Determine the target maintenance plan and execute the maintenance.

[0154] After obtaining the maintenance coupling diagram and the minimum recovery interval corresponding to each strongly coupled edge, multiple candidate maintenance plans are generated based on the maintenance duration and maintenance time limit of the calculation node to be maintained.

[0155] Candidate maintenance plans can be represented as:

[0156]

[0157] in:

[0158]

[0159] Indicates the maintenance sequence of computing nodes;

[0160]

[0161] This indicates the start time of maintenance for each computing node.

[0162] Since a node may be in different operational phases when it enters maintenance at different times, the impact of a single node interruption can also vary with the start of maintenance, denoted as:

[0163]

[0164] The maintenance impact value of the candidate maintenance plan can be determined according to the following objective function:

[0165]

[0166] in, This represents the set of computational nodes to be maintained in the candidate maintenance plan whose maintenance impact cycles are adjacent or overlap; This represents the preset coupling effect coefficient.

[0167] When data centers place greater emphasis on avoiding task instability caused by continuous maintenance of associated nodes, they can appropriately increase... When maintenance time limits are strict, the corresponding coupling influence coefficient can be reduced according to the actual system conditions.

[0168] The candidate maintenance plan also needs to meet the following requirements: each computing node completes maintenance within the corresponding maintenance time limit; the time interval between strongly coupled nodes is not less than the corresponding minimum recovery interval; and the affected tasks meet the preset operational safety constraints during the maintenance process.

[0169] Select the maintenance plan with the smaller maintenance impact from the candidate maintenance plans that meet the conditions as the target maintenance plan.

[0170] Subsequently, the cluster management controller sends maintenance control commands to the corresponding computing nodes to be maintained through the node management interface, based on the maintenance sequence and maintenance start time in the target maintenance plan.

[0171] After receiving the maintenance control command, the node can sequentially perform operations such as prohibiting the scheduling of new tasks, processing the running status of existing tasks, removing the node from the current computing resource pool and entering the maintenance state.

[0172] The term "entering maintenance state" here may include powering down a node, restarting it, replacing hardware, upgrading equipment, or other maintenance operations that require temporarily stopping the node from providing computing resources. The specific method is not intended to limit the scope of this invention.

[0173] A specific example

[0174] For example, a data center has three computing nodes awaiting maintenance:

[0175]

[0176] Among them, running tasks Simultaneously occupy The minimum requirement is to retain 3 valid computing nodes; run the task. Occupy It allows the system to continue running after a compute node exits, but the running speed decreases.

[0177] Obtained through single-node analysis:

[0178]

[0179]

[0180]

[0181] If sorted solely by the impact value of a single node, the most likely choice would be:

[0182]

[0183] or:

[0184]

[0185] However, further analysis revealed that when After entering maintenance mode, the task The remaining 3 valid computing nodes are already in the minimum operational state.

[0186] If the node at this time If it enters maintenance again within a short period of time, then With only two valid computing nodes remaining, the minimum requirements for running the task are no longer met, causing the entire task to be paused or rolled back.

[0187] Assumption:

[0188]

[0189] and:

[0190]

[0191]

[0192] but:

[0193]

[0194] Explanation Node and nodes There is a significant additional disruption effect in this direction and time interval.

[0195] With nodes Maintenance completed and task completed The number of available nodes was restored to 4, and the directional interruption coupling gradually decreased.

[0196] Assuming that:

[0197]

[0198] There are still times:

[0199]

[0200] While:

[0201]

[0202] hour:

[0203]

[0204] and:

[0205]

[0206] Then it can be determined that:

[0207]

[0208] In other words, nodes Once in maintenance mode, at least 35 minutes must be waited before the node can be allowed to operate. It has entered maintenance mode.

[0209] Through the above methods, this embodiment can discover the "continuous maintenance amplification effect" that traditional single-node impact assessment cannot identify, and further transform this impact into a practically executable maintenance sequence and maintenance interval constraint.

[0210] Dynamically adjust implementation methods

[0211] Because the operational status of data center tasks is dynamic, the cluster management controller continuously or periodically acquires the task status during the execution of the maintenance plan.

[0212] For example, if a new task is scheduled to be performed on a computing node that has not yet been inspected during the original scheduled execution period, or if a running task is completed ahead of schedule, the node occupancy relationship in the original maintenance coupling diagram may have changed.

[0213] For nodes that have changed, only the set of affected tasks and directed connection edges associated with that node are redefined, and the corresponding directional interruption coupling amount and minimum recovery interval are recalculated, without having to recalculate the entire maintenance coupling graph.

[0214] After obtaining the updated maintenance coupling diagram, the maintenance tasks that have not yet been executed are reordered or the maintenance start time is adjusted.

[0215] Thus, this invention forms a complete technical closed loop of "computing resource status acquisition - task interruption sensitivity determination - directional joint impact analysis - directional interruption coupling identification - minimum recovery interval determination - maintenance plan generation - node maintenance control - operation status feedback update", which directly links computing power maintenance scheduling with the actual computing resources and operation task status of the data center, avoiding the problem of amplified associated task interruptions caused by maintenance based only on a fixed sequence or single node load.

[0216] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A data center computing power maintenance scheduling method based on task interruption sensitivity, characterized in that, Includes the following steps: S1. Obtain maintenance information, computing resource status, and task execution information of multiple computing nodes to be maintained in the data center; wherein, the maintenance information includes at least the maintenance duration and maintenance time limit, and the task execution information includes at least the node occupancy relationship between the task and the computing node, the task execution status, and the minimum runnable conditions of the task. S2. Determine the set of affected tasks corresponding to each computing node to be repaired based on the node occupancy relationship; for each computing node to be repaired, set the computing resources corresponding to the computing node to be repaired to an unavailable state, determine the corresponding task interruption sensitivity based on at least one of the technical state changes in computing rollback, state recovery, decrease in available computing resources and task execution delay caused by each affected task, and determine the single node interruption impact value of the computing node to be repaired based on the task interruption sensitivity of each affected task. S3. For two computing nodes to be inspected that have common affected tasks or task dependencies, one of the computing nodes to be inspected is designated as the preceding inspection node and the other as the following inspection node. For different candidate time intervals τ, if the task impact generated by the preceding inspection node has not yet been recovered, the computing resources corresponding to the following inspection node are set to an unavailable state to obtain the corresponding directional joint interruption impact value. S4. Calculate the directional interruption coupling amount based on the combined directional interruption impact value, the single-node interruption impact value of the preceding maintenance node, and the reference interruption impact value corresponding to the subsequent maintenance node entering maintenance state alone under the same time interval: in, Indicates the computing node to be inspected. First enter maintenance mode and wait for the time interval. Subsequent maintenance calculation nodes Directional interruption coupling amount when entering maintenance mode This indicates the corresponding directional joint interruption impact value. Indicates the computing node to be inspected. The impact value of a single-node interrupt. Indicates the computing node to be inspected. Under reference conditions where maintenance has not yet commenced, after a time interval Subsequent maintenance calculation nodes Reference interrupt impact value corresponding to entering maintenance state independently; S5. When the directional interruption coupling amount exceeds the preset coupling threshold, or when two computing nodes to be repaired enter the repair state in sequence and there are affected tasks that do not meet the minimum operable conditions, it is determined that there is a directional repair coupling relationship between the two computing nodes to be repaired; a repair coupling graph is constructed with the computing nodes to be repaired as nodes and the directional repair coupling relationship as directed connection edges. S6. For each directional maintenance coupling relationship in the maintenance coupling diagram, based on the recovery status of the operation status of the affected tasks after the preceding maintenance node enters the maintenance state, determine the minimum recovery interval that allows the subsequent maintenance node to enter the maintenance state. S7. Based on the maintenance coupling diagram, the maintenance duration and maintenance time limit of each computing node to be maintained, and the minimum recovery interval, determine the maintenance sequence and maintenance start time of each computing node to be maintained, and send maintenance control instructions to the node management interface of the corresponding computing node to be maintained according to the determined maintenance sequence and maintenance start time, so that the corresponding computing node to be maintained enters the maintenance state.

2. The data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 1, characterized in that, The node occupancy relationship is used to characterize one or more computing nodes actually occupied by the running task; the task dependency association includes at least one of the following: computing collaboration relationship, communication synchronization relationship, replication relationship or task execution dependency relationship between computing nodes corresponding to the same running task.

3. The data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 1, characterized in that, The determination of the task interruption sensitivity includes: Determine at least one of the following interruption impact indicators when the computing node to be repaired enters an unavailable state: computing rollback amount, state recovery data amount, decrease in available computing resources, and task execution delay amount. The interruption impact index is processed to unify the dimensions, and the task interruption sensitivity of the corresponding affected task is determined based on the processed interruption impact index. The single-node interruption impact value of the computing node to be repaired is determined based on the task interruption sensitivity of each affected task corresponding to the computing node to be repaired.

4. The data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 1, characterized in that, The obtained directional joint interruption impact value includes: Set the computing resources of the preceding maintenance node to an unavailable state, and determine the task running status of the corresponding affected task over time based on the remaining available computing resources; For candidate time interval Furthermore, the computing resources of subsequent maintenance nodes are set to an unavailable state, and the running tasks that are simultaneously affected by two computing nodes to be maintained, as well as the running tasks that are indirectly affected through task dependency associations, are identified. Based on at least one of the following: remaining available computing resources, communication connection status, replica availability, and task execution status, determine whether the corresponding running task meets the minimum runnable conditions, and determine the directional joint interruption impact value. .

5. The data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 1, characterized in that, When the directional interruption coupling amount is greater than the preset coupling threshold, or when at least one affected task does not meet the minimum operable conditions after the preceding and subsequent maintenance nodes enter the maintenance state in sequence, the corresponding directed connection edge is determined as a strong coupling edge, and maintenance interval constraints are set for the two computing nodes to be maintained corresponding to the strong coupling edge.

6. The data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 5, characterized in that, For the computing nodes to be inspected Point to the computing node to be inspected Strongly coupled edges, in the candidate time interval set The minimum recovery interval is determined according to the following formula: in, Indicates the computing node to be inspected. Entering maintenance status until the computing node is allowed to be maintained. Minimum recovery interval before entering maintenance status. This represents the preset coupling threshold. Indicates the time interval between candidates The runtime state vector of the subsequently affected task. This represents the stable operating state domain determined by the minimum operability conditions of the affected task.

7. A data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 6, characterized in that, Generate a maintenance sequence based on the maintenance coupling diagram. And the candidate maintenance plans at the start time of maintenance for each node to be maintained, and the maintenance impact value of the candidate maintenance plans is determined according to the following formula: in, This indicates the maintenance impact value corresponding to the candidate maintenance plan. This represents the set of maintenance start times for each computational node to be inspected. Indicates the calculation node to be inspected. At the start of maintenance The impact value of a single-node interrupt is determined by the corresponding task's running status. This represents the set of computational nodes to be maintained in the candidate maintenance plan whose maintenance impact cycles are adjacent or overlap. This indicates the preset coupling influence coefficient; Under the condition that the maintenance time limit of each computing node to be maintained and the time interval corresponding to the strongly coupled edge is not less than the corresponding minimum recovery interval are met, the target maintenance plan is determined according to the maintenance impact value.

8. The data center computing power maintenance scheduling method based on task interruption sensitivity according to claim 1, characterized in that, During maintenance, when the node occupancy relationship, task execution status, computing resource status, or maintenance status of the computing node to be maintained changes, the nodes and directed connection edges affected by the changes in the maintenance coupling graph are determined, the corresponding directional interruption coupling amount and minimum recovery interval are recalculated, and the maintenance sequence or maintenance start time of the computing nodes to be maintained that have not yet been executed is adjusted according to the updated maintenance coupling graph.

9. A data center computing power maintenance and scheduling system based on task interruption sensitivity, characterized in that, It includes a cluster management controller, a task status acquisition interface, and a node management interface that is connected to multiple computing nodes respectively. The cluster management controller is communicatively connected to the task status acquisition interface and the node management interface respectively. The task status acquisition interface is used to obtain the task running information of the currently running task and the computing resource status of each computing node; The cluster management controller is used to determine the task interruption sensitivity and single-node interruption impact value of each computing node to be repaired based on the node occupancy relationship between tasks and computing nodes. For computing nodes to be repaired that have common affected tasks or task dependencies, it determines the directional joint interruption impact value under different repair sequence relationships and time intervals. Based on the directional joint interruption impact value, it determines the directional interruption coupling amount and constructs a repair coupling graph. Based on the repair coupling graph, it determines the minimum recovery interval, repair sequence, and repair start time among the computing nodes to be repaired. The node management interface is used to receive maintenance control commands sent by the cluster management controller and control the corresponding computing node to be maintained to enter the maintenance state.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 8.

Images