A server resource elasticity collapse method based on computing power fragment inference

Abstract

The present application belongs to the technical field of computing power fragment inference, and provides a server resource elastic collapse method based on computing power fragment inference, which comprises: collecting resource use indexes and performance indexes of physical node running tasks in real time, judging whether the target task has performance degradation based on the performance indexes; if so, performing correlation analysis combined with the resource use indexes to identify whether the performance degradation is caused by resource fragmentation caused by cross-node communication. In the power consumption overload protection mode, the present application realizes accurate quantification of dependence tightness by constructing a directed graph of inter-container dependence and calculating the dependence distance of each auxiliary container to the main container, generates a peeling priority sequence according to the dependence distance, ensures that the resources of auxiliary containers with no dependence or weak dependence on the core task are released preferentially, adopts a dynamic step strategy to suspend auxiliary containers in batches, and expands the peeling intensity in batches with a Fibonacci sequence increment, thereby achieving an optimal balance between response speed and fine control.

Description

Technical Field

[0001] This invention belongs to the field of computing power fragmentation inference technology, specifically a server resource elastic collapse method based on computing power fragmentation inference. Background Technology

[0002] With the rapid development of artificial intelligence and large model training, high-performance heterogeneous computing resources such as GPUs and NPUs have become the most scarce core resources in data centers. To improve computing power utilization efficiency, containerized deployment and cluster scheduling technologies are widely adopted. By dynamically scheduling tasks such as training and inference to run on physical nodes, the goal is to maximize resource sharing. Due to the strong exclusivity of AI tasks in terms of computing power and memory and the coarse granularity of resource requests, there are a large number of computing power fragments in the cluster. These are scattered computing power resources that have been allocated on physical nodes but have not been effectively utilized. To eliminate fragmentation and improve utilization, dynamic scheduling and elastic collapse technologies based on resource fragmentation awareness have emerged. The core idea is to aggregate distributed subtasks scattered across multiple nodes to the same physical node or the same high-speed interconnect domain to compress resource gaps and shorten communication distances, thereby improving overall execution efficiency. However, the essence of elastic collapse is to forcibly squeeze tasks that were originally distributed across different physical nodes onto fewer nodes. This process inevitably leads to a sharp increase in the computing power, memory bandwidth, and power density of the target node. In a real data center environment, physical servers are limited by heat dissipation capacity and power budget. Once the power consumption of the accelerated computing unit exceeds the hardware safety threshold, it will trigger a forced frequency reduction protection mechanism in the driver layer or firmware layer. The mechanism will cause a cliff drop in the frequency of the computing core, which will significantly lengthen the training iteration time that should have been shortened after the collapse, and may even result in performance worse than before the collapse. If the power overload continues to worsen, it will also trigger systemic risks such as node overheating protection shutdown or hardware lifespan degradation. Therefore, this invention provides a server resource elastic collapse method based on computing power fragment inference. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0004] The technical solution adopted by this invention to solve its technical problem is: a server resource elastic collapse method based on computing power fragmentation inference, comprising the following steps: Real-time collection of resource usage and performance metrics of tasks running on physical nodes; and determination of performance degradation of target tasks based on performance metrics. If it exists, then perform correlation analysis with resource usage indicators to identify whether the performance degradation is caused by resource fragmentation due to cross-node communication. If so, mark the target task as a task to be collapsed and mark the physical computing node where it is located as a node to be processed. Based on the task to be collapsed, the real-time power consumption value of the node to be processed is obtained, and the power consumption level is evaluated in real time. The collapse action mode is divided based on the power consumption level. The collapse action includes power overload protection. If it is classified as power consumption protection, a selective resource stripping strategy based on task dependencies will be executed; The selective resource stripping strategy includes: hierarchically sorting auxiliary containers according to priority rules, pausing and releasing resources in batches using a dynamic step-size strategy, dynamically determining the termination time through real-time power consumption feedback during the stripping process, and introducing computational granularity degradation as a secondary protection measure. When degradation fails, it triggers collapse rollback and node health profile update.

[0005] The beneficial effects of this invention are as follows: This invention constructs a diagnostic chain from performance degradation perception to fragmentation root cause localization. Based on a two-dimensional comprehensive judgment of throughput deviation rate and progress change deviation rate, combined with a watch window mechanism, it reduces misjudgments caused by instantaneous disturbances. It obtains the effective working time and total running time of the computing core of each subtask through hardware performance counters, calculates the communication waiting ratio and extracts the maximum value as a feature value to determine whether performance degradation is caused by cross-node communication fragmentation. The two-level judgment mechanism ensures that subsequent collapse actions are only triggered when the target task is indeed impaired by cross-node communication fragmentation, effectively avoiding the interference of invalid collapse on cluster stability. At the same time, by independently establishing a power consumption history database for each physical node and calculating the power consumption quantiles of normal, attention, and alert levels, a dynamic balance between collapse intensity and hardware safety is achieved, reducing the problem that performance may actually decrease after collapse due to hardware forced frequency reduction triggered by power overload. In the power overload protection mode, this invention constructs a directed graph of dependencies between containers and calculates the dependency distance from each auxiliary container to the main container, achieving precise quantification of dependency tightness. Based on the dependency distance, a stripping priority sequence is generated to ensure the priority release of auxiliary container resources that have no or weak dependencies on the core task. A dynamic step-size strategy is used to pause auxiliary containers in batches, gradually increasing the stripping intensity with Fibonacci sequence increments, achieving an optimal balance between response speed and fine control. Simultaneously, a four-level progressive protection chain is constructed, consisting of auxiliary container stripping, computational granularity degradation, collapse rollback, and node health profile weighting. This progressive and comprehensive approach ensures corresponding handling measures are in place for any power overload scenario. By writing complete data from each overload protection process into the node health profile and applying differentiated penalties to node health scores based on trigger consequences, the scheduling system possesses self-learning and continuous optimization capabilities. This enables the system to continuously reduce the probability of power overload events during long-term operation and maintenance, extending hardware lifespan. Attached Figure Description

[0006] The invention will now be further described with reference to the accompanying drawings.

[0007] Figure 1 This is a flowchart of the steps in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the execution of step one in an embodiment of the present invention; Figure 3 This is a partial operation interface example diagram of step one in an embodiment of the present invention. Detailed Implementation

[0008] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0009] Example Please see Figures 1-3 As shown in the embodiment of the present invention, a server resource elastic collapse method based on computing power fragment inference includes the following steps: Step 1: Collect resource usage and performance metrics of tasks running on physical nodes in real time, and determine whether the target task has performance degradation based on the performance metrics; Step one involves the following process: The physical node is a server node equipped with at least one accelerated computing unit; Among them, the accelerated computing units include, but are not limited to: GPU (Graphics Processing Unit), NPU (Neural Processing Unit), or TPU (Tensor Processing Unit). By deploying monitoring agent processes on each physical node, resource usage and performance metrics of all task containers running on the physical node are continuously collected at a preset sampling period. Specifically, performance metrics include at least: throughput value and progress change rate; Throughput is the number of samples processed per second by the task; progress change rate is the rate of change of backlog depth in the queue of tasks to be processed in the inference task. A sliding time window is used to smooth the raw data. The length of the sliding time window is a preset N consecutive sampling periods. The process of determining whether the target task has performance degradation based on the preprocessed throughput value and the progress change rate is as follows: Collect throughput data from several complete iterations prior to the target task, and calculate the average throughput value as the first initial baseline. During the continuous operation of the target task, the real-time throughput value is subtracted from the first initial baseline, and the absolute value of the difference is taken as the throughput deviation value. The percentage of the throughput deviation value relative to the first initial baseline is calculated as the throughput deviation rate. The throughput deviation rate is compared with the preset throughput deviation rate limit. If the throughput deviation rate is greater than the throughput deviation rate limit, the throughput performance is determined to be degraded. If the throughput deviation rate is less than or equal to the throughput deviation rate limit, the throughput performance is determined to be undegraded. The throughput deviation rate limit is set based on the historical fluctuation range of the target task's throughput under normal load conditions. Specifically, throughput data of the task is collected during the most recent maintenance cycle, when there are no node anomalies and no resource contention. The coefficient of variation is calculated, and the coefficient of variation is multiplied by the preset fluctuation amplification coefficient as the basic limit. At the same time, it should be dynamically adjusted according to the task operation stage. For training tasks, when the loss function is in a rapid decline period, the allowable deviation rate limit is relaxed to a higher level to accommodate throughput fluctuations in the early stage of training. When the training enters the convergence period, the limit is tightened to improve the sensitivity to performance degradation. It should not be lower than the converted percentage corresponding to the minimum performance guarantee threshold declared by the user when the task is first submitted, and the maximum value of each constraint condition is taken as the throughput deviation rate limit. Collect progress change rate data over several complete iteration cycles prior to the target task, and calculate the average progress change rate as the second initial baseline; During the continuous operation of the target task, the difference between the real-time progress change rate and the second initial baseline is calculated, and the absolute value of the difference is taken as the progress change rate deviation value. The percentage of the progress change rate deviation value to the second initial baseline is calculated as the progress change deviation rate. The schedule deviation rate is compared with the preset schedule deviation rate limit. If the schedule deviation rate is greater than the schedule deviation rate limit, the schedule change performance is determined to be deteriorated. If the schedule deviation rate is less than or equal to the schedule deviation rate limit, the schedule change performance is determined to be not deteriorated. The setting of the progress change deviation rate limit is based on the following: It is determined according to the natural fluctuation range of the backlog depth of the target task under normal conditions; historical data on the backlog depth change rate of the inference service under steady-state traffic and when the cache queue is at a normal level are collected; the standard deviation is calculated; and a preset multiple of the standard deviation is used as the base limit. It is also correlated with the actual request rate of the current traffic inlet. When the inlet traffic experiences a step-like jump, the progress change deviation rate limit is synchronously amplified according to the traffic increase ratio to accommodate the backlog acceleration caused by a legitimate traffic surge, avoiding misjudging normal load changes as performance degradation. Furthermore, it is limited by the physical upper limit of the node's output capacity. If the backlog change rate exceeds the percentage corresponding to the difference between the node's maximum processing rate without frequency reduction and the current input rate, it indicates that the node is unable to digest the existing backlog. In this case, the limit should not be further relaxed, but a degradation judgment should be triggered. The progress change deviation rate limit is the minimum value among the natural fluctuation baseline value, the traffic dynamic adjustment value, and the node's physical output limit constraint value. If both throughput performance and schedule change performance are determined to be degraded, then the target task is marked as having performance degradation. If it is determined that the throughput performance has not deteriorated or the progress change performance has not deteriorated, then the target task is marked as having a suspected deterioration state, and observation will continue for a preset observation time window. During the observation period, continuously monitor and calculate the throughput deviation rate and the schedule change deviation rate. If the degradation judgment conditions are met simultaneously during the observation period, degradation is finally confirmed. If no simultaneous degradation occurs within the observation period, the suspected degradation marker will be removed, and normal monitoring will be restored. The preferred duration of the observation window is between 30 and 120 seconds.

[0010] Step 2: If it exists, perform correlation analysis with resource usage indicators to identify whether the performance degradation is caused by resource fragmentation due to cross-node communication. If so, mark the target task as a task to be collapsed and the physical computing node where it is located as a node to be processed. Step two involves the following: resource usage metrics, including at least: core computation time and total runtime. Obtain all subtasks running on each physical node for the target task, and for each subtask, collect the core computation time and total runtime during the period of performance degradation. The process of obtaining core computing time is as follows: read the hardware performance counter provided by the driver layer of the accelerated computing unit on the physical node where the subtask is located. The counter directly records the total number of clock cycles when the computing core is in an active state and actually executes effective computing instructions such as floating-point operations, matrix multiplication, or tensor operations. Multiply the total number of clock cycles by the known physical time of a single clock cycle to obtain the core computing time. The process of obtaining the total runtime is as follows: the monitoring agent process deployed on the physical node records the actual physical time span of the subtask during the performance degradation period, that is, the total time length from the start time to the end time of the period, which is used as the total runtime. The actual utilization rate of core computing is obtained by calculating the ratio of core computing time to total runtime. Subtract the actual utilization rate of the calculation core from the value of 1, and then subtract the baseline idle rate to get the communication waiting ratio; The baseline idle rate is the percentage of the minimum idle overhead that the accelerated computing unit will inevitably generate due to inherent microarchitectural reasons such as instruction pipeline bubbles, cache misses and branch prediction failures under the baseline working conditions of no communication waiting, no resource contention and normal operation. It is predetermined by the hardware microarchitectural design specifications of the accelerated computing unit and is usually between three percent and eight percent. Extract the maximum value of the communication waiting percentage among all subtasks and use it as the waiting percentage feature value of the target task; Among them, the waiting ratio feature value represents the waiting degree of the subtask most severely damaged by cross-node communication in the target task. The waiting degree of the most severe subtask is used to characterize the fragmentation severity of the entire target task. The reason for using the waiting ratio feature value for correlation judgment is that when the target task is fragmented due to cross-node communication, the essential damage is that the high-speed interconnect bus bandwidth of the physical node where some subtasks are located is competed and occupied by multiple subtasks, resulting in increased cross-node data transmission delay. The computing core of the subtask cannot perform effective calculations while waiting for data to arrive, and thus remains idle, which is reflected in the increase of the communication waiting ratio. The waiting percentage feature value is compared with the preset communication fragmentation judgment threshold; If the waiting percentage characteristic value is less than or equal to the communication fragmentation judgment threshold, it is determined that the performance degradation is not caused by cross-node communication fragmentation, the current processing flow ends, and the subsequent collapse step is not entered. If the waiting percentage characteristic value is greater than the communication fragmentation judgment threshold, the performance degradation is determined to be caused by resource fragmentation due to cross-node communication. The physical node where the subtask corresponding to the waiting percentage characteristic value is located is determined as the main node to be processed, and all physical nodes involved in the task to be collapsed are included in the set of nodes to be processed. The threshold for determining communication fragmentation is based on the historical distribution of communication waiting percentage under normal operating conditions of the cluster. Specifically, the communication waiting percentage data of all distributed tasks in the cluster during periods without performance degradation is collected, and the average value and standard deviation are calculated. The average value plus twice the standard deviation is used as the basic threshold.

[0011] By combining steps one and two, it was determined whether the current performance degradation of the target task was caused by resource fragmentation due to cross-node communication, and the bottleneck subtask with the most severe degradation and its physical node were identified. This ensures that subsequent resource elastic collapse is only triggered when the target task is indeed impaired due to cross-node communication fragmentation, thereby reducing unnecessary interference with cluster stability caused by ineffective collapse.

[0012] Step 3: Based on the task to be collapsed, obtain the real-time power consumption value of the node to be processed, and evaluate the power consumption level in real time. Based on the power consumption level, classify the collapse action mode. The collapse actions include: full-scale directional collapse, step-by-step restricted collapse, and power overload protection. Step 3 involves establishing an independent power consumption history database for each physical computing node in the cluster. The power consumption history database is maintained by using power consumption monitoring modules deployed on each physical node to continuously collect the total power consumption value of the node at a preset sampling period, and appending the collected power consumption sampling value to the power consumption history database of the corresponding node in the format of (node ​​identifier, timestamp, power consumption value). Historical power consumption data within a preset time period is retained, and historical data exceeding the time period is automatically discarded in order to control storage overhead and enable the historical distribution to reflect the recent (such as the last 72 hours or the last week) power consumption characteristics of nodes, thereby adaptively tracking the slow evolution of power consumption characteristics caused by seasonal changes, fluctuations in data center air conditioning conditions, or hardware aging. The historical power consumption data is cleaned and processed. The lower quartile and upper quartile of all sampled values ​​in the power consumption history database are calculated, and the difference between the two is calculated as the interquartile range. Sampled values ​​that are less than the lower quartile minus a preset multiple, and sampled values ​​that are greater than the upper quartile plus a preset multiple, are all marked as outliers and removed. The preferred value for the preset multiple is 1.5 times. After cleaning, the historical effective power consumption samples of the node to be processed are statistically sorted and arranged in ascending order. The power consumption values ​​corresponding to the following quantiles are then calculated: The power consumption value at the 50th percentile after sorting is determined as the normal power consumption quantile, which represents the typical power consumption level of the node under normal operating conditions. The power consumption value at the 75th percentile after sorting is determined as the attention power consumption quantile, which represents the power consumption level of the node under high load but still within the normal range. The power consumption value at the 90th percentile after sorting is determined as the warning power consumption quantile, which represents the high power consumption level that the node rarely reaches in its historical operation. The quantile calculation is performed in real time each time a power consumption grading assessment is required, or it is pre-calculated and cached by the system periodically at a preset update cycle to reduce the calculation latency during grading decisions. By comparing the real-time power consumption value with the attention power consumption quantile and the warning power consumption quantile, the power consumption level is divided into three levels: low, medium and high. These correspond to full-scale directional collapse, step-by-step restricted collapse and power overload protection, respectively. This achieves a fine dynamic balance between the collapse execution intensity and node hardware safety, effectively reducing the problem that the training iteration time after collapse increases instead of decreasing due to the hardware forced frequency reduction protection triggered by power overload. Real-time power consumption values ​​are obtained through power monitoring devices on the nodes to be processed. When the real-time power consumption value is less than or equal to the power consumption quantile, it is determined that the node to be processed is currently in a low-power state. Execute full-scale targeted collapse mode: Aggregate resources of multiple distributed subtasks contained in the task to be collapsed, and reschedule and bind multiple distributed subtasks to the same high-speed interconnect domain within the node to be processed without restarting the container in place, so as to minimize the communication distance between subtasks and eliminate the performance loss caused by cross-node communication fragmentation. When the real-time power consumption value is greater than the attention power consumption quantile but less than the warning power consumption quantile, it is determined that the node to be processed is currently in a medium power consumption state. Execute step-by-step restricted collapse mode: First, perform an initial collapse action with a lower aggregation degree than the full-scale directional collapse mode. That is, each time, only some subtasks of the task to be collapsed are migrated and aggregated, rather than all subtasks. After each migration action is completed, wait for a preset power consumption stabilization observation time window, reread the current real-time power consumption value, and re-compare it with the dynamic quantile to evaluate the change in power consumption degree. If the real-time power consumption value is detected to rise to the level equal to or higher than the warning power consumption quantile during the step-by-step execution, the current collapse action is immediately paused. If, after multiple step-by-step migrations, all subtasks of the task to be collapsed have been aggregated and the real-time power consumption value has never reached the warning power consumption quantile, the step-by-step restricted collapse is determined to be completed. When the real-time power consumption value is greater than or equal to the warning power consumption percentile, the node to be processed is determined to be in a high power consumption state, which is a power overload protection. Instead of performing any collapse and aggregation actions, a selective resource stripping strategy based on task dependencies is implemented. This strategy prioritizes releasing the resources occupied by non-core auxiliary containers to reduce node power consumption and create conditions for subsequent collapse.

[0013] Step 4: If the resource is classified as power consumption protection, then a selective resource stripping strategy based on task dependencies will be executed. The selective resource stripping strategy includes: hierarchically sorting auxiliary containers by priority rules, pausing and releasing resources in batches by adopting a dynamic step size strategy, dynamically determining the termination time through real-time power consumption feedback during the stripping process, and introducing computational granularity degradation as a secondary protection measure. When degradation fails, it triggers collapse rollback and node health profile update. Step four involves obtaining a list of all currently running containers on the node to be processed. The containers include the main container running the subtasks of the task to be collapsed and one or more auxiliary containers running auxiliary functions. The main container is a container instance that executes the core computational logic of the task to be collapsed, and is the main workload of the task to be collapsed on the node to be processed. Auxiliary containers are container instances deployed on the same physical node as the main container that provide auxiliary function support, including but not limited to data loading containers, monitoring and collection containers, log collection containers, and low-priority inference preheating containers; Establish a dependency graph for each container on the node to be processed; The dependency graph is a directed graph, including multiple nodes and directed edges, where each node corresponds to a container on the node to be processed, and the containers include the main container and all auxiliary containers. Directed edges represent the direction of a dependency relationship between two containers. The specific construction rules are as follows: When the first container sends data to the second container via a shared storage volume or network connection, and the second container receives and processes the data, it is assumed that the normal operation of the second container depends on the data input provided by the first container. A directed edge is established from the first container to the second container, indicating that the dependency direction is from the data provider to the data consumer. When a change in the state of the first container (including starting, stopping, restarting, or a failed health check) triggers the second container to perform the corresponding state change operation, it is considered that the state of the second container is controlled by the state of the first container. A directed edge is established from the first container to the second container, indicating that the dependency direction is from the state controller to the state controlled. After the dependency graph is constructed, starting from the main container and each auxiliary container, search for directed paths from the auxiliary container nodes to the main container nodes in the dependency graph. If there exists a path that starts from the auxiliary container node, passes through several intermediate nodes along the directed edge direction, and finally reaches the main container node, then the number of directed edges contained in the path is defined as the dependency distance from the auxiliary container to the main container. If, starting from a certain auxiliary container node, it is impossible to reach the main container node along any possible directed edge direction, then the dependency distance from the auxiliary container to the main container is positive infinity. The physical meaning of dependency distance is: the length of the dependency chain established between the auxiliary container and the task to be collapsed through data flow or control flow. A dependency distance of zero means that the auxiliary container is directly depended on by the task to be collapsed. The larger the dependency distance value, the longer the dependency chain between the auxiliary container and the task to be collapsed, and the more indirect transmission through intermediate containers, resulting in a lower dependency tightness. For example, suppose the following containers are running on the node to be processed: The main container runs the computational logic for the task to be collapsed; The data loading container periodically reads training samples from the shared storage volume and writes them to the message queue; The message queue container stores training samples to be consumed and provides a data consumption interface; The monitoring system reads the resource usage metrics of each container and reports them to the remote monitoring system. The log collection container reads the log files from each container and forwards them to the remote log storage. In this process, the data loading container writes data to the message queue container, and the message queue container provides a data consumption interface to the main container for the main container to read training samples. Therefore, there are directed edges in the dependency graph pointing from the data loading container to the message queue container and directed edges pointing from the message queue container to the main container. Starting from the data loading container, the main container can be reached by following the direction of the directed edges through the message queue container. The number of directed edges on the path is two, and the dependency distance of the data loading container is two. The monitoring and acquisition container only reads the indicator data of each container and reports it to the remote end. It does not write any data to the main container. The operation of the main container does not depend on any output of the monitoring and acquisition container. There is no directed path from the monitoring and acquisition container to the main container. The dependency distance is positive infinity. Log collection containers are similar to monitoring collection containers, only reading logs from each container without providing any input to the main container, and the dependency distance is also positive infinity; Based on the dependency distance of each auxiliary container, the stripping priority of each auxiliary container is determined as follows: The greater the dependency distance of the auxiliary container, the lower the dependency tightness with the main container, and the smaller the impact of stripping the container on the collapse task. Therefore, the stripping priority is higher. Auxiliary containers with a dependency distance of positive infinity have no dependency relationship with the main container. Stripping containers has no impact on the collapse task, so they have the highest stripping priority and are placed at the beginning of the stripping sequence. Auxiliary containers with a dependency distance of one have a direct dependency on the main container. Pausing the container will directly interrupt the main container's data input or state control. Therefore, they are marked as non-detachable and excluded from the detachment sequence. Auxiliary containers with a dependency distance value of two or higher have an indirect dependency relationship with the main container. The effect of pausing the container needs to be passed through one or more intermediate containers before reaching the main container. There is a certain buffer space, so they are marked as peelable and arranged in the peeling sequence in descending order of dependency distance value. When multiple auxiliary containers have the same dependency distance value, their relative priority is determined according to the amount of accelerated computing unit resources currently occupied by each auxiliary container in descending order. The larger the resource occupancy, the higher the priority of stripping. The final stripping priority sequence is generated according to the above rules, which serves as the basis for the execution order of batch stripping operations; A dynamic step-size strategy is adopted to pause auxiliary containers in batches and release the occupied acceleration computing resources. The specific implementation method of the dynamic step size strategy is as follows: the number of auxiliary containers to be paused in the first batch is determined by a preset initial step size. The initial step size is preferably 20% to 30% of the total number of currently strippable auxiliary containers. Select the corresponding number of auxiliary containers from high to low according to the stripping priority sequence, send a pause command to the auxiliary containers, switch the auxiliary containers to be run by the central processing unit of the node to be processed or put them into a suspended state, and reclaim the acceleration computing unit memory resources and computing resources originally occupied by the auxiliary containers. After each batch of auxiliary containers is paused, wait for a preset power consumption response time window. The power consumption response time window is the time required for the power consumption of the computing unit to stabilize at a new equilibrium state from the completion of the operation. It is preferably set to between 100 milliseconds and 500 milliseconds. After the waiting period ends, reread the real-time power consumption value of the node to be processed; If the real-time power consumption value is lower than the warning power consumption percentile, the stripping process is terminated, and the number and type of auxiliary containers stripped, as well as the power consumption reduction after each batch of stripping, are recorded as the node's stripping history data and stored in the aforementioned stripping history database. The total stripping ratio and total time required to reach the power safety state are written into the power recovery efficiency field of the node's health profile. If the real-time power consumption value is still not lower than the warning power consumption quantile, the step size is increased according to the preset step size increment strategy. The preferred step size increment strategy is the Fibonacci sequence increment, that is, the number of stripped in the second batch is increased by the sum of the number of the first two batches compared to the previous batch, and the pause operation of the next batch of auxiliary containers is continued. If there are no more auxiliary containers to be stripped in the stripping priority sequence, then a computational granularity downgrade operation is performed. The specific method for reducing computational granularity is as follows: send instructions to the computation engine of the task to be collapsed to dynamically reduce the computational batch size of the task in a single iteration. The batch size is the number of samples input to the accelerated computing unit for parallel processing in each iteration. Reducing the batch size can reduce the peak memory usage and computing unit load of operations such as matrix multiplication in a single iteration, thereby reducing instantaneous power consumption. The batch size is gradually reduced according to a preset degradation step size. After each reduction, a preset power consumption response time window is waited and the real-time power consumption value is reread. The degradation stops when the real-time power consumption value drops below the warning power consumption percentile, and the reduced batch size is recorded as the current task's adapted batch size on the node. If the batch size has been reduced to the preset minimum tolerable batch size limit during the continuous reduction process, and the real-time power consumption value is still not lower than the warning power consumption quantile, then it is determined that the current node cannot withstand any form of task aggregation at the physical level, and a collapse rollback operation is performed. Performing a collapse rollback operation includes: undoing all operations of this collapse action, restoring the task to be collapsed to its distributed running state before the judgment in step two, migrating and aggregating subtasks back to their original physical computing nodes, and redistributing the released resources to the original tasks. The data recorded in this power overload protection process are written into the node health profile of the global scheduler. The node health profile is a comprehensive health status record of each physical node stored in the scheduler. The specific data written includes: the trigger time when the physical computing node is marked as a high-power state, the real-time power consumption value and corresponding power consumption level at the time of triggering, the number and type of auxiliary containers stripped in this stripping process, the total stripping ratio required to reach the power safety threshold, whether the computing granularity degradation was triggered and the batch size after degradation, and whether the collapse rollback was triggered. The health score is updated based on the written data, where: Each time the power overload protection is triggered but the device recovers to normal after being stripped or downgraded, the health score of the physical computing node is reduced by a preset first penalty score. Each time the power overload protection is triggered and eventually the collapse rollback is triggered, the health score of the physical computing node is reduced by a preset second penalty score, which is greater than the first penalty score. When the health score of a physical computing node falls below the preset health pass threshold, the physical computing node is marked as a high-power sensitive node. In subsequent scheduling decisions, the node's priority for being selected as a collapse target node is reduced until the physical computing node does not trigger power overload protection again within a preset number of consecutive running cycles, and the health score gradually recovers.

[0014] This step generates a stripping priority sequence based on the dependency distance value. Dependencies with a positive infinity are stripped first, dependencies with two or more dependencies can be stripped, and dependencies with a one dependency cannot be stripped. This ensures that auxiliary container resources that have no or weak dependencies on the core task are always released first during the resource stripping process, thus fully guaranteeing the continuity and stability of the core computing logic of the task to be collapsed. A dynamic step-size strategy is adopted to pause auxiliary containers in batches. The initial step size is used to quickly respond to power overload, and the stripping intensity is gradually increased in batches when the power consumption does not meet the standard using Fibonacci sequence increments. This achieves the optimal balance between response speed and fine control. Meanwhile, a four-level progressive protection chain was constructed, consisting of auxiliary container stripping, computational granularity degradation, collapse rollback, and node health profile de-weighting: resources are released first by stripping dependent auxiliary containers; if this is ineffective, peak power consumption is reduced by decreasing the computation batch size; if this is still ineffective, the collapse action is revoked to restore the distributed state, and the node is marked as a high-power sensitive node to reduce its priority for subsequent selection as a collapse target. The four-level protection is progressive and provides a safety net at each level, ensuring that there are corresponding measures to deal with any power overload scenario, and fully guaranteeing cluster stability and hardware security. By writing complete data such as the trigger time, power level, stripping ratio, and whether degradation and rollback are triggered in each power overload protection process into the node health profile, and applying differentiated penalties to the node health score based on the triggering consequences, the penalty score for triggering collapse and rollback is higher than the score for recovering to normal after stripping or degradation, thus realizing continuous learning and quantitative evaluation of the hardware characteristics of each physical node. When a node's health score falls below the passing threshold, the node is downgraded to a collapse target node in subsequent scheduling decisions until its score gradually recovers after long-term stable operation. This gives the scheduling system self-learning and continuous optimization capabilities, enabling it to continuously reduce the probability of power overload events during long-term operation and maintenance, and extend the lifespan of hardware.

[0015] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for elastic collapse of server resources based on computing power fragmentation inference, characterized in that: Includes the following steps: Real-time collection of resource usage and performance metrics of tasks running on physical nodes; and determination of performance degradation of target tasks based on performance metrics. If it exists, then perform correlation analysis with resource usage indicators to identify whether the performance degradation is caused by resource fragmentation due to cross-node communication. If so, mark the target task as a task to be collapsed and mark the physical computing node where it is located as a node to be processed. Based on the task to be collapsed, the real-time power consumption value of the node to be processed is obtained, and the power consumption level is evaluated in real time. The collapse action mode is divided based on the power consumption level. The collapse action includes power overload protection. If it is classified as power consumption protection, a selective resource stripping strategy based on task dependencies will be executed; The selective resource stripping strategy includes: hierarchically sorting auxiliary containers according to priority rules, pausing and releasing resources in batches using a dynamic step-size strategy, dynamically determining the termination time through real-time power consumption feedback during the stripping process, and introducing computational granularity degradation as a secondary protection measure. When degradation fails, it triggers collapse rollback and node health profile update.

2. The server resource elastic collapse method based on computing power fragmentation inference according to claim 1, characterized in that: The process of determining whether the target task has performance degradation is as follows: Performance metrics include throughput and progress rate; The throughput value is the number of samples processed per second by the task, and the progress change rate is the rate of change of the backlog depth of the queue to be processed in the inference task. Based on the preprocessed throughput value and the progress change rate value, determine whether the throughput and progress change have degraded in performance. If the throughput performance is determined to be degraded and the progress change performance is determined to be degraded, then mark the target task as having degraded performance.

3. The server resource elastic collapse method based on computing power fragmentation inference according to claim 2, characterized in that: The process for determining whether changes in throughput and schedule indicate performance degradation is as follows: The throughput value acquired in real time is subtracted from the first initial baseline and the absolute value is taken as the throughput deviation value. The percentage of the throughput deviation value relative to the first initial baseline is calculated as the throughput deviation rate. If the throughput deviation rate is greater than the throughput deviation rate limit, then the throughput performance is determined to be degraded. The difference between the real-time acquired progress change rate and the second initial baseline is taken as the absolute value of the progress change rate deviation value. The percentage of the progress change rate deviation value relative to the second initial baseline is calculated as the progress change deviation rate. If the schedule change deviation rate is greater than the schedule change deviation rate limit, the schedule change performance is determined to be degraded.

4. The server resource elastic collapse method based on computing power fragmentation inference according to claim 1, characterized in that: The process of identifying whether performance degradation is caused by resource fragmentation due to cross-node communication is as follows: By analyzing resource usage indicators, the communication waiting ratio is calculated, and the maximum value of the communication waiting ratio in all subtasks is extracted as the waiting ratio feature value of the target task. If the waiting percentage characteristic value is greater than the communication fragmentation judgment threshold, then the performance degradation is determined to be caused by resource fragmentation due to cross-node communication.

5. The server resource elastic collapse method based on computing power fragmentation inference according to claim 4, characterized in that: The process for obtaining the communication waiting percentage is as follows: Resource usage metrics include: core computing time and total runtime. Get all subtasks running on each physical node of the target task, read the total number of clock cycles that are active and actually execute valid computation instructions on the physical node where the subtask is located, and multiply the total number of clock cycles by the known physical time of a single clock cycle to obtain the core computation time. The total running time is the total duration of the time interval from the start time to the end time. The actual utilization rate of core computing is obtained by calculating the ratio of core computing time to total runtime. Subtract the actual utilization rate of the calculation core from the value of 1, and then subtract the baseline idle rate to get the communication waiting ratio.

6. The server resource elastic collapse method based on computing power fragmentation inference according to claim 1, characterized in that: The process of classifying collapse action modes based on power consumption is as follows: Statistical sorting is performed on the historical sampled values ​​of the effective power consumption of the node to be processed after cleaning. The power consumption value at the 90th percentile after sorting is determined as the warning power consumption quantile. Real-time power consumption values ​​are obtained through power monitoring devices on the nodes to be processed. When the real-time power consumption value is greater than or equal to the warning power consumption quantile, the node to be processed is determined to be in a high power consumption state, which is a power overload protection.

7. The server resource elastic collapse method based on computing power fragmentation inference according to claim 1, characterized in that: The process of prioritizing auxiliary containers by performing hierarchical sorting is as follows: Obtain a list of all currently running containers on the node to be processed, and establish a dependency graph for each container on the node to be processed. The dependency graph is a directed graph with each container as a node and the data flow dependency and control flow dependency between containers as edges. Starting from the main container and each auxiliary container, search the dependency graph for directed paths from the auxiliary container node to the main container node. The number of directed edges on the path is defined as the dependency distance from the auxiliary container to the main container. If there is no path, the dependency distance is positive infinity. The stripping priority is determined based on the dependency distance value of each auxiliary container: the larger the dependency distance, the higher the stripping priority; The auxiliary container with a dependency distance value of positive infinity has the highest priority and is placed at the beginning of the stripping sequence; Auxiliary containers with a dependency distance value of one are marked as non-strippable and excluded from the stripping sequence; Auxiliary containers with a dependency distance value of two or higher are marked as peelable, and are sorted in descending order of dependency distance value; When multiple auxiliary containers have the same dependency distance value, their relative priority is determined in descending order of the amount of accelerated computing unit resources currently occupied by each auxiliary container. Generate the final stripping priority sequence, which will serve as the basis for the execution order of batch stripping operations.

8. The server resource elastic collapse method based on computing power fragmentation inference according to claim 7, characterized in that: The dependency distance is the number of directed edges contained in the directed path from the auxiliary container node, passing through several intermediate nodes along the directed edge direction, and finally reaching the main container node. The length of the dependency chain established between the auxiliary container and the task to be collapsed through data flow or control flow. A dependency distance of one indicates that there is a direct dependency between the auxiliary container and the main container. A dependency distance of two or more indicates that there is an indirect dependency between the two through one or more intermediate containers. A dependency distance of positive infinity indicates that there is no dependency between the two.

9. The server resource elastic collapse method based on computing power fragmentation inference according to claim 1, characterized in that: The process of the dynamic step size strategy is as follows: The number of auxiliary containers to be paused in the first batch is determined by a preset initial step size; Select the corresponding number of auxiliary containers in descending order of stripping priority, and send a pause command to the selected auxiliary containers; After each batch of auxiliary containers is paused, wait for a preset power consumption response time window, and then reread the real-time power consumption value of the node to be processed after the wait is over. If the real-time power consumption value is lower than the warning power consumption percentile, the stripping process is terminated. If the real-time power consumption value is still not lower than the warning power consumption percentile, the step size is increased according to the preset step size increment strategy, that is, the number of stripped in the second batch is increased by the sum of the number of the first two batches compared to the previous batch, and the pause operation of the next batch of auxiliary containers continues.

10. A server resource elastic collapse method based on computing power fragmentation inference according to claim 1, characterized in that: The calculation process for granularity degradation is as follows: Send instructions to the computing engine of the task to be collapsed to dynamically reduce the computing batch size of the task in a single iteration. The computing batch size is the number of samples input to the accelerated computing unit for parallel processing in each iteration. The batch size is gradually reduced according to the preset degradation step size. After each reduction, the power consumption response time window is waited and the real-time power consumption value is reread until the real-time power consumption value drops below the warning power consumption percentile and then the degradation stops. If the batch size has decreased to the preset minimum tolerable batch size limit during the continuous reduction process, but the real-time power consumption value is still not lower than the warning power consumption quantile, then a collapse rollback operation is performed.

Images