Intelligent scheduling optimization system adaptive to distributed computing power resources

Abstract

The application discloses an intelligent scheduling optimization system suitable for distributed computing power resources, relates to the field of computing power resource scheduling optimization, and mainly aims at the technical problems that a single linear progress model of a traditional scheduler cannot adapt to nonlinear execution characteristics of iterative convergence and data skew, causes serious underestimation of residual time, identification failure of long-tail stagnation tasks, and migration decision distortion, specifically through a closed-loop feedback type scheduling enhancer, accurately mines nonlinear execution characteristics of tasks, dynamically adapts a residual time prediction model, completely solves the misjudgment problem of long-tail and stagnation tasks, greatly improves the accuracy of task migration decisions, reduces invalid scheduling overhead, activates inefficiently occupied computing power resources of a cluster, and further through a double-loop scheduling controller, multi-dimensionally quantifies node dynamic load characteristics, adaptively adjusts an overload determination threshold, and relies on inertia lag and prediction compensation to realize load smooth scheduling, and avoids scheduling shock and mismigration problems caused by instantaneous load disturbance.

Description

Technical Field

[0001] This invention relates to the field of computing power resource scheduling and optimization technology, specifically to an intelligent scheduling and optimization system adapted to distributed computing power resources. Background Technology

[0002] Current distributed computing platforms are widely used in core business scenarios such as big data processing, real-time stream computing, artificial intelligence iterative training, and high-frequency data processing. Mainstream scheduling frameworks include Hadoop YARN, Spark native scheduler, Flink dynamic scheduler, and Kubernetes resource scheduler, forming a basic scheduling model with unified supply of computing resources from a pool and centralized decision-making by a central scheduler.

[0003] The core operating logic of a traditional distributed computing power scheduling system is that the resource pool uniformly collects the computing power resources of each data node, and the resource scheduler monitors the node load and task execution status in real time. Based on a fixed load threshold and a linear progress prediction model, it completes scheduling decisions such as task migration, resource allocation, and node online / offline determination.

[0004] In normal and stable batch computing scenarios, traditional scheduling modes can meet the basic resource scheduling requirements. However, as distributed business scenarios become more complex, complex scenarios such as iterative convergence computing, data-skewed Reduce computing, I / O-intensive tasks with checkpoints, real-time stream processing of microservices, and low-latency computing for high-frequency transactions become the norm, and the shortcomings of traditional scheduling technologies are fully exposed.

[0005] On the one hand, traditional schedulers assume that task execution progress and time have a constant linear relationship. They rely on extrapolating the remaining execution time at a fixed rate, which cannot adapt to the nonlinear execution characteristics that are common in actual business, such as step-by-step stagnation, slow tail, and exponential deceleration. This can easily lead to a serious underestimation of the remaining task time, resulting in problems such as invalid migration, missed migration, and incorrect migration timing.

[0006] On the other hand, traditional load scheduling uses a fixed-period sampling, sliding window mean filtering and static high and low water level thresholds to determine whether a node is overloaded or idle. It relies solely on the average CPU utilization to determine the node's overload and idle state, and cannot identify dynamic load characteristics such as sub-millisecond instantaneous load spikes and high-frequency load oscillations. This results in problems such as load judgment lag, short-term disturbances triggering scheduling falsely, overly conservative allocation of steady-state loads, and repeated swings in scheduling state.

[0007] To address the aforementioned technical shortcomings, an intelligent scheduling and optimization system adapted to distributed computing resources is proposed. This system aims to construct a closed-loop intelligent scheduling system with accurate task-level prediction and dynamic node-level control by deploying dual optimization units in a bypass manner, while maintaining full compatibility with the original distributed scheduling architecture and without intrusive modifications. Summary of the Invention

[0008] The purpose of this invention is to solve the problems mentioned above by proposing an intelligent scheduling and optimization system adapted to distributed computing resources.

[0009] The objective of this invention can be achieved through the following technical solution: an intelligent scheduling and optimization system adapted to distributed computing resources, wherein the system is configured with a computing resource pool and a resource scheduler, and further includes:

[0010] A closed-loop feedback scheduling enhancer is deployed in the bypass of the original scheduler; it dynamically modulates the input parameters of the original scheduler, namely the predicted remaining time, and the output action, namely whether migration is allowed; it mines the progress time series of each task in real time, extracts the local progress decay coefficient and progress pattern fingerprint; and sets the sampling time t.

[0011] The progress time series of each task is mined in real time. Based on the completion progress p(t) and sampling interval ΔT, the first-order difference v(t) and the second-order difference a(t) are calculated, and the attenuation coefficient α is also calculated. t = |a(t)| / [v(t)+ε], where ε is a very small positive number;

[0012] The (α) of the most recent L sampling points t ,v t For a progression pattern fingerprint Φ mapped as an extremum pair, the extremum pair includes the maximum decay coefficient αmax and the corresponding velocity v at that moment. α The maximum attenuation coefficient and its duration τ; identify and perform nonlinear predictions;

[0013] A dual-loop scheduling controller with inertial hysteresis and predictive compensation inserts a dynamic admission filter before the original scheduler's migration decision, which senses the rate of load change in real time and calculates the sample standard deviation σ of the load values ​​within the window. L The weighted average of the range is used as the intensity; the current load L(t), the rate of change r(t), and the intensity are combined to form the trend intensity index S; based on the comparison results of the trend intensity index S with the first-level index threshold and the second-level index threshold, as well as the comparison results of the load fluctuation intensity with the intensity threshold, the dynamic high water level is output, and inertial hysteresis compensation analysis and predictive compensation analysis are performed.

[0014] Furthermore, the nonlinear prediction process is as follows:

[0015] Determine the current progress p(t), current rate v(t), and decay coefficient α. t αmax and v in pattern fingerprint α max;

[0016] If α t If the coefficient is less than the set threshold and αmax is lower than the maximum threshold, then the task is inferred to be in a stable execution phase, and the output is T.nonlin =T lin Specifically, non-linear time is equivalent to linear time; that is, execution follows the decisions of the original scheduler.

[0017] Furthermore, if α t If the rate exceeds the set coefficient threshold and the maximum coefficient threshold, and continues to rise, the system enters a long-tail warning mode. At this point, a constant rate is no longer assumed; instead, the remaining progress is assumed to follow a logarithmic decay model.

[0018] time left , among them β is the empirical scaling factor;

[0019] If τ (maximum decay duration) exceeds 3 sampling periods, and the current v(t) < 0.1 × v α max: If the condition is determined to be depth stagnation, set T directly. nonlin Set a preset upper limit value and mark the task as a non-transferable candidate.

[0020] Furthermore, when the original scheduler executes the migration decision, if the difference between the nonlinear time of the current task and the linear time of the original path continues to increase and the maximum attenuation coefficient still shows an increasing trend, the closed-loop feedback scheduling enhancer intervenes and makes a migration decision based on the real-time progress of the task and the remaining nonlinear time; if the current task recovers to a linear relationship, the original scheduler continues to execute the migration decision after the migration task is completed by the closed-loop feedback scheduling enhancer.

[0021] Furthermore, the intensity of load fluctuations is collected, and the weighted mean of the sample standard deviation σL and the range (maximum value - minimum value) of the load values ​​within the window is calculated: Intensity = 0.6 × σL L +0.4×(max(L)-min(L)).

[0022] Furthermore, the trend strength index S:

[0023] ;

[0024] Where: L base L is the security baseline defined for the system. max r is the danger threshold. ref This is the reference rate used for normalization.

[0025] Furthermore, when the trend intensity index exceeds the first-level index threshold and the load fluctuation intensity is less than the intensity threshold, the system is in a steady state, and a strict threshold is adopted; dynamic high water level = high water level base value − offset;

[0026] When the trend strength index exceeds the first-level index threshold but does not exceed the second-level index threshold, the threshold is increased linearly according to the trend strength. The formula is: Dynamic high water level = high water level base value + 5% × (S - 0.3) / 0.4.

[0027] When the trend strength index exceeds the secondary index threshold, the dynamic high water level will rise further, but it is constrained by the recent peak: dynamic high water level = min(90%, max(recent peak + 2%, 85%)).

[0028] Output dynamic high water level.

[0029] Furthermore, when the dynamic high water level exceeds the set high water level threshold, predictive compensation is performed.

[0030] When the load change rate fluctuates continuously and the real-time load repeatedly swings between overload and non-overload, inertial hysteresis compensation is performed.

[0031] Furthermore, the predictive compensation is performed as follows: if the load does not exceed the threshold, and the trend strength index S rises for two consecutive sampling points and exceeds 0.6, micro-migration is performed in advance; the inertial hysteresis compensation is performed as follows: a time hysteresis loop is introduced: after a node enters the overload state, it must satisfy the condition that the load is continuously lower than the dynamic high water level minus a hysteresis width and continues to do so, then it exits the overload state.

[0032] Furthermore, when the load change rate r(t) exceeds the rate threshold, it is determined that the load at the current sampling point is increasing, and the dual-loop scheduling controller intervenes; when the load change rate r(t) does not exceed the rate threshold, it is determined that the load at the current sampling point is decreasing, and the dual-loop scheduling controller does not intervene.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] 1. This invention retains the core architecture and decision-making logic of the original computing power resource pool and basic resource scheduler without requiring disruptive modifications to the underlying distributed cluster architecture and native scheduling code. It is highly compatible with mainstream distributed computing frameworks such as Spark, Flink, Hadoop, and Kubernetes, and has strong versatility and portability.

[0035] The computing resource pool continues to undertake the basic functions of collecting and uniformly supplying computing resources across all nodes, ensuring the overall scheduling of cluster computing resources. The basic resource scheduler retains the core capabilities of native task migration and resource allocation, ensuring the basic stability of cluster scheduling. By adding a bypass optimization unit, the original scheduling input parameters and output actions are dynamically modulated. Without disrupting the original scheduling system, the solidification defects of the basic scheduling logic are made up for, realizing a smooth upgrade from traditional static scheduling to intelligent dynamic scheduling. This significantly reduces system transformation costs and deployment risks, and ensures seamless iterative upgrades of cluster services.

[0036] 2. It adopts closed-loop feedback modulation logic, which does not replace the original scheduling decision, but only dynamically corrects the input and output parameters. When the task returns to a linear and stable execution state, it automatically exits the optimized modulation mode and hands over the original scheduler to execute the decision. This effectively avoids scheduling logic conflicts and prevents data migration turbulence and decision delays.

[0037] By accurately marking deeply stalled tasks as non-migratable candidates, we avoid wasting network bandwidth and computing resources on invalid migration operations, significantly improve the rationality and accuracy of task migration decisions, effectively solve the problems of long-tail stuttering and node load accumulation in iterative computing, data skew, and I / O intensive tasks, and improve the overall task execution efficiency of the cluster.

[0038] 3. Based on the dual-loop control logic of inertial hysteresis compensation and predictive compensation, the node overload high water level threshold is dynamically and adaptively adjusted. Different threshold strategies are adapted for three types of load states: steady state, attention zone, and warning zone. Strict thresholds are adopted for steady state scenarios to ensure resource utilization. For scenarios with rapid load increase, instantaneous fluctuations are tolerated to avoid false migration. For high load scenarios, the threshold is relaxed to prevent load deterioration.

[0039] Meanwhile, the time lag mechanism solves the problem of repeated jumps in node scheduling status, and the advanced micro-migration mechanism avoids the risk of load backlog, achieving a smooth soft transition in node load scheduling. This effectively reduces problems such as scheduling misjudgment, invalid migration, and scheduling oscillation caused by instantaneous load spikes and high-frequency load fluctuations, significantly reducing cluster scheduling overhead and improving the stability and accuracy of distributed node load management. Attached Figure Description

[0040] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0041] Figure 1 This is a system principle block diagram of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0044] Please see Figure 1 As shown, the intelligent scheduling and optimization system adapted to distributed computing resources sets up a computing resource pool and a resource scheduler. The computing resource pool is used to supply computing resources to each data node, and the resource scheduler performs resource migration scheduling.

[0045] The intelligent scheduling optimization system constructs a closed-loop feedback scheduling enhancer, which is deployed in the bypass of the original scheduler. It does not replace the original migration decision logic, but dynamically modulates its input parameters (remaining time prediction value) and output actions (whether migration is allowed).

[0046] Scenario setting: In distributed computing platforms (such as Spark, Flink, Hadoop), the dynamic resource scheduler continuously monitors the task execution status of each node. When it detects that a node has uneven load or that the expected remaining time of a task is long, it will evaluate the benefits of migrating the task to other idle nodes. This evaluation depends on the prediction of the remaining execution time of the task.

[0047] Traditional schedulers assume a linear relationship between task execution progress and time, i.e., "percentage completed / time used = constant rate", and extrapolate the remaining time using this rate. However, in iterative convergence calculations (such as PageRank and LDA), the Reduce phase under data skew, and I / O-intensive tasks with periodic checkpoints, the actual progress exhibits a non-linear pattern of step-like stagnation or extremely slow tail. Linear extrapolation will severely underestimate the remaining time (e.g., misestimating a task with 10 minutes of actual remaining time as 1 minute).

[0048] The operation process of the closed-loop feedback scheduling enhancer is as follows:

[0049] Real-time mining of the progress time series of each task, extraction of "local progress decay coefficient" and "progress pattern fingerprint", and setting the sampling time t;

[0050] Local progression attenuation coefficient (denoted as α) t This feature is not a simple rate of change, but rather the ratio of the second difference to the first difference, used to distinguish between "naturally slow" and "about to stop".

[0051] Let the progress of the task at time t be p(t)∈[0,1], and the sampling interval be ΔT;

[0052] First-order difference: v(t) = [p(t) − p(tT)] / ΔT;

[0053] Second-order difference: a(t) = [v(t) − v(t - ΔT)] / ΔT;

[0054] The attenuation coefficient is then: α t =∣a(t)∣ / [v(t)+ε], where ε is a very small positive number to prevent division by zero.

[0055] Physical meaning: α t Reflecting the rate of decline relative to its current value, when α t A value consistently greater than 1.0 indicates that the rate is decreasing at a rate exceeding 100% of its own value (i.e., exponential decay), a typical sign of long-tailed tasks. In contrast, the alpha of normal tasks... t It fluctuates around 0, and occasional negative values ​​(acceleration) are temporary.

[0056] The progress pattern fingerprint is denoted as Φ;

[0057] The (α) of the most recent L sampling points t ,v t For each element mapped to a two-dimensional tuple, instead of averaging, we take the extreme value pairs: the maximum decay coefficient αmax and the corresponding velocity v at that moment. α max, and the duration τ of the maximum attenuation coefficient.

[0058] Physical meaning: Extremal pairs describe the "most drastic deceleration event" experienced by a task. A characteristic of a long-tailed task is that αmax is very large (e.g., > 2.0), and v α The maximum value has been reduced to less than 10% of the initial rate, and the τ has exceeded two sampling periods. This fingerprint does not depend on the global distribution and is effective independently for each task.

[0059] The original scheduler performs scheduling based on a linear relationship, while the closed-loop feedback scheduling enhancer defaults to nonlinear prediction. This predictor does not use a fixed formula, but dynamically selects the prediction model based on the attenuation coefficient.

[0060] Determine the current progress p(t), current rate v(t), and decay coefficient α. t αmax and v in pattern fingerprint α max;

[0061] If α t If the coefficient is less than the set threshold and αmax is lower than the maximum threshold, then the task is inferred to be in a stable execution phase, and the output is T. nonlin =T lin Specifically, non-linear time is equivalent to linear time, meaning that the execution follows the original scheduler's decision.

[0062] If α t If the rate exceeds the set coefficient threshold and the maximum coefficient threshold, and continues to rise, it enters the "long-tail warning mode." At this point, a constant rate is no longer assumed; instead, the remaining progress is assumed to follow a logarithmic decay model.

[0063] time left , among them β is the empirical scaling factor. The physical meaning of this model is that the rate decreases exponentially as the proportion of completion increases, and the more severe the long tail (the larger αmax is), the faster the decay.

[0064] If τ (maximum decay duration) exceeds 3 sampling periods, and the current v(t) < 0.1 × v α max: If determined to be "deep stagnation", directly set T. nonlin Set a preset upper limit (e.g., 30 minutes) and mark the task as a "non-transferable candidate";

[0065] Output non-linear time;

[0066] If the difference between the nonlinear time of the current task and the linear time of the original path continues to increase, and the maximum attenuation coefficient still shows an increasing trend, the closed-loop feedback scheduling enhancer will intervene and make a migration decision based on the real-time progress of the task and the remaining nonlinear time to avoid migration lag. If the current task recovers to a linear relationship, the original scheduler will continue to make migration decisions after the migration task is completed by the closed-loop feedback scheduling enhancer to avoid data migration turbulence and inability to quickly determine the migration decision.

[0067] The intelligent scheduling optimization system constructs a dual-loop scheduling controller with inertial hysteresis and predictive compensation, which is deployed in the scheduling agent of each node. It does not directly replace the original migration decision, but inserts a dynamic admission filter before the original decision. By sensing the rate of change and fluctuation intensity of the load in real time, it dynamically adjusts the "overload judgment condition" of the node, performs inertial hysteresis compensation analysis and predictive compensation analysis, and achieves a smooth soft transition.

[0068] Existing distributed schedulers (such as Kubernetes and Hadoop YARN) typically use fixed-period load sampling to calculate the sliding window average of node CPU utilization (such as the average over the past minute) and compare it with preset static upper and lower thresholds (such as 80% and 30%).

[0069] When the mean is higher than the high watermark threshold, the node is marked as "overloaded" and tasks are migrated out. When it is lower than the low watermark threshold, it is marked as "idle" and accepts more tasks. In scenarios such as microservice call chains, real-time stream processing (Flink backpressure propagation), and high-frequency trading, the load often experiences sub-millisecond instantaneous spikes (e.g., caused by JVM global GC, lock release contention, or network micro-bursts, with the CPU surging to 100% within 20ms). The conventional sliding window mean will smooth these spikes into a "mild increase" (e.g., from 40% to 55%). The scheduler believes that the node still has spare capacity and continues to allocate tasks. However, during the very short period of time that the spike lasts, the task queue has already accumulated. When the window mean finally exceeds the high watermark threshold (usually after the spike disappears), the scheduler triggers migration. At this time, the node load has returned to normal, and migration introduces unnecessary overhead.

[0070] Instead of sampling CPU utilization, we calculate the following three process quantities, with the sampling period fixed as the sensing interval.

[0071] The load change rate r(t) is obtained and defined as: ;

[0072] L(t) represents the CPU utilization rate (percentage) of the current sampling point, and Δt = sensing interval.

[0073] Physical meaning: r(t) > 0 indicates that the load is increasing, the larger the positive value, the steeper the increase; r(t) < 0 indicates that the load is decreasing.

[0074] Engineering value: The rate of rise of a momentary spike can reach over 200% per second, while the rate of rise of a sustained overload is usually below 5% per second. This characteristic indicates that "an impending impact" is a prediction rather than an a posteriori assessment.

[0075] If the rate of change of load exceeds the rate threshold, it is determined that the load at the current sampling point is increasing; conversely, if the rate of change of load does not exceed the rate threshold, it is determined that the load at the current sampling point is decreasing.

[0076] When the trend is downward, the dual-loop scheduling controller does not intervene; when the trend is upward, the dual-loop scheduling controller intervenes.

[0077] The intensity of load fluctuations is collected, defined as follows: a short window (intensity window length, default 10 sampling points, i.e., 2 seconds) is maintained, and the sample standard deviation σ of the load values ​​within the window is calculated.L The weighted mean of the range (maximum value - minimum value): Intensity = 0.6 × σ L +0.4×(max(L)-min(L));

[0078] Physical meaning: The intensity of the quantitative load is the "duration of jitter". The intensity is low for steady-state tasks (<3%), while the intensity is high for periodic batch processing or spike-like disturbances (>15%).

[0079] Engineering value: During high-intensity events, the dispatcher should not "overreact" but should proactively increase inertia (i.e., delay response) to avoid oscillations;

[0080] The overload trend intensity is collected and defined as follows: a dimensionless trend intensity index S is formed by combining the current load L(t), the rate of change r(t), and the intensity.

[0081] ;

[0082] Where: L base For the system's defined safety baseline (e.g., 60%), L max For the danger threshold (e.g., 90%), r ref Used as a reference rate (e.g., 30% / second) for normalization;

[0083] The 1+intensity in the denominator causes high intensity to suppress trend strength and inhibit false triggering.

[0084] Physical meaning: S∈[0,2), when S>0.8 indicates that overload may soon occur. This index combines “current load position”, “rate of change” and “duration”, and is more robust than any single index.

[0085] Inertial hysteresis compensation analysis and predictive compensation analysis are performed on the intensity of load fluctuation and the strength of overload trend.

[0086] Define a high water level threshold. When the trend intensity index exceeds the first-level index threshold and the load fluctuation intensity is less than the intensity threshold, the system is in a steady state. A strict threshold is adopted. Dynamic high water level = high water level base value - offset.

[0087] When the trend strength index exceeds the first-level index threshold but does not exceed the second-level index threshold, the system is in the "attention zone". At this time, the threshold is increased linearly according to the trend strength. The formula is: Dynamic high water level = high water level base value + 5% × (S-0.3) / 0.4. The physical meaning is: when the load rises rapidly, it tolerates higher instantaneous values ​​to avoid false triggering of migration due to short-term spikes.

[0088] When the trend strength index exceeds the secondary index threshold, the system enters the "warning zone." At this point, the dynamic high water level rises further, but is constrained by recent peak values.

[0089] Dynamic high water level = min(90%, max(recent peak value + 2%, 85%)), physical meaning: if the load has already reached a high level, it means that a spike has occurred. At this time, the threshold should not be lowered, but the tolerance should be relaxed to prevent "adding insult to injury" migration.

[0090] Output dynamic high water level (floating-point number, percentage) for subsequent determination of whether the node is overloaded;

[0091] When the dynamic high water level exceeds the set high water level threshold, it is inferred that the node is overloaded and predictive compensation is performed. Instead of waiting for the load to exceed the threshold, micro-migration is performed in advance when the trend strength index S rises for two consecutive sampling points and exceeds 0.6.

[0092] When the load change rate fluctuates continuously and the real-time load repeatedly swings between overload and non-overload, inertial hysteresis compensation is performed, introducing a time hysteresis loop: after a node enters the "overload state", it must satisfy the condition that the load is continuously lower than the dynamic high water level minus a hysteresis width for a certain period of time before it can exit the overload state.

[0093] In summary, by mining the time series features of task progress, a two-dimensional feature system of second-order differential decay coefficient and progress pattern fingerprint is constructed to accurately quantify the nonlinear decay characteristics of task execution, fundamentally solving the prediction bias problem of traditional linear prediction models. By distinguishing between three task execution states—natural slowness, exponential decay, and deep stagnation—three types of models—linear prediction, logarithmic decay prediction, and fixed upper limit stagnation prediction—the remaining execution time of tasks can be accurately calculated under different business scenarios, completely eliminating the underestimation of the remaining time of long-tail tasks and stagnant tasks. Relying on the local progress decay coefficient, the system can accurately distinguish between normal slow execution and the critical state of impending stagnation, avoiding erroneous intervention in normal tasks. Through the extraction of extreme value features from the progress pattern fingerprint, the system independently characterizes the most severe deceleration event of a single task, unaffected by the global task distribution of the cluster, resulting in higher feature recognition accuracy and stronger adaptability.

[0094] Abandoning the traditional single-mean load judgment model, this system innovatively introduces three dynamic process indicators: load change rate, fluctuation intensity, and overload trend strength index. These indicators comprehensively and multidimensionally quantify the real-time status, change trend, and fluctuation characteristics of node loads, addressing the limitations and lag of traditional single-indicator judgments. The load change rate enables predictive perception of load trends, identifying instantaneous load spikes and persistent overloads in advance, upgrading traditional a posteriori scheduling to predictive scheduling. The load fluctuation intensity quantifies the degree of load jitter, automatically increasing scheduling inertia for high-intensity disturbance scenarios to avoid overreaction and oscillation. The dimensionless overload trend strength index integrates multiple characteristics such as load location, change rate, and fluctuation intensity, significantly improving the robustness and accuracy of load overload judgment.

[0095] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An intelligent scheduling and optimization system adapted to distributed computing resources, characterized in that, The system configures a computing resource pool and a resource scheduler, and also includes: The closed-loop feedback scheduling enhancer is deployed in the bypass of the original scheduler. It dynamically modulates the input parameters of the original scheduler, namely the predicted value of the remaining time, and the output action, namely whether migration is allowed. It mines the progress time series of each task in real time, extracts the local progress decay coefficient and progress pattern fingerprint, and sets the sampling time t. The progress time series of each task is mined in real time. Based on the completion progress p(t) and sampling interval ΔT, the first-order difference v(t) and the second-order difference a(t) are calculated, and the attenuation coefficient α is also calculated. t =∣a(t)∣ / [v(t)+ε], where ε is a very small positive number; The (α) of the most recent L sampling points t ,v t For a progression pattern fingerprint Φ mapped as an extremum pair, the extremum pair includes the maximum decay coefficient αmax and the corresponding velocity v at that moment. α Identify and perform nonlinear predictions based on max and the duration τ of the maximum decay coefficient; A dual-loop scheduling controller with inertial hysteresis and predictive compensation inserts a dynamic admission filter before the original scheduler's migration decision, which senses the rate of load change in real time and calculates the sample standard deviation σ of the load values ​​within the window. L The intensity is calculated by weighting the range and the mean. The current load L(t), rate of change r(t), and intensity are combined to form a trend intensity index S. Based on the comparison results of the trend intensity index S with the first-level index threshold and the second-level index threshold, as well as the comparison results of the load fluctuation intensity with the intensity threshold, the dynamic high water level is output, and inertial hysteresis compensation analysis and predictive compensation analysis are performed.

2. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 1, characterized in that, The nonlinear prediction process is as follows: Determine the current progress p(t), current rate v(t), and decay coefficient α. t αmax and v in pattern fingerprint α max; If α t If the coefficient is less than the set threshold and αmax is lower than the maximum threshold, then the task is inferred to be in a stable execution phase, and the output is T. nonlin =T lin Specifically, non-linear time is equivalent to linear time, meaning that the execution follows the original scheduler's decision.

3. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 2, characterized in that, If α t If the rate exceeds the set coefficient threshold and the maximum coefficient threshold, and continues to rise, it enters the long-tail warning mode. At this time, a constant rate is no longer assumed; instead, it is assumed that the remaining progress follows a logarithmic decay model. time left , among them β is the empirical scaling factor; If τ (maximum decay duration) exceeds 3 sampling periods, and the current v(t) < 0.1 × v α max: If the condition is determined to be depth stagnation, set T directly. nonlin Set a preset upper limit value and mark the task as a non-transferable candidate.

4. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 3, characterized in that, When the original scheduler executes a migration decision, if the difference between the nonlinear time of the current task and the linear time of the original path continues to increase and the maximum attenuation coefficient still shows an increasing trend, the closed-loop feedback scheduling enhancer intervenes and makes a migration decision based on the real-time progress of the task and the remaining nonlinear time. If the current task recovers to a linear relationship, the original scheduler continues to execute the migration decision after the migration task is completed by the closed-loop feedback scheduling enhancer.

5. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 1, characterized in that, Collect load fluctuation intensity data and calculate the weighted mean of the sample standard deviation σL and range (maximum value - minimum value) of the load values ​​within the window: Intensity = 0.6 × σL L +0.4×(max(L)-min(L)).

6. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 5, characterized in that, Trend strength index S: ； L base L is the security baseline defined for the system. max r is the danger threshold. ref This is the reference rate used for normalization.

7. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 6, characterized in that, When the trend intensity index exceeds the first-level index threshold and the load fluctuation intensity is less than the intensity threshold, the system is in steady state. A strict threshold is adopted, and the dynamic high water level = high water level base value - displacement. When the trend strength index exceeds the primary index threshold but not the secondary index threshold, the threshold is increased linearly according to the trend strength, with the formula: Dynamic high water level = High water level base value. ; When the trend strength index exceeds the secondary index threshold, the dynamic high water level will rise further, but it is constrained by the recent peak: dynamic high water level = min(90%, max(recent peak + 2%, 85%)). Output dynamic high water level.

8. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 7, characterized in that, When the dynamic high water level exceeds the set high water level threshold, predictive compensation is performed. When the load change rate fluctuates continuously and the real-time load repeatedly swings between overload and non-overload, inertial hysteresis compensation is performed.

9. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 8, characterized in that, The predictive compensation is performed as follows: if the load does not exceed the threshold, and the trend strength index S rises for two consecutive sampling points and exceeds 0.6, micro-migration is performed in advance. The inertial hysteresis compensation is as follows: a time hysteresis loop is introduced: after a node enters the overload state, it must satisfy the condition that the load is continuously lower than the dynamic high water level minus a hysteresis width and this condition is maintained for a period of time before it exits the overload state.

10. The intelligent scheduling and optimization system adapted to distributed computing resources according to claim 9, characterized in that, When the load change rate r(t) exceeds the rate threshold, it is determined that the load at the current sampling point is increasing, and the dual-loop scheduling controller intervenes. When the load change rate r(t) does not exceed the rate threshold, it is determined that the load at the current sampling point is decreasing, and the dual-loop scheduling controller does not intervene.

Images