Resource scheduling optimization method and system based on deep learning

Abstract

The invention discloses a resource scheduling optimization method and system based on deep learning, and particularly relates to the technical field related to resource scheduling, real-time indexes such as CPU utilization rate, memory occupancy rate and network bandwidth are acquired through a lightweight monitoring agent, and resource demands in the future 3-10 minutes are predicted by using an improved LSTM (including a cross-cycle attention mechanism), so that resource scheduling optimization is realized. The heterogeneous resource matching degree is calculated in combination with a graph attention network, a hierarchical scheduling strategy and a dynamic fault-tolerant mechanism are implemented, and the scheduling effect is evaluated through a multi-objective optimization function. The system comprises a distributed sensing terminal, a predictive analysis engine, a decision center and other modules, and supports federated learning, elastic capacity expansion and contraction and visual evaluation. The resource utilization rate can be improved, delay and energy consumption are reduced, and the method is suitable for heterogeneous resource scheduling scenes such as cloud computing and edge computing.

Description

Technical Field

[0001] The present invention relates to the field of resource scheduling, and in particular to a resource scheduling optimization method and system based on deep learning. Background Art

[0002] In the field of cloud computing and edge computing, existing resource scheduling technologies are mainly based on static rule engines and threshold alarm mechanisms. Typical solutions include: (1) using the ARIMA model based on time series analysis to predict resource demand, and using the sliding average method with a fixed time window to process data; (2) using a weighted polling algorithm combined with resource tag matching to implement task scheduling, relying on manual experience to set weight parameters; (3) achieving fault tolerance through heartbeat detection and timeout retry mechanism, and adopting the master-slave node cold switching solution; (4) using linear programming methods to construct a single-objective optimization model, with resource utilization or task completion time as the single optimization indicator. These technologies generally adopt a centralized architecture and make decisions through a predefined policy library.

[0003] However, existing technologies often fail to effectively capture cross-cycle resource fluctuations, leading to the accumulation of long-term forecast errors. Static threshold strategies can easily lead to resource allocation imbalances in dynamic load scenarios, and passive fault-tolerance mechanisms suffer from high response delays, making it difficult to ensure business continuity. Furthermore, single-objective optimization models lack the dynamic balancing capabilities of multi-dimensional constraints, which can easily lead to mutual constraints between indicators. Resource matching methods based on manual rules fail to quantify the synergistic effects of heterogeneous resources, resulting in long-term low overall system resource utilization and severely restricting scheduling efficiency optimization in large-scale distributed environments. Summary of the Invention

[0004] The main purpose of the present invention is to provide a resource scheduling optimization method and system based on deep learning, which can effectively solve the problems in the background technology.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A resource scheduling optimization method based on deep learning, the specific steps are as follows:

[0007] Step 1: A lightweight monitoring agent deployed on the computing node collects triplet metrics consisting of CPU utilization μ, memory usage ρ, and network bandwidth β at adjustable intervals of 300ms-1s, where μ∈[0,1] represents the normalized CPU usage; ρ∈[0,1] represents the memory usage ratio; and β∈[0,1] represents the ratio of current bandwidth to maximum bandwidth.

[0008] Step 2: Input the time series data of 15 consecutive minutes into the improved LSTM prediction model and output the resource demand forecast value for the next t∈[3,10] minutes. The improved LSTM introduces a cross-cycle attention mechanism, and its attention weight calculation satisfies:

[0009]

[0010] Where: α ij represents the attention weight of the i-th time step to the j-th time step, W is the trainable parameter matrix, h i represents the LSTM hidden state at the i-th time step, and T is the time window length;

[0011] Step 3: Build a heterogeneous resource joint scheduling model, quantify the number of CPU cores c, GPU floating-point computing power f, and storage IOPS into a 32-dimensional feature vector V, and calculate the resource matching score S∈[0,1] through the graph attention network;

[0012] Step 4: Execute a hierarchical scheduling strategy based on the matching score S: when S ≥ 0.8, resources are immediately allocated; when 0.6 ≤ S < 0.8, the resource enters the candidate queue; when S < 0.6, cross-cluster scheduling is triggered;

[0013] Step 5: Based on the real-time health score H t Implement dynamic fault tolerance, when H t <θ (θ = 0.7 is the preset threshold), the backup resource switching is completed within Δt ≤ 50ms;

[0014] Step 6: Evaluate the scheduling effect through the multi-objective optimization function:

[0015]

[0016] Where: T resp is the task response delay (unit: milliseconds), T max is the maximum allowed delay threshold, C used is the actual resource consumption cost (unit: yuan / hour), C total is the upper limit of budget cost, E avg is the average energy consumption of the system (unit: kWh), E base is the benchmark energy consumption value, λ1, λ2, λ3 are adjustable weight coefficients and satisfy λ1+λ2+λ3=1.

[0017] Preferably, in step 1: the memory occupancy of the lightweight monitoring agent is ≤15MB, and it supports data collection through three protocols: IPMI, REDFISH, and SNMP; data transmission adopts a hierarchical compression strategy, the real-time data stream is compressed using the Zstandard algorithm (compression ratio ≥3:1), and the historical log is compressed using the brotli algorithm (compression ratio ≥5:1).

[0018] Preferably, in step 2: the improved LSTM prediction model includes 2 bidirectional LSTM layers (number of hidden units: 64) and 1 multi-head attention layer (number of heads: h=8); and the model training adopts an adaptive loss function.

[0019] Preferably, in step 3, the graph attention network includes three GATv2 convolutional layers, each with an output dimension of 64; and the resource matching score is calculated as:

[0020]

[0021] Where: σ is the sigmoid activation function, GAP represents the global average pooling operation, w k is the learnable weight of the k-th resource, V (k) The feature vector representing the k-th resource type, where K=3 represents three types of resources: CPU, GPU, and storage.

[0022] Preferably, the hierarchical scheduling strategy of step 4 includes:

[0023] Set elastic scaling boundary conditions: the scaling trigger condition is that μ>0.75 for 5 consecutive sampling periods, and the scaling condition is that μ<0.3 for 10 consecutive periods;

[0024] Implement a progressive grayscale release, pushing the new scheduling policy in three stages (10% / 30% / 100%), with an interval of ≥15 minutes between each stage;

[0025] Configure an automatic rollback mechanism. When the new policy causes the F-score to drop by more than 20%, it will roll back to the stable version within t≤300s.

[0026] Preferably, in step 5: health score H t The calculation of the fusion hardware failure probability P h , network packet loss rate η, and service timeout rate τ are calculated according to the following formula:

[0027] H t =1-min(1,αP h +βη+γτ)

[0028] Among them: α = 0.6 is the hardware failure weight coefficient, β = 0.3 is the network packet loss weight coefficient, and γ = 0.1 is the service timeout weight coefficient; the backup resource switching adopts a two-phase commit protocol to ensure transaction consistency.

[0029] Preferably, the step 6 further comprises:

[0030] Establish a visual evaluation dashboard to display resource utilization heat maps, scheduling decision flow charts, and cost-benefit analysis matrices in real time;

[0031] The Bayesian optimization algorithm is used to automatically adjust the weights of λ1, λ2, and λ3, and generate a Pareto frontier solution set every 24 hours;

[0032] Set up a policy simulation environment and support injecting 12 typical load patterns for stress testing.

[0033] A deep learning-based resource scheduling optimization system, used in the above method, comprising: distributed sensing terminals, a prediction and analysis engine, a decision center, a policy executor, a fault-tolerant controller, and an evaluation platform;

[0034] The distributed perception terminal is deployed through the edge computing node and establishes a gRPC long connection with the prediction analysis engine;

[0035] The decision center has a built-in resource map database and a policy knowledge base, and communicates with the policy executor through a message queue.

[0036] Preferably, the prediction analysis engine includes a feature extractor, a model trainer, and an online service module;

[0037] The feature extractor realizes wavelet noise reduction and trend decomposition functions;

[0038] The model trainer supports federated learning mode, aggregating gradient updates from multiple data centers;

[0039] The online service provides a prediction API interface with a latency of ≤10ms.

[0040] Preferably, the fault-tolerant controller includes a fault prediction module, a backup route calculation module, and a fast switching module;

[0041] The fault prediction module integrates the survival analysis model and outputs the remaining available time TTL;

[0042] The backup route calculation module maintains K = 3 redundant paths, and the path selection is based on an improved version of the Dijkstra algorithm;

[0043] The fast switching module realizes protocol-imperceptible migration and ensures session state continuity.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This invention significantly improves the accuracy and timeliness of heterogeneous resource scheduling through the synergy of multi-dimensional monitoring and intelligent prediction. The improved graph attention network achieves accurate quantitative evaluation of resource matching, and combined with the hierarchical scheduling strategy, effectively balances the needs of immediate response and global optimization. The dynamic fault-tolerance mechanism reduces operation and maintenance costs while ensuring service continuity through a health scoring model and a two-stage switching protocol. The deep integration of the visual evaluation platform and the multi-objective optimization algorithm provides a closed-loop feedback mechanism for the continuous improvement of scheduling strategies, ultimately achieving a triple improvement in resource utilization, service quality, and cost control. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the working steps of the present invention. DETAILED DESCRIPTION

[0047] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0048] The present invention provides a resource scheduling optimization method based on deep learning, which is implemented according to the following process:

[0049] Step 1: Resource monitoring and data collection:

[0050] (1) Deploy a lightweight monitoring agent with a memory footprint of ≤15MB, supporting parallel data collection using IPMI, REDFISH, and SNMP protocols, with data collection intervals dynamically configurable to a range of 300ms-1s;

[0051] (2) The collection indicators are defined as:

[0052] (3) Data transmission adopts a hierarchical compression strategy: real-time data stream: applying the Zstandard algorithm to ensure a compression ratio of ≥3:1 - historical log: applying the brotli algorithm to ensure a compression ratio of ≥5:1.

[0053] Step 2: Demand forecast modeling:

[0054] (1) An improved LSTM model structure is adopted, as follows:

[0055] Layer 1: Bidirectional LSTM, 64 hidden units, returns sequence data;

[0056] Layer 2: Bidirectional LSTM, 64 hidden units;

[0057] Attention layer: 8-head attention mechanism, weight calculation strictly follows:

[0058]

[0059] Where: α ij represents the attention weight of the i-th time step to the j-th time step, W is the trainable parameter matrix, h i represents the LSTM hidden state at the i-th time step, T is the time window length, and the model output is the resource demand forecast value for the next t∈[3,10] minutes. During training, an adaptive loss function is used to dynamically adjust the weight of each time step;

[0060] The model is trained using an adaptive loss function.

[0061] Step 3: Make resource matching decisions;

[0062] (1) Constructing a 32-dimensional feature vector:

[0063] V=φ(c,f,iops)

[0064] Where φ(·) is the feature encoding function, c represents the number of CPU cores, f represents the GPU floating point computing power, and iops represents the storage IOPS.

[0065] (2) The graph attention network consists of three GATv2 convolutional layers, and the output dimension of each layer is strictly kept at 64. The matching score is calculated as:

[0066]

[0067] Where: σ is the sigmoid activation function, GAP represents the global average pooling operation, w k is the learnable weight of the k-th resource, V (k) The feature vector representing the k-th resource type, where K=3 represents three types of resources: CPU, GPU, and storage.

[0068] Step 4: Based on the matching degree calculated in step 3, set the hierarchical scheduling strategy shown below;

[0069] (1) Scheduling threshold execution standards:

[0070]

[0071] (2) Elastic expansion and contraction conditions:

[0072] Expansion trigger: μ>0.75 for 5 consecutive sampling periods

[0073] Shrinkage trigger: μ<0.3 for 10 consecutive cycles

[0074] (3) Grayscale release is implemented in three phases:

[0075]

[0076] The specific process is as follows: Set elastic expansion and contraction boundary conditions: the expansion trigger condition is that μ>0.75 for 5 consecutive sampling periods, and the contraction condition is that μ<0.3 for 10 consecutive periods;

[0077] Implement a progressive grayscale release, pushing the new scheduling policy in three stages (10% / 30% / 100%), with an interval of ≥15 minutes between each stage;

[0078] Configure an automatic rollback mechanism. When the new policy causes the F-score to drop by more than 20%, it will roll back to the stable version within t≤300s.

[0079] Step 5: Establish a dynamic fault tolerance mechanism:

[0080] First, establish the health score H t , in practice, the health score H t Calculate the probability of fusion hardware failure P h , network packet loss rate η, and service timeout rate τ are calculated according to the following formula:

[0081] H t =1-min(1,αP h +βη+γτ)

[0082] Among them: α = 0.6 is the hardware failure weight coefficient, β = 0.3 is the network packet loss weight coefficient, and γ = 0.1 is the service timeout weight coefficient; the backup resource switching adopts a two-phase commit protocol to ensure transaction consistency.

[0083] Fault tolerance triggering condition: H < θ (θ = 0.7 is the preset threshold, and its specific value can be adjusted according to actual conditions), switch to backup resources, and ensure that the backup switch is completed within Δt ≤ 50ms. A two-phase commit protocol is used:

[0084]

[0085] Step 6: Multi-objective optimization evaluation;

[0086] The specific assessment is calculated strictly according to the following formula:

[0087]

[0088] Where: T resp is the task response delay (unit: milliseconds), T max is the maximum allowed delay threshold, C used is the actual resource consumption cost (unit: yuan / hour), C total is the upper limit of budget cost, E avg is the average energy consumption of the system (unit: kWh), E baseis the benchmark energy consumption value, λ1, λ2, λ3 are adjustable weight coefficients and satisfy λ1+λ2+λ3=1.

[0089] In practice, a visual evaluation dashboard was established to display real-time resource utilization heat maps, scheduling decision flow charts, and cost-benefit analysis matrices. A Bayesian optimization algorithm was used to automatically adjust the weights of λ1, λ2, and λ3, generating a Pareto frontier solution set every 24 hours. A policy simulation environment was set up to support stress testing by injecting 12 typical load patterns, including a burst traffic pattern (a 300% increase in requests per second), a periodic fluctuation pattern (amplitude ±40%), and a gradual growth pattern (linear growth slope 0.5% / min).

[0090] The present invention also discloses a system for implementing the above method steps, which includes: a distributed perception terminal, a prediction analysis engine, a decision center, a policy executor, a fault-tolerant controller, and an evaluation platform.

[0091] Distributed perception terminals are deployed through edge computing nodes and establish gRPC persistent connections with the prediction and analysis engine;

[0092] The predictive analysis engine includes a feature extractor, a model trainer, and an online service module. The feature extractor implements wavelet noise reduction and trend decomposition. The model trainer supports federated learning mode, aggregating gradient updates from multiple data centers. The online service module provides a prediction API interface with a latency of ≤10ms.

[0093] The decision center has a built-in resource map database and policy knowledge base, and interacts with the policy executor through a message queue;

[0094] The fault-tolerant controller includes a fault prediction module, a backup route calculation module, and a fast switching module; the fault prediction module integrates a survival analysis model and outputs the remaining available time TTL; the backup route calculation module maintains K = 3 redundant paths, and the path selection is based on an improved version of the Dijkstra algorithm; the fast switching module implements protocol-unaware migration to ensure session state continuity.

[0095] The following further discloses the method and system of the invention with reference to specific examples:

[0096] This example uses cloud computing resource scheduling as an example. The specific implementation process is as follows:

[0097] 1. Infrastructure configuration

[0098] The infrastructure notes in this example are as follows: Cluster composition: 3 availability zones, each containing 42 physical servers; Single server specifications: CPU: 2 × Intel Xeon Platinum 8480CL (56 cores / 112 threads); GPU: 4 × NVIDIA H100 (FP8 computing power 197.9 TFLOPS); Storage: 30TB NVMe (random read and write 1.2MIOPS); Network: 200Gbps InfiniBand HDR.

[0099] 2. Monitoring data collection

[0100] Acquisition parameters: Sampling interval: 500ms (dynamically adjustable to 300ms-1s); Data accuracy: μ, ρ, β all retain 4 decimal places; Transmission delay: ≤80ms (including compression / decompression time);

[0101] Compression performance verification:

[0102] Real-time data: 3.2:1 compression ratio (ZstandardLevel 3), historical log: 5.3:1 compression ratio (BrotliLevel 5).

[0103] 3. Implementation of demand forecast

[0104] LSTM model inference process: Input: 15 minutes of time series data (1800 sampling points); Attention mechanism calculation example:

[0105]

[0106] Then the output is: the predicted value for the next 5 minutes μ=0.82±0.03 (95% confidence interval).

[0107] 4. Resource matching decision

[0108] Feature vector construction: CPU features: 8 dimensions such as core utilization, context switch rate, and cache hit rate; GPU features: 12 dimensions such as SM utilization, tensor core occupancy, and memory bandwidth; Storage features: 12 dimensions such as IOPS, throughput, and queue depth.

[0109] Graph Attention Network Inference: First layer output: 64-dimensional feature vector (8 attention heads); final matching score: S = 0.79 (triggering candidate queue strategy)

[0110] 5. See Table 1 for hierarchical scheduling execution:

[0111] Table 1: Execution time, CPU average, connection cycle number and specific operation details of the hierarchical segment in this example

[0112] Timestamp CPU average Number of consecutive cycles operate 2025-05-18T10:00 0.76 1 monitor 2025-05-18T10:02 0.81 5 Expansion +2 nodes 2025-05-18T10:30 0.28 10 Shrink -1 node

[0113] 6. Fault-tolerant switching process

[0114] Health calculation example: H = 1 - (0.6 × 0.15 + 0.3 × 0.08 + 0.1 × 0.12) = 0.874.

[0115] Table 2: Decomposition details of the two-stage switching delay:

[0116] stage Operation content Time consumed (ms) Pre-submit Resource verification and reservation 18 State Migration Memory snapshot transfer (12GB) 22 Final submission Transaction confirmation and routing switching 7 total —— 47

[0117] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A resource scheduling optimization method based on deep learning, characterized in that: The specific steps are as follows: Step 1: A lightweight monitoring agent deployed on the computing node collects triplet metrics consisting of CPU utilization μ, memory usage ρ, and network bandwidth β at adjustable intervals of 300ms-1s, where μ∈[0,1] represents the normalized CPU usage; ρ∈[0,1] represents the memory usage ratio; and β∈[0,1] represents the ratio of current bandwidth to maximum bandwidth. Step 2: Input the time series data of 15 consecutive minutes into the improved LSTM prediction model and output the resource demand forecast value for the next t∈[3,10] minutes. The improved LSTM introduces a cross-cycle attention mechanism, and its attention weight calculation satisfies: Where: α ij represents the attention weight of the i-th time step to the j-th time step, W is the trainable parameter matrix, h i represents the LSTM hidden state at the i-th time step, and T is the time window length; Step 3: Build a heterogeneous resource joint scheduling model, quantify the number of CPU cores c, GPU floating-point computing power f, and storage IOPS into a 32-dimensional feature vector V, and calculate the resource matching score S∈[0,1] through the graph attention network; Step 4: Execute a hierarchical scheduling strategy based on the matching score S: when S ≥ 0.8, resources are immediately allocated; when 0.6 ≤ S < 0.8, the resource enters the candidate queue; when S < 0.6, cross-cluster scheduling is triggered; Step 5: Based on the real-time health score H t Implement dynamic fault tolerance, when H t <θ (θ = 0.7 is the preset threshold), the backup resource switching is completed within Δt ≤ 50ms; Step 6: Evaluate the scheduling effect through the multi-objective optimization function: Where: T resp is the task response delay (unit: milliseconds), T max is the maximum allowed delay threshold, C used is the actual resource consumption cost (unit: yuan / hour), C total is the upper limit of budget cost, E avg is the average energy consumption of the system (unit: kWh), E base is the benchmark energy consumption value, λ1, λ2, λ3 are adjustable weight coefficients and satisfy λ1+λ2+λ3=1.

2. The resource scheduling optimization method based on deep learning according to claim 1, characterized in that: In step 1: the memory occupancy of the lightweight monitoring agent is ≤15MB, and it supports data collection through three protocols: IPMI, REDFISH, and SNMP; data transmission adopts a hierarchical compression strategy, the real-time data stream is compressed using the Zstandard algorithm (compression ratio ≥3:1), and the historical log is compressed using the brotli algorithm (compression ratio ≥5:1).

3. The resource scheduling optimization method based on deep learning according to claim 1 is characterized in that In step 2: the improved LSTM prediction model includes 2 bidirectional LSTM layers (number of hidden units: 64) and 1 multi-head attention layer (number of heads: h = 8); the model training adopts an adaptive loss function.

4. The resource scheduling optimization method based on deep learning according to claim 1, characterized in that: In step 3, the graph attention network includes three GATv2 convolutional layers, each with an output dimension of 64. The resource matching score is calculated as: Where: σ is the sigmoid activation function, GAP represents the global average pooling operation, w k is the learnable weight of the k-th resource, V (k) The feature vector representing the k-th resource type, where K=3 represents three types of resources: CPU, GPU, and storage.

5. The resource scheduling optimization method based on deep learning according to claim 1, characterized in that: The hierarchical scheduling strategy of step 4 includes: Set elastic scaling boundary conditions: the scaling trigger condition is that μ>0.75 for 5 consecutive sampling periods, and the scaling condition is that μ<0.3 for 10 consecutive periods; Implement a progressive grayscale release, pushing the new scheduling policy in three stages (10% / 30% / 100%), with an interval of ≥15 minutes between each stage; Configure an automatic rollback mechanism. When the new policy causes the F-score to drop by more than 20%, it will roll back to the stable version within t≤300s.

6. The resource scheduling optimization method based on deep learning according to claim 1, characterized in that: In step 5: health score H t The calculation of the fusion hardware failure probability P h , network packet loss rate η, and service timeout rate τ are calculated according to the following formula: H t =1-min(1,αP h +βε+γτ) Among them: α = 0.6 is the hardware failure weight coefficient, β = 0.3 is the network packet loss weight coefficient, and γ = 0.1 is the service timeout weight coefficient; the backup resource switching adopts a two-phase commit protocol to ensure transaction consistency.

7. The resource scheduling optimization method based on deep learning according to claim 1, characterized in that: The step 6 further comprises: Establish a visual evaluation dashboard to display resource utilization heat maps, scheduling decision flow charts, and cost-benefit analysis matrices in real time; The Bayesian optimization algorithm is used to automatically adjust the weights of λ1, λ2, and λ3, and generate a Pareto frontier solution set every 24 hours; Set up a policy simulation environment and support injecting 12 typical load patterns for stress testing.

8. A resource scheduling optimization system based on deep learning, used to implement the method according to any one of claims 1 to 7, characterized in that: include: Distributed sensing terminals, prediction and analysis engines, decision centers, policy executors, fault-tolerant controllers, and evaluation platforms; The distributed perception terminal is deployed through the edge computing node and establishes a gRPC long connection with the prediction analysis engine; The decision center has a built-in resource map database and a policy knowledge base, and interacts with the policy executor through a message queue.

9. The system according to claim 8, characterized in that: The prediction analysis engine includes a feature extractor, a model trainer, and an online service module; The feature extractor realizes wavelet noise reduction and trend decomposition functions; The model trainer supports federated learning mode, aggregating gradient updates from multiple data centers; The online service provides a prediction API interface with a latency of ≤10ms.

10. The system according to claim 8, characterized in that: The fault-tolerant controller includes a fault prediction module, a backup route calculation module, and a fast switching module; The fault prediction module integrates the survival analysis model and outputs the remaining available time TTL; The backup route calculation module maintains K = 3 redundant paths, and the path selection is based on an improved version of the Dijkstra algorithm; The fast switching module realizes protocol-imperceptible migration and ensures session state continuity.

Images