Data cloud resource queue hierarchical scheduling method and system based on artificial intelligence

Abstract

The invention discloses a data cloud resource queue hierarchical scheduling method and system based on artificial intelligence, and relates to the field of resource scheduling, and the method comprises the following operation steps: S1, constructing a dynamic multi-level feedback array; s2, carrying out dual-channel space-time prediction and elastic pre-distribution; s3, virtual scheduling proxy and gradient release; and S4, optimizing energy consumption. According to the data cloud resource queue hierarchical scheduling method and system based on artificial intelligence, the priority of tasks can be dynamically adjusted, an elastic time slice mechanism is provided, the task processing efficiency can be effectively improved, the response time of the tasks can be effectively prolonged, and the cost can be effectively reduced; according to the resource pre-allocation method, the situation that high-priority tasks are delayed or low-priority tasks occupy too much resources due to fixed rules can be avoided, through resource pre-allocation of dual-channel space-time prediction and elastic pre-allocation, the resource demand can be accurately predicted, the resources are reserved in advance according to the prediction result, and the phenomena of excessive pre-allocation of the resources and resource shortage can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of resource scheduling, and particularly to a method and system for hierarchical scheduling of data cloud resource queues based on artificial intelligence. Background Art

[0002] When transmitting services over a network, queue scheduling is used to implement how relay nodes and routers in the network select a queue to be forwarded from one or more packet queues.

[0003] When performing hierarchical scheduling of resource queues, traditional scheduling methods for resource queues all allocate resources using preset weights and cannot adapt to changes in task types. As a result, some latency-sensitive tasks are still processed according to fixed priorities, which may lead to response timeouts. At the same time, the server is controlled to be turned on and off based on a fixed threshold, resulting in frequent restarts of the server and waste of resources.

[0004] Therefore, it is necessary to propose a method and system for hierarchical scheduling of data cloud resource queues based on artificial intelligence to solve the above problems. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method and system for hierarchical scheduling of data cloud resource queues based on artificial intelligence, which can effectively solve the problems in the background art.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A method for hierarchical scheduling of data cloud resource queues based on artificial intelligence includes the following operating steps:

[0008] S1: Construction of a dynamic multi-level feedback array. Tasks that need to use resources are classified into three-level queues according to resource requirements, and the initial time slices are set to 2s, 4s, and 8s respectively. Through the calculation result of the dynamic adjustment priority formula, real-time operations are performed on the separated array;

[0009] S2: Dual-channel spatio-temporal prediction and elastic pre-allocation. Based on the dual-channel spatio-temporal attention network, spatio-temporal features are fused through the time channel and the space channel to predict the resource requirements for the next hour, and precise resource adjustment is achieved through the method of elastic pre-allocation. At the same time, the predicted data is associated with S1 for dynamic adjustment;

[0010] S3: Virtual scheduling agent and gradient release, which are used to monitor the queue status in real time and flexibly allocate resources;

[0011] S4: Energy consumption optimization. By means of the DDPG algorithm, the task resources and device status are monitored in real time, and continuous action is output to regulate the frequency, migrate virtual machines, and adjust the voltage, so as to optimize the operation with a multi-objective reward score online, which is used to balance energy consumption, performance, and cost, and achieve dynamic and fine control.

[0012] Preferably, in S1, the following steps are specifically included:

[0013] S101: Divide the queues of all tasks that need to be hierarchically scheduled into Q1 queue, Q2 queue, and Q3 queue. Among them, the Q1 queue is the VIP channel, the Q2 queue is the main computing force, and the Q3 queue is the storage warehouse;

[0014] S102: Queue dynamic adjustment mechanism, including Q1 compression: when the task response time of Q1 exceeds 50 ms, its time slice is shortened by 20%, and the minimum shortened time slice is 1 second, which is used to speed up the processing speed;

[0015] Q3 release: when the resource idle rate of the Q3 queue > 30%, release 30% of the resources to the Q1 queue and the Q2 queue, and extend the time slice to 16 seconds, which is used to reduce resource waste;

[0016] S103: Calculate the priority of the tasks assigned to the corresponding queues. The formula is:

[0017]

[0018] where w1, w2, and w3 are the weights of the delay factor, cost factor, and service level agreement compliance factor respectively, and w1 + w2 + w3 = 1; L a is the maximum allowable delay time of the task; L r is the actual delay time generated by the task; C b is the pre-set budget cost of the task; C a is the actual cost consumed by the task; S a is the actual compliance rate of the task; S p is the compliance rate promised by the service level agreement; P is the priority. The higher the calculated P, the higher the priority of the task. At the same time, create a virtual machine to ensure that high-priority tasks have sufficient resources;

[0019] S104: Dynamically adjust the weights. Adjustment of the delay weight w1: when the queue length of Q1-type tasks exceeds the preset threshold, w1 increases in steps of 0.05, but does not exceed 0.8. When the queue length of Q1-type tasks returns to normal, w1 gradually decreases to the original value;

[0020] Adjustment of the cost weight w2: When the market price of resources fluctuates, causing the average cost of tasks to rise by more than 10%, w2 increases in steps of 0.03, with a maximum of 0.6. When the cost drops back to the normal range, w2 is correspondingly reduced to the original value.

[0021] Adjustment of the service level agreement weight w3: When the passing rate of the service level agreement for 5 consecutive tasks is lower than 95%, w3 increases in steps of 0.04 until it reaches 0.7. When the passing rate of the service level agreement returns to the normal level, w3 gradually decreases to the original value.

[0022] Preferably, the division criteria for the Q1 queue include: allocation object, tasks sensitive to latency; admission condition: the maximum allowable latency of the task < 50 ms; initial configuration: each task runs for a maximum of 2 seconds of time slice.

[0023] The division criteria for the Q2 queue include: allocation object, tasks requiring a large amount of computing resources; admission condition: the task is expected to consume CPU resources > 70%; initial configuration: each task runs for a maximum of 4 seconds of time slice.

[0024] The division criteria for the Q3 queue include: allocation object, tasks dependent on storage; admission condition: the task is expected to generate an IO throughput > 200 MB / s; initial configuration: each task runs for a maximum of 8 seconds of time slice.

[0025] Preferably, in S2, it specifically includes the following steps:

[0026] S201: Dual-channel spatio-temporal attention network. Input the time-dimensional data and space-dimensional data of the current task into the dual-channel spatio-temporal attention network to form a time channel and a space channel. Merge the features of the time channel and the space channel through a splicing operation, and finally output the resource demand prediction value for the next 1 hour, which is used to analyze the periodicity of historical data and the dependency relationship between the servers used by tasks.

[0027] S202: Perform elastic pre-allocation on the virtual machines used to carry out task execution. Reserve resources in advance according to the prediction results. When the error exceeds the self-set threshold, trigger dynamic adjustment. Dynamic adjustment includes expansion adjustment and contraction adjustment. The condition for expansion adjustment is: prediction value > current resource volume × 115%; the condition for contraction adjustment is actual demand < prediction value × 85%. Calculate the required resource increase or decrease after dynamic adjustment according to the formula:

[0028]

[0029] where ΔR is the resource increase or decrease; y pred is the resource demand prediction value for the next 1 hour; R current is the currently allocated available resource volume; Runit Minimum expansion and contraction unit;

[0030] S203: Progressive resource adjustment. The newly started virtual machine first undertakes 10% of the total task load, and increases by 10% every 5 minutes until 100%, to avoid service crashes caused by excessive instantaneous load.

[0031] S204: Associate the resource demand prediction value in step S201 with w1. When the resource demand prediction value is higher than the self-set threshold, provide the weight of w1, otherwise reduce it.

[0032] Feed the calculated ΔR back to the queue dynamic adjustment mechanism. When ΔR is positive and the resources in queue Q1 are tense, trigger Q1 compression; when ΔR is negative and the resources in queue Q3 are idle, trigger Q3 release.

[0033] Preferably, in S3, it specifically includes the following steps:

[0034] S301: Virtual scheduling agent, which monitors the number of queue tasks, task response time, server energy consumption, and resource cost in real time. When the tasks in queue Q1 time out, forcibly preempt the resources of low-priority tasks. When the resources in Q3 are limited, release the idle resources to the common pool.

[0035] S303: According to the gap between the current resource amount and the minimum guaranteed amount, release resources according to the calculated release speed. The calculation formula is:

[0036]

[0037] Where is the resource release speed; 0.2 is the attenuation coefficient; R cur is the resource amount of the current virtual machine; R min is the guaranteed resource amount of the virtual machine; R max is the historical maximum resource amount of the virtual machine;

[0038] S304: When the actual load of the virtual machine is lower than 85% of the predicted value of the resource demand, start the resource release of S303.

[0039] Preferably, in S4, it specifically includes the following steps:

[0040] S401: Real-time status monitoring, monitoring the weight of the task queue priority, the distribution of the task queue types, the predicted value of the resource demand in the next 1 hour, the resource increase and decrease amount, the real-time load rate of the task queue, and the released resource amount.

[0041] S402: Define optimization operations, including CPU frequency adjustment operations, virtual machine migration operations, and dynamic voltage adjustment operations. The CPU frequency adjustment operation includes adjustment from the minimum frequency 0 to the maximum frequency 1; the virtual machine migration operation includes adjustment from no migration 0 to full migration 1; the dynamic voltage adjustment operation includes adjustment from the minimum voltage 0 to the maximum voltage 1;

[0042] S403: Calculate the reward score, with the formula:

[0043] r t = 0.6×(1 - E)+0.3×T - 0.1×C;

[0044] where E is the energy consumption ratio, T is the on-time completion rate of queue tasks, C is the migration cost of virtual machines, 0.6, 0.3, and 0.1 are default weights, and r t is the reward score;

[0045] S404: Based on the DDPG algorithm, optimize the operations through the following loop, including recording historical states, actions, rewards, and new states, adjusting the selection of optimization operations according to the reward score, predicting future rewards, and optimizing the judgment of operation values;

[0046] S405: Execute the optimization operations and provide feedback, adjust the defined optimization operations based on S404, and at the same time monitor changes in energy consumption, task completion time, and cost, and continue to adjust the optimization operations based on this.

[0047] An artificial intelligence-based data cloud resource queue hierarchical scheduling system, including a dynamic multi-level feedback queue management module, a dual-channel spatio-temporal prediction and pre-allocation module, a virtual scheduling agent module, and a multi-objective energy consumption optimization module. The dynamic multi-level feedback queue management module is used to classify task resources by priority and dynamically adjust queue parameters;

[0048] The dual-channel spatio-temporal prediction and pre-allocation module is used to predict resource requirements and adjust the number of virtual machines in advance;

[0049] The virtual scheduling agent module is used to monitor the queue status in real time and dynamically allocate resources;

[0050] The multi-objective energy consumption optimization module is used to reduce energy consumption while ensuring performance.

[0051] Preferably, the dynamic multi-level feedback queue management module includes a queue classification module, a priority calculation module, and a queue adjustment module. The queue classification module is used to classify queues into Q1, Q2, and Q3 queues and automatically allocate them to appropriate queues according to the task type. The priority calculation module calculates the priority level of the current queue based on the priority calculation formula. The queue adjustment module compresses and extends the time slices of Q1 and Q3 based on their working states and processes the idle resources of Q3.

[0052] Preferably, the dual-channel spatio-temporal prediction and pre-allocation module includes a spatio-temporal prediction module and an elastic pre-allocation module. The spatio-temporal prediction module outputs the predicted values of the resource requirements of each queue in the next hour through the time channel and the space channel in the spatio-temporal prediction engine. The elastic pre-allocation module triggers the expansion and contraction processing based on the predicted resource requirement prediction values.

[0053] Preferably, the virtual scheduling agent module includes a status monitoring module, a decision-making engine module, and a gradient release module. The status monitoring module monitors the number of queue tasks, the queue load rate, the task response time, the server energy consumption, and the resource cost in real time. The decision-making engine module includes preemptive scheduling: if the Q1 task times out, the task with a lower priority is forcibly terminated and its resources are released. It also includes resource degradation: when the Q3 resources are idle, they are marked as recyclable resources, and after verifying that they are resource-independent, they are released. The gradient release module is used to gradually release resources.

[0054] Compared with the prior art, the present invention provides an artificial intelligence-based data cloud resource queue hierarchical scheduling method and system, which have the following beneficial effects:

[0055] 1. The artificial intelligence-based data cloud resource queue hierarchical scheduling method and system can dynamically adjust the priority of tasks and have an elastic time slice mechanism, enabling the time slice of Q1 to be compressed to 1 second and the time slice of Q3 to be extended to 16 seconds. It can effectively improve the task processing efficiency. Through the priority calculation formula, it can comprehensively consider delay, cost, and service level, avoid the extremeization of a single goal, effectively improve the task response time and reduce costs, and avoid the delay of high-priority tasks or excessive resource occupation by low-priority tasks caused by fixed rules.

[0056] 2. The artificial intelligence-based data cloud resource queue hierarchical scheduling method and system can simultaneously analyze data in the time dimension and the space dimension through dual-channel spatio-temporal prediction and elastic pre-allocation of resources. Based on this, it can accurately predict resource requirements. Through elastic pre-allocation, resources can be reserved in advance according to the prediction results, avoiding the phenomena of over-pre-allocation of resources and resource shortages. At the same time, through progressive resource adjustment, it can reduce the instantaneous pressure on virtual machines and reduce the failure rate of servers.

[0057] 3. The artificial intelligence-based data cloud resource queue hierarchical scheduling method and system can monitor the load, response time, energy consumption, and cost of the queue in real time. Based on this, the response time of scheduling optimization operations can be improved, and the energy consumption can be reduced at the same time. Resources can be gradually released according to the load decline rate, avoiding system crashes caused by instantaneous release and improving the resource utilization efficiency.

[0058] 4. The artificial intelligence-based data cloud resource queue hierarchical scheduling method and system can adjust the frequency of the CPU through energy consumption optimization, and can preferentially reduce the frequency of high-energy-consuming servers, effectively improving the task completion rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a step diagram of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0060] In order to make the technical means, creative features, achieved purposes, and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

[0061] Embodiment 1:

[0062] As Figure 1 shown, an artificial intelligence-based data cloud resource queue hierarchical scheduling system includes S1: construction of a dynamic multi-level feedback array. Tasks that need to use resources are classified into three-level queues according to resource requirements, and the initial time slices are set to 2s, 4s, and 8s respectively. The array is operated in real time by dynamically adjusting the calculation results of the priority formula.

[0063] Specifically, it includes the following steps:

[0064] S101: Divide the queues for all tasks that need hierarchical scheduling into a Q1 queue, a Q2 queue, and a Q3 queue. The Q1 queue is the VIP channel, the Q2 queue is the main computing force, and the Q3 queue is the storage warehouse. The division criteria for the Q1 queue include: allocation object, tasks sensitive to latency; admission condition: the maximum allowable latency of the task < 50ms; initial configuration: each task runs for a maximum of 2 seconds of time slice;

[0065] The division criteria for the Q2 queue include: allocation object: tasks that require a large amount of computing resources; admission condition: the task is expected to consume > 70% of the CPU resources; initial configuration: each task runs for a maximum of 4 seconds of time slice;

[0066] The division criteria for the Q3 queue include: allocation object: tasks dependent on storage; admission condition: the task is expected to generate an IO throughput > 200MB / s; initial configuration: each task runs for a maximum of 8 seconds of time slice;

[0067] S102: Queue dynamic adjustment mechanism, including Q1 compression: When the task response time of Q1 exceeds 50 ms, its time slice is shortened by 20%, and the minimum shortened time slice is 1 second, which is used to speed up the processing speed;

[0068] Q3 release: When the idle rate of Q3 queue resources > 30%, 30% of the resources are released to Q1 queue and Q2 queue, and the time slice is extended to 16 seconds, which is used to reduce resource waste;

[0069] S103: Calculate the priority of the tasks assigned to the corresponding queues. The formula is:

[0070]

[0071] where w1, w2, and w3 are the weights of the delay factor, cost factor, and service level agreement compliance factor respectively, and w1 + w2 + w3 = 1; L a is the maximum allowable delay time of the task; L r is the actual delay time generated by the task; C b is the pre-set budget cost of the task; C a is the actual cost consumed by the task; S a is the actual compliance rate of the task; S p is the compliance rate promised by the service level agreement; P is the priority; the higher the calculated P, the higher the priority of the task. At the same time, a virtual machine is created to ensure that high-priority tasks have sufficient resources;

[0072] S104: Dynamically adjust the weights. Adjustment of the delay weight w1: When the queue length of Q1-type tasks exceeds the preset threshold, w1 increases in steps of 0.05, but does not exceed 0.8. When the queue length of Q1-type tasks returns to normal, w1 gradually decreases to the original value;

[0073] Adjustment of the cost weight w2: When the market price of resources fluctuates, resulting in the average cost of tasks rising by more than 10%, w2 increases in steps of 0.03, with a maximum of 0.6. When the cost drops to the normal range, w2 decreases accordingly to the original value;

[0074] Adjustment of the service level agreement weight w3: When the service level agreement compliance rate of 5 consecutive tasks is lower than 95%, w3 increases in steps of 0.04 until it reaches 0.7. When the service level agreement compliance rate returns to the normal level, w3 gradually decreases to the original value.

[0075] Example 2:

[0076] As Figure 1As shown in the figure, an artificial intelligence-based data cloud resource queue hierarchical scheduling system includes S2: Dual-channel spatio-temporal prediction and elastic pre-allocation. Based on the dual-channel spatio-temporal attention network, spatio-temporal features are fused through the time channel and the space channel to predict the resource requirements for the next hour, and precise resource adjustment is achieved through elastic pre-allocation. At the same time, the predicted data is associated with S1 for dynamic adjustment, which specifically includes the following steps:

[0077] S201: Dual-channel spatio-temporal attention network. Input the time-dimensional data and space-dimensional data of the current task into the dual-channel spatio-temporal attention network to form a time channel and a space channel. Merge the features of the time channel and the space channel through a concatenation operation, and finally output the predicted value of the resource requirements for the next hour, which is used to analyze the periodicity of historical data and the dependency relationship between the servers used by the tasks. The time-dimensional data includes the CPU utilization rate, memory usage, IO throughput, and network latency in the past 6 hours; the space-dimensional data includes the physical connection relationship of the server cluster required by the task;

[0078] S202: Perform elastic pre-allocation on the virtual machines used to carry out task execution. Reserve resources in advance according to the prediction results. When the error exceeds the self-set threshold, dynamic adjustment is triggered. Dynamic adjustment includes expansion adjustment and contraction adjustment. The condition for expansion adjustment is: predicted value > current resource volume × 115%; the condition for contraction adjustment is actual demand < predicted value × 85%. Calculate the required resource increase or decrease after dynamic adjustment according to the formula:

[0079]

[0080] where ΔR is the resource increase or decrease; y pred is the predicted value of the resource requirements for the next hour; R current is the currently allocated available resource volume; R unit is the minimum expansion and contraction unit;

[0081] S203: Progressive resource adjustment. The newly started virtual machine first undertakes 10% of the total task load, and increases by 10% every 5 minutes until 100%, which is used to avoid service crashes caused by excessive instantaneous load. During contraction, first migrate the task to other servers, and then shut down the target server to ensure service continuity;

[0082] S204: Associate the resource requirement prediction value in step S201 with w1. When the resource requirement prediction value is higher than the self-set threshold, provide the weight of w1, otherwise reduce it;

[0083] Feed the calculated ΔR back to the queue dynamic adjustment mechanism. When ΔR is positive and the resources in the Q1 queue are tense, trigger Q1 compression; when ΔR is negative and the resources in the Q3 queue are idle, trigger Q3 release.

[0084] Example 3:

[0085] As Figure 1 shown, an artificial intelligence-based hierarchical scheduling system for data cloud resource queues includes S3, virtual scheduling agents, and gradient release, which are used to monitor the queue status in real time and flexibly allocate resources. The specific steps are as follows:

[0086] S301: The virtual scheduling agent monitors the number of queue tasks, task response time, server energy consumption, and resource cost in real time. When the tasks in the Q1 queue time out, it forcibly preempts the resources of low-priority tasks. When there are resource limitations in Q3, it releases idle resources to the common pool;

[0087] S303: According to the gap between the current resource amount and the minimum guaranteed amount, resources are released according to the calculated release speed. The calculation formula is:

[0088]

[0089] where is the resource release speed; 0.2 is the attenuation coefficient; R cur is the resource amount of the current virtual machine; R min is the guaranteed resource amount of the virtual machine; R max is the historical maximum resource amount of the virtual machine;

[0090] S304: When the actual load of the virtual machine is lower than 85% of the predicted value of the resource demand, the resource release of S303 is started.

[0091] Example 4:

[0092] As Figure 1 shown, an artificial intelligence-based hierarchical scheduling system for data cloud resource queues includes S4: energy consumption optimization. With the help of the DDPG algorithm, it monitors the task resources and device status in real time, outputs continuous action regulation frequencies, migrates virtual machines, and adjusts voltages to optimize operations online with multi-objective reward scores, for balancing energy consumption, performance, and cost, and achieving dynamic fine control. The specific steps are as follows:

[0093] S401: Real-time status monitoring, monitoring the weights of task queue priorities, the distribution of task queue types, the predicted value of resource demand in the next 1 hour, the increase and decrease of resources, the real-time load rate of the task queue, and the released resource amount;

[0094] S402: Define optimization operations, including CPU frequency adjustment operations, virtual machine migration operations, and dynamic voltage adjustment operations. Among them, the CPU frequency adjustment operation includes adjustment from the minimum frequency 0 to the maximum frequency 1; the virtual machine migration operation includes adjustment from no migration 0 to full migration 1; the dynamic voltage adjustment operation includes adjustment from the minimum voltage 0 to the maximum voltage 1;

[0095] S403: Calculate the reward score using the formula:

[0096] r t = 0.6×(1 - E)+0.3×T - 0.1×C;

[0097] where E is the energy consumption ratio, T is the on-time completion rate of the queue tasks, C is the migration cost of the virtual machine, 0.6, 0.3, and 0.1 are the default weights, and r t is the reward score;

[0098] S404: Based on the DDPG algorithm, optimize the operations through the following loop, including recording historical states, actions, rewards, and new states, adjusting the selection of optimization operations according to the reward score, predicting future rewards, and optimizing the judgment of operation values. When multiple frequency reductions result in a negative reward, the algorithm will reduce the frequency reduction and preferentially select virtual machine migration;

[0099] S405: Execute the optimization operations and provide feedback. Adjust the defined optimization operations based on S404, and at the same time monitor changes in energy consumption, task completion time, and cost, and continue to adjust the optimization operations based on this.

[0100] Example 5:

[0101] An artificial intelligence-based data cloud resource queue hierarchical scheduling system, including a dynamic multi-level feedback queue management module, a dual-channel spatio-temporal prediction and pre-allocation module, a virtual scheduling agent module, and a multi-objective energy consumption optimization module. The dynamic multi-level feedback queue management module is used to classify task resources by priority and dynamically adjust queue parameters;

[0102] The dual-channel spatio-temporal prediction and pre-allocation module is used to predict resource requirements and adjust the number of virtual machines in advance;

[0103] The virtual scheduling agent module is used to monitor the queue status in real time and dynamically allocate resources;

[0104] The multi-objective energy consumption optimization module is used to reduce energy consumption while ensuring performance.

[0105] The dynamic multi-level feedback queue management module includes a queue classification module, a priority calculation module, and a queue adjustment module. The queue classification module is used to divide the queue into Q1, Q2, and Q3 queues and automatically allocate them to the appropriate queue according to the task type; the priority calculation module calculates the priority level of the current queue based on the priority calculation formula; the queue adjustment module compresses, extends the time slice of Q1 and Q3 based on their working status, and processes the idle resources of Q3.

[0106] The dual-channel spatio-temporal prediction and pre-allocation module includes a spatio-temporal prediction module and an elastic pre-allocation module. The spatio-temporal prediction module outputs the predicted values of the resource requirements of each queue in the next hour through the time channel and the space channel in the spatio-temporal prediction engine; the elastic pre-allocation module triggers the expansion and contraction processing based on the predicted resource requirement values.

[0107] The virtual scheduling agent module includes a status monitoring module, a decision engine module, and a gradient release module, which monitors the number of queue tasks, queue load rate, task response time, server energy consumption, and resource cost in real time; the decision engine module includes preemptive scheduling: if the Q1 task times out, the task with a lower priority is forcibly terminated and its resources are released, and it also includes resource degradation: when the Q3 resources are idle, they are marked as recyclable resources, and after verifying that they are resources without dependencies, they are released; the gradient release module is used to gradually release resources.

[0108] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for hierarchical scheduling of data cloud resource queues based on artificial intelligence, characterized in that: It includes the following operation steps: S1: Construction of a dynamic multi-level feedback array. Tasks that require resource usage are classified into three-level queues according to resource requirements, with initial time slices set to 2s, 4s, and 8s respectively. Through dynamically adjusting the calculation results of the priority formula, real-time operations are performed on the divided array; S2: Dual-channel spatio-temporal prediction and elastic pre-allocation. Based on the dual-channel spatio-temporal attention network, spatio-temporal features are fused through the time channel and the space channel to predict the resource requirements for the next hour, and precise resource adjustment is achieved through elastic pre-allocation. At the same time, the predicted data is associated with S1 for dynamic adjustment; S3: Virtual scheduling agent and gradient release, which are used to monitor the queue status in real time and flexibly allocate resources; S4: Energy consumption optimization. With the help of the DDPG algorithm, the task resources and device status are monitored in real time, and continuous action regulation frequencies, migrating virtual machines, and adjusting voltages are output to optimize operations online with multi-objective reward scores, which are used to balance energy consumption, performance, and cost to achieve dynamic fine control.

2. The hierarchical scheduling method for the data cloud resource queue based on artificial intelligence according to claim 1, characterized in that: In S1, it specifically includes the following steps: S101: Divide the queues for all tasks that require hierarchical scheduling into Q1 queue, Q2 queue, and Q3 queue. Among them, the Q1 queue is the VIP channel, the Q2 queue is the main computing force, and the Q3 queue is the storage repository; S102: Queue dynamic adjustment mechanism, including Q1 compression: When the task response time of Q1 exceeds 50ms, its time slice is shortened by 20%, and the minimum shortened time slice is 1 second, which is used to speed up the processing speed; Q3 release: When the resource idle rate of the Q3 queue > 30%, 30% of the resources are released to the Q1 queue and the Q2 queue, and the time slice is extended to 16 seconds, which is used to reduce resource waste; S103: Calculate the priorities of the tasks divided into the corresponding queues. The formula is: where w1, w2, and w3 are the weights of the delay factor, cost factor, and service level agreement compliance factor respectively, and w1 + w2 + w3 = 1; L a is the maximum allowable delay time for the task; L r is the actual delay time generated by the task; C b is the pre-set budget cost for the task; C a is the actual cost consumed by the task; S a is the actual compliance rate of the task; S p is the compliance rate promised by the service level agreement; P is the priority; the higher the calculated P, the higher the priority of the task. At the same time, a virtual machine is created to ensure that high-priority tasks have sufficient resources; S104: Dynamically adjust the weights. Adjustment of the delay weight w1: When the queue length of Q1-type tasks exceeds the preset threshold, w1 increases in steps of 0.05, but does not exceed 0.

8. When the queue length of Q1-type tasks returns to normal, w1 gradually decreases to the original value; Adjustment of the cost weight w2: When the market price of resources fluctuates, resulting in the average cost of tasks rising by more than 10%, w2 increases in steps of 0.03, with a maximum of 0.

6. When the cost drops to the normal range, w2 is correspondingly reduced to the original value; Adjustment of the service level agreement weight w3: When the service level agreement compliance rate of 5 consecutive tasks is lower than 95%, w3 increases in steps of 0.04 until it reaches 0.

7. When the service level agreement compliance rate returns to the normal level, w3 gradually decreases to the original value.

3. The method for hierarchical scheduling of data cloud resource queues based on artificial intelligence according to claim 2, characterized in that: The division criteria for the Q1 queue include: allocation object, tasks sensitive to delay; admission condition: the maximum allowable delay of the task < 50ms; initial configuration: each task runs for a maximum of 2 seconds of time slice; The division criteria for the Q2 queue include: allocation object: tasks that require a large amount of computing resources; admission condition: the task is expected to consume CPU resources > 70%; initial configuration: each task runs for a maximum of 4 seconds of time slice; The division criteria for the Q3 queue include: Assignment object: Tasks that rely on storage; Admission condition: The task is expected to generate an IO throughput > 200 MB / s; Initial configuration: Each task runs for a maximum of 8 seconds of time slice.

4. A method for hierarchical scheduling of data cloud resource queues based on artificial intelligence according to claim 2, characterized in that: In S2, it specifically includes the following steps: S201: Dual-channel spatio-temporal attention network. Input the time-dimensional data and space-dimensional data of the current task into the dual-channel spatio-temporal attention network to form a time channel and a space channel. Merge the features of the time channel and the space channel through a splicing operation, and finally output the predicted value of resource requirements for the next 1 hour, which is used to analyze the periodicity of historical data and the dependency relationship between the servers used by the tasks. S202: Elastic pre-allocation of the virtual machines used to carry out task execution. Reserve resources in advance according to the prediction results. When the error exceeds the self-set threshold, trigger dynamic adjustment. Dynamic adjustment includes expansion adjustment and contraction adjustment. The condition for expansion adjustment is: predicted value > current resource amount × 115%; The condition for contraction adjustment is actual demand < predicted value × 85%. Calculate the required increase or decrease in resources after dynamic adjustment according to the formula: where ΔR is the resource increase or decrease; y pred is the predicted resource demand value for the next 1 hour; R current is the currently allocated available resource quantity; R unit the minimum unit for capacity expansion and contraction; S203: Progressive resource adjustment. The newly started virtual machines first bear 10% of the total task load, and increase by 10% every 5 minutes until 100%, which is used to avoid service crashes caused by excessive instantaneous load. Associate the resource requirement prediction value in step S201 with w1. When the resource requirement prediction value is higher than the self-set threshold, provide the weight of w1, otherwise reduce it. Feed the calculated ΔR back to the queue dynamic adjustment mechanism. When ΔR is positive and the Q1 queue resources are tense, trigger Q1 compression; when ΔR is negative and the Q3 queue resources are idle, trigger Q3 release.

5. The hierarchical scheduling method for data cloud resource queues based on artificial intelligence according to claim 1 is characterized in that: In S3, it specifically includes the following steps: S301: Virtual scheduling agent. Real-time monitor the number of queue tasks, task response time, server energy consumption, and resource cost. When the tasks in the Q1 queue time out, forcibly preempt the resources of low-priority tasks. When there are resource restrictions in Q3, release the idle resources to the common pool. S303: Release resources according to the calculated release speed based on the gap between the current resource amount and the minimum guarantee amount. The calculation formula is: Among them is the release rate of resources; 0.2 is the attenuation coefficient; R cur is the resource amount of the current virtual machine; R min is the guaranteed resource amount of the virtual machine; R max is the historical maximum resource amount of the virtual machine; When the actual load of the virtual machine is lower than 85% of the predicted value of resource requirements, start the resource release in S303.

6. The data cloud resource queue hierarchical scheduling method based on artificial intelligence according to claim 1, characterized in that: In S4, it specifically includes the following steps: S401: Real-time status monitoring. Monitor the weights of task queue priorities, the distribution of task queue types, the predicted value of resource requirements for the next 1 hour, the increase or decrease in resources, the real-time load rate of the task queue, and the amount of released resources. S402: Define optimization operations, including CPU frequency adjustment operations, virtual machine migration operations, and dynamic voltage adjustment operations. Among them, the CPU frequency adjustment operation includes adjustment from the minimum frequency 0 to the maximum frequency 1; The virtual machine migration operation includes adjustment from no migration 0 to full migration 1; The dynamic voltage adjustment operation includes adjustment from the minimum voltage 0 to the maximum voltage 1. S403: Calculate the reward score. The formula is: r t = 0.6×(1 - E)+0.3×T - 0.1×C; Among them, E is the energy consumption ratio, T is the on-time completion rate of queue tasks, C is the migration cost of virtual machines, 0.6, 0.3, and 0.1 are default weights, and r t is the reward score; S404: Based on the DDPG algorithm, optimize the operations through the following loop, including recording historical states, actions, rewards, and new states, adjusting the selection of optimization operations according to the reward scores, predicting future rewards, and optimizing the judgment of operation values; S405: Execute the optimization operations and provide feedback. Adjust the defined optimization operations based on S404, while monitoring the energy consumption, task completion time, and cost changes, and continue to adjust the optimization operations based on this.

7. A data cloud resource queue hierarchical scheduling system based on artificial intelligence, which adopts a data cloud resource queue hierarchical scheduling method based on artificial intelligence as described in any one of the above claims 1-6, and includes a dynamic multi-level feedback queue management module, a dual-channel spatio-temporal prediction and pre-allocation module, a virtual scheduling agent module, and a multi-objective energy consumption optimization module, and is characterized in that: The dynamic multi-level feedback queue management module is used to classify task resources by priority and dynamically adjust the queue parameters; The dual-channel spatio-temporal prediction and pre-allocation module is used to predict resource requirements and adjust the number of virtual machines in advance; The virtual scheduling agent module is used to monitor the queue status in real time and dynamically allocate resources; The multi-objective energy consumption optimization module is used to reduce energy consumption while ensuring performance.

8. The hierarchical scheduling system for data cloud resource queues based on artificial intelligence according to claim 7, characterized in that: The dynamic multi-level feedback queue management module includes a queue classification module, a priority calculation module, and a queue adjustment module. The queue classification module is used to divide the queue into Q1, Q2, and Q3 queues and automatically allocate them to the appropriate queue according to the task type; the priority calculation module calculates the priority level of the current queue based on the priority calculation formula; the queue adjustment module compresses, extends the time slice of Q1 and Q3 based on their working states, and processes the idle resources of Q3.

9. A method for hierarchical scheduling of data cloud resource queues based on artificial intelligence according to claim 7, characterized in that: The dual-channel spatio-temporal prediction and pre-allocation module includes a spatio-temporal prediction module and an elastic pre-allocation module. The spatio-temporal prediction module outputs the predicted values of the resource requirements of each queue in the next hour through the time channel and space channel in the spatio-temporal prediction engine; the elastic pre-allocation module triggers the expansion and contraction processing based on the predicted resource requirement prediction values.

10. A data cloud resource queue hierarchical scheduling method based on artificial intelligence according to claim 7, characterized in that: The virtual scheduling agent module includes a status monitoring module, a decision engine module, and a gradient release module. The real-time monitoring includes the number of queue tasks, queue load rate, task response time, server energy consumption, and resource cost; the decision engine module includes preemptive scheduling: if the Q1 task times out, forcibly terminate the task with a lower priority and release its resources, and also includes resource degradation: when the Q3 resources are idle, mark them as recyclable resources, and release them after verifying that they are resource-independent; The gradient release module is used to gradually release resources.

Images