Cloud computing task storage resource intelligent scheduling method based on adaptive graph neural network

Abstract

The invention discloses a cloud computing task storage and computing resource intelligent scheduling method based on an adaptive graph neural network. The method comprises the following steps: S1, constructing a cloud computing basic information set; s2, constructing a graph neural network model of tasks and resources; s3, using the cloud computing basic information set to initialize the graph neural network model; s4, gradually optimizing a resource allocation strategy of the task; s5, updating the embedded vector of the task node based on the optimized graph neural network model; s6, scheduling computing resources and storage resources in real time according to the task priority and the generated resource allocation decision; s7, dynamically updating the embedding vector and the resource allocation decision of the task node; and S8, optimizing the distribution efficiency of storage and computing resources in the cloud computing environment. The resource utilization rate of the system is improved by about 18%, and meanwhile the task execution success rate is improved by about 9%.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to an intelligent scheduling method for storage and computing resources of cloud computing tasks based on an adaptive graph neural network. Background Art

[0002] In the prior art, task scheduling and resource allocation in cloud computing environments usually rely on static or rule-driven scheduling algorithms. Traditional methods mainly allocate resources based on the predetermined requirements and historical data of tasks. Scheduling algorithms generally allocate computing resources and storage resources to each task in a fixed ratio in a static manner. With the increase in the scale and complexity of cloud computing, the dynamic nature of tasks and the uncertainty of resource requirements have significantly increased. Traditional static scheduling methods have exposed many problems:

[0003] First, the traditional static scheduling method cannot respond to changes in task requirements in real time, resulting in low resource utilization. Once the computing or storage requirements of the task fluctuate, the system cannot flexibly adjust resource allocation, resulting in resource waste or task execution delays, ultimately affecting the overall performance and response speed of the system.

[0004] Secondly, existing scheduling algorithms have limitations when dealing with complex dependencies between tasks. Cloud computing tasks often have complex dependencies such as data sharing and calculation result transmission. Especially in large-scale distributed systems, the relationships between tasks are complex and dynamic. Traditional scheduling methods are difficult to effectively handle these dependencies and are prone to conflicts during resource allocation, resulting in task scheduling failure or intensified resource competition, further reducing the execution efficiency of tasks. In addition, existing technologies lack flexibility in dealing with load balancing problems of different resource nodes and cannot dynamically adjust resource allocation strategies according to the real-time load of the nodes, resulting in frequent overloads of some nodes and idle resources of other nodes.

[0005] In addition, traditional scheduling methods are also difficult to optimize in combination with historical task execution data. Most existing technologies only rely on static historical data or simple rule guidance, and lack the ability to dynamically adjust based on historical task execution data and current resource usage. They cannot optimize scheduling strategies based on task execution results during actual operation. Therefore, resource allocation plans cannot adapt to changes in task requirements in a timely manner, and the system's adaptability and flexibility are insufficient.

[0006] To sum up, the existing technology has the following main defects: first, the traditional static scheduling method lacks dynamics and is difficult to cope with real-time changes in task requirements and resource status; second, improper handling of complex dependencies between tasks can easily cause resource conflicts and scheduling failures; finally, there is a lack of optimization mechanism based on historical task execution data, and the scheduling strategy lacks adaptability and flexibility, making it difficult to achieve efficient resource scheduling in a complex cloud computing environment. Summary of the invention

[0007] One purpose of the present invention is to propose an intelligent scheduling method for storage and computing resources of cloud computing tasks based on an adaptive graph neural network. The present invention improves the resource utilization of the system by about 18%, and at the same time improves the task execution success rate by about 9%.

[0008] According to an embodiment of the present invention, a method for intelligently scheduling storage and computing resources for cloud computing tasks based on an adaptive graph neural network comprises the following steps:

[0009] S1. Collect computing requirements, storage requirements, task dependencies, task priorities, and task-related resource status information of multiple tasks in a cloud computing environment to build a cloud computing basic information set;

[0010] S2. Based on the task computing requirements, storage requirements and task dependencies, the task nodes are modeled with the computing nodes and storage nodes in the cloud computing environment. The task nodes and resource nodes are represented as nodes in the graph structure. The dependencies between tasks and the resource status of resource nodes are used as the edge information in the graph to build a graph neural network model of tasks and resources.

[0011] S3. Initialize the graph neural network model using the cloud computing basic information set, embed the task nodes and resource nodes in the graph structure, generate the embedding vectors of the task nodes and the embedding vectors of the resource nodes, and the embedding vectors reflect the task resource requirements and the dependencies between tasks and the current availability status of resources;

[0012] S4. The historical task execution data and current resource usage in the cloud computing basic information set are used as the input of the reinforcement learning model. Based on the reward mechanism of the reinforcement learning model, the parameters in the graph neural network model are dynamically adjusted according to the historical execution results of the task and the resource utilization rate. The reinforcement learning model evaluates the effect of each resource allocation through the reward function and gradually optimizes the resource allocation strategy of the task.

[0013] S5. Update the embedding vector of the task node based on the optimized graph neural network model and generate new resource allocation decisions so that the task can be executed under the optimal resource conditions.

[0014] S6. Schedule computing resources and storage resources in real time according to the task priority and the generated resource allocation decision, allocate the required computing resources and storage resources to the task nodes, and reasonably arrange the execution time of the tasks according to the execution order of the tasks;

[0015] S7. During the task execution process, the cloud computing resource status information and task execution progress are monitored in real time. If changes are detected in task requirements or system resource status, the parameters of the graph neural network model are readjusted to dynamically update the embedding vectors of the task nodes and resource allocation decisions.

[0016] S8. By continuously applying the joint optimization strategy of reinforcement learning and graph neural network, after the task is completed, the resource allocation strategy optimized by the reinforcement learning model is applied to the scheduling of subsequent tasks, thereby optimizing the allocation efficiency of storage and computing resources in the cloud computing environment.

[0017] Optionally, S1 includes the following sub-steps:

[0018] S11. Collect each task T i The computational demand C(T i ), defined as task T i The total amount of computing resources required during the entire execution process;

[0019] S12. Collect each task T i Storage requirement S(T i ), defined as task T i The total amount of storage space required during execution;

[0020] S13, build the dependency matrix D between tasks, each task T i With task T j The dependence D(T i , T j ) is defined as:

[0021]

[0022] The dependencies between all tasks form a dependency matrix D, with a dimension of N×N, where N is the total number of tasks;

[0023] S14, collection task priority P(T i ), the priority of the task P(T i ) is the task urgency U(T i ) and task dependency D(T i , T j ) and combined with resource allocation and task scheduling to calculate:

[0024]

[0025] Among them, U(T i ) is task T i The urgency of the task indicates the time sensitivity of the task, D(T i , T j) is task T i With task T j The degree of dependency between them indicates the strength of the dependency relationship, W j For task T j The current weight of the task represents the overall resource requirements and its relative importance, R c (T i ) is task T i The amount of computing resources currently available, R s (T i ) is task T i The amount of storage resources currently available, C(T i ) is task T i The computational demand, S(T i ) is task T i The storage demand of , α1, β1 and γ1 are weight coefficients used to balance urgency, dependency and resource utilization, ∈ is a small constant to avoid the denominator being zero;

[0026] S15. Collect resource status information related to the task, including available computing resources R in the cloud computing environment c and available storage resources R s ;

[0027] S16. Build a cloud computing basic information set through S11-S15:

[0028] I={C(T i ), S(T i ), D(T i , T i ), P(T i ), R c , R s}.

[0029] Optionally, S2 includes the following sub-steps:

[0030] S21, each task T in the cloud computing task i Represented as task node v i , the computing resource nodes and storage resource nodes in the cloud computing environment are represented as computing nodes and storage nodes Build a graph structure of tasks and resources, where:

[0031] Task Node v i Represents task T i and its corresponding computing and storage requirements;

[0032] Compute Node Represents the kth computing resource node, including the amount of available computing resources

[0033] Storage Node Indicates the first storage resource node, including the amount of available storage resources

[0034] S22, based on the dependency relationship between tasks D(T i , T j ) at task node v i and task node v j Create an edge between ij , and define the edge weight w ij Indicates the strength of the dependency between tasks. The dependency weight is defined as follows:

[0035] w ij =D(T i , T j );

[0036] Among them, D(T i , T j ) represents task T i With task T j The dependency between them constitutes the dependency edge set E between task nodes. T ;

[0037] S23, at task node v i and compute nodes Create an edge between Edge Weight Reflection Task T i The intensity of computing resource requirements:

[0038]

[0039] Among them, C(T i ) is task T i The computational requirements For computing nodes The amount of available computing resources constitutes the edge set E between the task node and the computing node C ;

[0040] S24, at task node v i and storage nodes Create an edge between Its edge weight Reflection Task T i The intensity of demand for storage resources:

[0041]

[0042] Among them, S(T i ) is task T istorage requirements, For storage nodes The amount of available storage resources constitutes the edge set E between the task node and the storage node S ;

[0043] S25. Construct a graph neural network model including task nodes, computing nodes, storage nodes and the edge information between them:

[0044] G=(V,E T ∪E C ∪E S ):

[0045] in, is a node set, including the task node v i , compute nodes and storage nodes E T ∪E C ∪E S is an edge set, including the dependency edge set E between task nodes T , the edge set E between the task node and the computing node C and the edge set E between the task node and the storage node S .

[0046] Optionally, S3 includes the following sub-steps:

[0047] S31. Use the cloud computing basic information set to initialize the graph neural network model and construct the graph structure G = (V, E T ∪E C ∪E S ) is input into the graph neural network;

[0048] S32. Initialize the feature vector of each node in the graph structure and set the initial feature vector For task node v i , the initial eigenvector is:

[0049]

[0050] For compute nodes Initial eigenvector:

[0051]

[0052] For storage nodes Initial eigenvector:

[0053]

[0054] S33. In the graph neural network model, the message passing mechanism is used to iteratively update the node feature vector:

[0055]

[0056] in, is the feature vector of node v after the first iteration, N(v) is the set of neighbor nodes of node v, φ(·) is the message function, combining the feature vectors of node v and its neighbor node u and the edge weight w vu Calculate the message, w vu is the edge weight between node v and node u, reflecting the strength of the relationship between nodes, and σ(·) is the activation function;

[0057] S34, repeat step S33 until the number of iterations reaches the preset number of layers L or the feature vector converges, and finally obtain the embedding vector of each node

[0058] S35, obtain the final embedding vectors of the task node and the resource node, the embedding vector of the task node Combined with Task T i The resource requirements, task priorities and dependencies between tasks are calculated, and the embedding vectors of the computing nodes are Reflects the computing resource nodes The current available computing resource status and its association with the task node, and the embedding vector of the storage node Reflects the storage resource node The current status of available storage resources and their association with task nodes.

[0059] Optionally, S4 includes the following sub-steps:

[0060] S41, the historical task execution data H(T i ) and current resource usage U(R c , R s ) is used as the input of the reinforcement learning model, and the historical task execution data H(T i ) includes task T i The historical execution time, resource utilization rate and task execution success rate of the task, and the current resource usage U(R c , R s ) including the real-time availability of computing resources and storage resources;

[0061] S42. Define the state s in the reinforcement learning model based on the input historical task execution data and current resource usage t ;

[0062] S43, based on state s t and the embedding vector of the task node The reinforcement learning model generates action a for task resource allocationt :

[0063]

[0064] Among them, Q(s t , a; θ t ) indicates that the current policy function is in state s t and action a t The expected reward value under t is the current weight of the graph neural network model, w ik Represents the task node v i and its neighbor node v k The edge weight between i ) represents the task node v i The set of neighbor nodes of It is an evaluation function for matching computing resource requirements with available resources, used to measure the task T i On the current computing node The resource allocation efficiency on the ,λ1 and λ2 are the trade-off coefficients;

[0065] S44. Evaluate each resource allocation result through the reward mechanism of the reinforcement learning model and define the reward function R(s t , a t ), the reward function is evaluated based on task execution time, resource utilization, and execution success rate:

[0066]

[0067] Among them, T exec (T i ) represents task T i The actual execution time, R c (T i ) and R s (T i ) represent tasks T i The computing resources and storage resources allocated, C(T i ) and S(T i ) are respectively tasks T i The computing demand and storage demand of the task are as follows: α2, α3, and α4 are the weight coefficients for evaluating the task execution time, resource utilization, and execution success rate, respectively;

[0068] S45, according to the reward function R(s t , a t ), the reinforcement learning model dynamically adjusts the parameters θ in the graph neural network model, and the optimized policy function Gradually improve the efficiency of resource allocation for tasks:

[0069]

[0070] Among them, η1 is the learning rate, Q(s t , a t θ t ) is the expected reward of the policy function, w ij is the dependency weight between task nodes, which is used to balance the impact between tasks during the update process. is the policy function for the model parameters θ t The gradient of , which measures the impact of the current strategy on the parameters;

[0071] S46, optimize the task resource allocation strategy through multiple iterations until the reward function R(s t , a t ) converges, the reinforcement learning model gradually optimizes the parameters in the graph neural network according to historical data and current resource usage to achieve the optimal match between tasks and resources.

[0072] Optionally, S6 includes the following sub-steps:

[0073] S61, based on task priority P(T i ) and resource allocation decisions, and calculate each task node v in real time i Required computing resources R c (T i ) and storage resources R s (T i ), and allocate corresponding resources to task nodes through scheduling strategies;

[0074] S62, according to the task priority P(T i ) for all task nodes v i Sorting: Task nodes with high priority will be given priority to obtain computing resources and storage resources, and will be allocated execution time based on the priority scheduling algorithm. The task scheduling order is:

[0075]

[0076] Among them, O(T) represents the task scheduling order, P(T i ) is task T i The priority of T exec (T i ) is task T i The estimated execution time of tasks with high priority and short execution time is guaranteed to be executed first;

[0077] S63. Based on the task scheduling sequence O(T) and resource allocation results, reasonably arrange the task execution time T start (T i ) and T end (Ti ), during the task execution process, monitor the execution progress and resource utilization of the task nodes in real time. If bottlenecks or sudden changes in computing resources or storage resources are detected during the task execution process, dynamically adjust the task execution order and resource allocation strategy;

[0078] S64: After all tasks are completed, output the task execution log, including the start time T of each task. start (T i ), end time T end (T i ), the amount of computing resources and storage resources used, as well as the success rate of task execution and resource utilization.

[0079] The beneficial effects of the present invention are:

[0080] (1) The present invention adopts an adaptive graph neural network to model tasks and resource nodes, generates embedded vectors of tasks and resources by real-time collection of the computing requirements, storage requirements and dependencies between tasks, and combines with the reinforcement learning model to dynamically adjust the parameters in the graph neural network according to the historical data of task execution and resource utilization. When the task requirements or resource status change, the system can update the resource allocation strategy in real time and make adjustments quickly to avoid resource waste and scheduling delay problems in traditional static scheduling methods. Experimental results show that compared with traditional static scheduling algorithms, the method of the present invention reduces scheduling delay by about 12.5% ​​and improves resource utilization by about 15% in a large-scale cloud computing environment.

[0081] (2) The present invention can capture the complex dependencies between task nodes and generate embedded vectors of task nodes through a message passing mechanism, reflecting the dependency strength and resource requirements between tasks. During the scheduling process, the present invention dynamically optimizes the task execution order based on task priority and task dependency, ensuring that tasks with high priority and short execution time obtain resources first to avoid resource competition and scheduling conflicts. Compared with traditional methods, the task execution order is more reasonable, the task execution time is reduced by about 8%, and the task conflict rate is reduced by about 10%.

[0082] (3) The present invention realizes dynamic optimization of graph neural network parameters by combining historical task execution data and current resource usage with a reinforcement learning model. After each resource allocation, the system evaluates the effect of the allocation strategy through the reward mechanism of the reinforcement learning model and adjusts the scheduling strategy. Through multiple iterative optimizations, the present invention can effectively improve the efficiency of resource allocation so that resources can quickly adapt to dynamically changing task requirements, thus avoiding the limitation of traditional scheduling methods that cannot optimize scheduling strategies based on historical execution data. Experimental data show that the method of the present invention increases the resource utilization of the system by about 18%, and at the same time increases the task execution success rate by about 9%. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0084] Figure 1 This is a flow chart of a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on an adaptive graph neural network proposed by the present invention;

[0085] Figure 2 This is a schematic diagram of modeling cloud computing task nodes and resource nodes in a method for intelligent scheduling of cloud computing task storage and computing resources based on an adaptive graph neural network proposed by the present invention. DETAILED DESCRIPTION

[0086] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0087] refer to Figure 1-2 , a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network, comprising the following steps:

[0088] S1. Collect computing requirements, storage requirements, task dependencies, task priorities, and task-related resource status information of multiple tasks in a cloud computing environment to build a cloud computing basic information set;

[0089] S2. Based on the task computing requirements, storage requirements and task dependencies, the task nodes are modeled with the computing nodes and storage nodes in the cloud computing environment. The task nodes and resource nodes are represented as nodes in the graph structure. The dependencies between tasks and the resource status of resource nodes are used as the edge information in the graph to build a graph neural network model of tasks and resources.

[0090] S3. Initialize the graph neural network model using the cloud computing basic information set, embed the task nodes and resource nodes in the graph structure, generate the embedding vectors of the task nodes and the embedding vectors of the resource nodes, and the embedding vectors reflect the task resource requirements and the dependencies between tasks and the current availability status of resources;

[0091] S4. The historical task execution data and current resource usage in the cloud computing basic information set are used as the input of the reinforcement learning model. Based on the reward mechanism of the reinforcement learning model, the parameters in the graph neural network model are dynamically adjusted according to the historical execution results of the task and the resource utilization rate. The reinforcement learning model evaluates the effect of each resource allocation through the reward function and gradually optimizes the resource allocation strategy of the task.

[0092] S5. Update the embedding vector of the task node based on the optimized graph neural network model and generate new resource allocation decisions so that the task can be executed under the optimal resource conditions.

[0093] S6. Schedule computing resources and storage resources in real time according to the task priority and the generated resource allocation decision, allocate the required computing resources and storage resources to the task nodes, and reasonably arrange the execution time of the tasks according to the execution order of the tasks;

[0094] S7. During the task execution process, the cloud computing resource status information and task execution progress are monitored in real time. If changes are detected in task requirements or system resource status, the parameters of the graph neural network model are readjusted to dynamically update the embedding vectors of the task nodes and resource allocation decisions.

[0095] S8. By continuously applying the joint optimization strategy of reinforcement learning and graph neural network, after the task is completed, the resource allocation strategy optimized by the reinforcement learning model is applied to the scheduling of subsequent tasks, thereby optimizing the allocation efficiency of storage and computing resources in the cloud computing environment.

[0096] In this implementation, S1 includes the following sub-steps:

[0097] S11. Collect each task T i The computational demand C(T i ), defined as task T i The total amount of computing resources required during the entire execution process;

[0098] S12. Collect each task T i Storage requirement S(T i ), defined as task T i The total amount of storage space required during execution;

[0099] S13, build the dependency matrix D between tasks, each task T i With task T j The dependence D(T i , T j ) is defined as:

[0100]

[0101] The dependencies between all tasks form a dependency matrix D, with a dimension of N×N, where N is the total number of tasks;

[0102] S14, collection task priority P(T i ), the priority of the task P(T i ) is the task urgency U(T i ) and task dependency D(Ti , T j ) and combined with resource allocation and task scheduling to calculate:

[0103]

[0104] Among them, U(T i ) is task T i The urgency of the task indicates the time sensitivity of the task, D(T i , T j ) is task T i With task T j The degree of dependency between them indicates the strength of the dependency relationship, W j For task T j The current weight of the task represents the overall resource requirements and its relative importance, R c (T i ) is task T i The amount of computing resources currently available, R s (T i ) is task T i The amount of storage resources currently available, C(T i ) is task T i The computational demand, S(T i ) is task T i The storage demand of , α1, β1 and γ1 are weight coefficients used to balance urgency, dependency and resource utilization, ∈ is a small constant to avoid the denominator being zero;

[0105] S15. Collect resource status information related to the task, including available computing resources Rc and available storage resources Rc in the cloud computing environment. s ;

[0106] S16. Build a cloud computing basic information set through S11-S15:

[0107] I={C(T i ), S(T i ), D(T i , T j ), P(T i ), R c , R s}.

[0108] In this implementation, S2 includes the following sub-steps:

[0109] S21, each task T in the cloud computing task j Represented as task node v j , the computing resource nodes and storage resource nodes in the cloud computing environment are represented as computing nodes and storage nodes Build a graph structure of tasks and resources, where:

[0110] Task Node v i Represents task T i and its corresponding computing and storage requirements;

[0111] Compute Node Represents the kth computing resource node, including the amount of available computing resources

[0112] Storage Node Indicates the first storage resource node, including the amount of available storage resources

[0113] S22, based on the dependency relationship between tasks D(T i , T j ) at task node v i and task node v j Create an edge between ij , and define the edge weight w ij Indicates the strength of the dependency between tasks. The dependency weight is defined as follows:

[0114] w ij =D(T i , T j );

[0115] Among them, D(T i , T j ) represents task T i With task T j The dependency between them constitutes the dependency edge set E between task nodes. T ;

[0116] S23, at task node v i and compute nodes Create an edge between Edge Weight Reflection Task T i The intensity of computing resource requirements:

[0117]

[0118] Among them, C(T i ) is task T i The computational requirements For computing nodes The amount of available computing resources constitutes the edge set E between the task node and the computing node C ;

[0119] S24, at task node vi and storage nodes Create an edge between Its edge weight Reflection Task T i The intensity of demand for storage resources:

[0120]

[0121] Among them, S(T i ) is task T i storage requirements, For storage nodes The amount of available storage resources constitutes the edge set E between the task node and the storage node S ;

[0122] S25. Construct a graph neural network model including task nodes, computing nodes, storage nodes and the edge information between them:

[0123] G=(V,E T ∪E C ∪E S ):

[0124] in, is a node set, including the task node v i , compute nodes and storage nodes E T ∪E C ∪E s is an edge set, including the dependency edge set E between task nodes T , the edge set E between the task node and the computing node C and the edge set E between the task node and the storage node s .

[0125] In this implementation, S3 includes the following sub-steps:

[0126] S31. Use the cloud computing basic information set to initialize the graph neural network model and construct the graph structure G = (V, E T ∪E C ∪E S ) is input into the graph neural network;

[0127] S32. Initialize the feature vector of each node in the graph structure and set the initial feature vector For task node v i , the initial eigenvector is:

[0128]

[0129] For compute nodes Initial eigenvector:

[0130]

[0131] For storage nodes Initial eigenvector:

[0132]

[0133] S33. In the graph neural network model, the message passing mechanism is used to iteratively update the node feature vector:

[0134]

[0135] in, is the feature vector of node v after the first iteration, N(v) is the set of neighbor nodes of node v, φ(·) is the message function, combining the feature vectors of node v and its neighbor node u and the edge weight w vu Calculate the message, w vu is the edge weight between node v and node u, reflecting the strength of the relationship between nodes, and σ(·) is the activation function;

[0136] S34, repeat step S33 until the number of iterations reaches the preset number of layers L or the feature vector converges, and finally obtain the embedding vector of each node

[0137] S35, obtain the final embedding vectors of the task node and the resource node, the embedding vector of the task node Combined with Task T i The resource requirements, task priorities and dependencies between tasks are calculated, and the embedding vectors of the computing nodes are Reflects the computing resource nodes The current available computing resource status and its association with the task node, and the embedding vector of the storage node Reflects the storage resource node The current status of available storage resources and their association with task nodes.

[0138] In this implementation, S4 includes the following sub-steps:

[0139] S41, the historical task execution data H(T i ) and current resource usage U(R c , R s ) is used as the input of the reinforcement learning model, and the historical task execution data H(T i ) includes task T i The historical execution time, resource utilization rate and task execution success rate of the task, and the current resource usage U(R c , Rs ) including the real-time availability of computing resources and storage resources;

[0140] S42. Define the state s in the reinforcement learning model based on the input historical task execution data and current resource usage t ;

[0141] S43, embedding vector based on state sx and task node The reinforcement learning model generates action a for task resource allocation t :

[0142]

[0143] Among them, Q(s t , a; θ t ) indicates that the current policy function is in state s t and action a t The expected reward value under t is the current weight of the graph neural network model, w ik Represents the task node v i and its neighbor node v k The edge weight between i ) represents the task node v i The set of neighbor nodes of It is an evaluation function for matching computing resource requirements with available resources, used to measure the task T i On the current computing node The resource allocation efficiency on the ,λ1 and λ2 are the trade-off coefficients;

[0144] S44. Evaluate each resource allocation result through the reward mechanism of the reinforcement learning model and define the reward function R(s t , a t ), the reward function is evaluated based on task execution time, resource utilization, and execution success rate:

[0145]

[0146] Among them, T exec (T i ) represents task T i The actual execution time, R c (T i ) and R s (T i ) represent tasks T i The computing resources and storage resources allocated, C(T i ) and S(T i ) are respectively tasks T iThe computing demand and storage demand of the task are as follows: α2, α3, and α4 are the weight coefficients for evaluating the task execution time, resource utilization, and execution success rate, respectively;

[0147] S45, according to the reward function R(s t , a t ), the reinforcement learning model dynamically adjusts the parameters θ in the graph neural network model, and the optimized policy function Gradually improve the efficiency of resource allocation for tasks:

[0148]

[0149] Among them, η1 is the learning rate, Q(s t , a t θ t ) is the expected reward of the policy function, w ij is the dependency weight between task nodes, which is used to balance the impact between tasks during the update process. is the policy function for the model parameters θ t The gradient of , which measures the impact of the current strategy on the parameters;

[0150] S46, optimize the task resource allocation strategy through multiple iterations until the reward function R(s t , a t ) converges, the reinforcement learning model gradually optimizes the parameters in the graph neural network according to historical data and current resource usage to achieve the optimal match between tasks and resources.

[0151] In this implementation, S6 includes the following sub-steps:

[0152] S61, based on task priority P(T i ) and resource allocation decisions, and calculate each task node v in real time i Required computing resources R c (T i ) and storage resources R s (T i ), and allocate corresponding resources to task nodes through scheduling strategies;

[0153] S62, according to the task priority P(T i ) for all task nodes v i Sorting: Task nodes with high priority will be given priority to obtain computing resources and storage resources, and will be allocated execution time based on the priority scheduling algorithm. The task scheduling order is:

[0154]

[0155] Among them, O(T) represents the task scheduling order, P(T i) is task T i The priority of T exec (T i ) is task T i The estimated execution time of tasks with high priority and short execution time is guaranteed to be executed first;

[0156] S63. Based on the task scheduling sequence O(T) and resource allocation results, reasonably arrange the task execution time T start (T i ) and T end (T i ), during the task execution process, monitor the execution progress and resource utilization of the task nodes in real time. If bottlenecks or sudden changes in computing resources or storage resources are detected during the task execution process, dynamically adjust the task execution order and resource allocation strategy;

[0157] S64: After all tasks are completed, output the task execution log, including the start time T of each task. start (T i ), end time T end (T i ), the amount of computing resources and storage resources used, as well as the success rate of task execution and resource utilization.

[0158] Embodiment 1:

[0159] In order to verify the feasibility and effectiveness of the present invention in the intelligent scheduling of storage and computing resources for cloud computing tasks, the experimenters selected two publicly available data sets: Google Cluster Data and Alibaba Cluster Trace for experiments. Google Cluster Data contains the large-scale cluster resource scheduling information running in the production environment of Google, with a time span from May 2011 to November 2011. Alibaba Cluster Trace is the task scheduling data running in the production environment of Alibaba, covering 8 days of task records in 2017. The two data sets represent cloud computing task scheduling scenarios under different scales and load conditions.

[0160] The experimenters first divided the two data sets into a training set and a test set in a ratio of 8:2. The training set was used for model training, and the test set was used to verify the generalization ability of the model. During the task scheduling process, the experimenters collected the computing requirements, storage requirements, task priorities and dependencies between tasks of each task, and recorded the real-time availability information of resource nodes as the input of the adaptive graph neural network model of the present invention.

[0161] For feature extraction of task nodes, the experimenters used the ReLU activation function to perform nonlinear transformation on the computing and storage requirements of the task, and the task priority was mapped to between 0 and 1 through normalization. The embedding vector of each task node is composed of its computing requirements, storage requirements, and priority. For the features of resource nodes, computing resources and storage resources are directly input into the graph neural network for embedding. The parameters of the graph neural network model are optimized by the Adam optimizer, the learning rate is set to 0.001, and the size of the small batch training is set to 64.

[0162] In order to evaluate the scheduling effect of the present invention, the experimenters selected the average execution time of the task, computing resource utilization, storage resource utilization and task failure rate as evaluation indicators. During the experiment, the experimenters conducted a detailed comparative analysis of the method of the present invention and the traditional static scheduling algorithm.

[0163] The experimental results on the GoogleClusterData dataset are shown in Table 1:

[0164] Table 1 Experimental results on the Google Cluster Data dataset

[0165]

[0166] The experimental results on the AlibabaClusterTrace dataset are shown in Table 2:

[0167] Table 2 Experimental results on the AlibabaClusterTrace dataset

[0168]

[0169] From the experimental results in Table 1 and Table 2, it can be seen that the intelligent scheduling method of the adaptive graph neural network combined with reinforcement learning of the present invention performs better than the traditional static scheduling method on different data sets. In the Google Cluster Data data set, the method of the present invention reduces the average task execution time by 20.3%, increases the computing resource utilization by 15%, and increases the storage resource utilization by 13%. In the Alibaba Cluster Trace data set, the method of the present invention reduces the average task execution time by 22%, increases the computing resource utilization by 15%, and increases the storage resource utilization by 13%. In addition, the task failure rate is greatly reduced on both data sets, by 3.3% and 3.6% respectively.

[0170] In the GoogleClusterData dataset, the experimenters selected a complex task T1, whose computing requirement is 50 CPU cores, storage requirement is 300 MB, and task priority is 0.8. Task T1 is dependent on task T2 and must wait for T2 to complete before execution. When using the traditional static scheduling method, the system pre-allocates fixed resources for task T1, resulting in a long waiting time for task T1 and low utilization of computing resources and storage resources. In the end, the actual execution time of task T1 is 15 seconds, and the task failure rate is 4%.

[0171] When using the method of the present invention, the system models the dependencies between tasks through an adaptive graph neural network model, and dynamically adjusts the resource allocation strategy according to the current resource availability and task priority. After task T2 is completed, the system quickly allocates sufficient computing resources and storage resources to T1, avoiding long-term resource idleness and waiting. In the end, the actual execution time of task T1 is 10 seconds, and the task failure rate is reduced to 1.2%.

[0172] Similarly, in the test of AlibabaClusterTrace data set, the computing requirement of a high priority task T3 is 60 CPU cores and the storage requirement is 500 MB. Under the traditional method, because the system does not consider the priority of the task, the resource allocation is relatively random, resulting in the execution time of T3 being as long as 16 seconds. However, the method system of the present invention optimizes the task scheduling strategy through the reinforcement learning model, ensures that high priority tasks are allocated resources first, and the execution time of task T3 is shortened to 11 seconds, and the resource utilization rate is greatly improved.

[0173] Through the above experiments, the scheduling method of the present invention has demonstrated good performance in task scenarios with high load and complex dependencies. The scheduling strategy can dynamically adapt to changes in task requirements and resource status, greatly improving the overall resource utilization efficiency of the system and significantly reducing task execution delays and failure rates, verifying the feasibility and effectiveness of the present invention.

[0174] The present invention adopts an adaptive graph neural network to model tasks and resource nodes, generates embedded vectors of tasks and resources by real-time collection of task computing requirements, storage requirements and dependencies between tasks, and combines with a reinforcement learning model to dynamically adjust the parameters in the graph neural network according to historical data of task execution and resource utilization. When task requirements or resource status change, the system can update the resource allocation strategy in real time and make adjustments quickly to avoid resource waste and scheduling delay problems in traditional static scheduling methods. Experimental results show that compared with traditional static scheduling algorithms, the method of the present invention reduces scheduling delays by about 12.5% ​​and improves resource utilization by about 15% in a large-scale cloud computing environment.

[0175] The present invention can capture the complex dependencies between task nodes and generate embedded vectors of task nodes through a message passing mechanism, reflecting the dependency strength and resource requirements between tasks. During the scheduling process, the present invention dynamically optimizes the task execution order based on task priorities and task dependencies, ensuring that tasks with high priority and short execution time obtain resources first to avoid resource competition and scheduling conflicts. Compared with traditional methods, the task execution order is more reasonable, the task execution time is reduced by about 8%, and the task conflict rate is reduced by about 10%.

[0176] The present invention realizes dynamic optimization of graph neural network parameters through a reinforcement learning model combined with historical task execution data and current resource usage. After each resource allocation, the system evaluates the effect of the allocation strategy through the reward mechanism of the reinforcement learning model and adjusts the scheduling strategy. Through multiple iterative optimizations, the present invention can effectively improve the efficiency of resource allocation so that resources can quickly adapt to dynamically changing task requirements, avoiding the limitation of traditional scheduling methods that cannot optimize scheduling strategies based on historical execution data. Experimental data show that the method of the present invention increases the resource utilization of the system by about 18%, and the task execution success rate by about 9%.

[0177] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network, characterized in that: The steps include: S1. Collect computing requirements, storage requirements, task dependencies, task priorities, and task-related resource status information of multiple tasks in a cloud computing environment to build a cloud computing basic information set; S2. Based on the task computing requirements, storage requirements and task dependencies, the task nodes are modeled with the computing nodes and storage nodes in the cloud computing environment. The task nodes and resource nodes are represented as nodes in the graph structure. The dependencies between tasks and the resource status of resource nodes are used as the edge information in the graph to build a graph neural network model of tasks and resources. S3. Initialize the graph neural network model using the cloud computing basic information set, embed the task nodes and resource nodes in the graph structure, generate the embedding vectors of the task nodes and the embedding vectors of the resource nodes, and the embedding vectors reflect the task resource requirements and the dependencies between tasks and the current availability status of resources; S4. The historical task execution data and current resource usage in the cloud computing basic information set are used as the input of the reinforcement learning model. Based on the reward mechanism of the reinforcement learning model, the parameters in the graph neural network model are dynamically adjusted according to the historical execution results of the task and the resource utilization rate. The reinforcement learning model evaluates the effect of each resource allocation through the reward function and gradually optimizes the resource allocation strategy of the task. S5. Update the embedding vector of the task node based on the optimized graph neural network model and generate new resource allocation decisions so that the task can be executed under the optimal resource conditions. S6. Schedule computing resources and storage resources in real time according to the task priority and the generated resource allocation decision, allocate the required computing resources and storage resources to the task nodes, and reasonably arrange the execution time of the tasks according to the execution order of the tasks; S7. During the task execution process, the cloud computing resource status information and task execution progress are monitored in real time. If changes are detected in task requirements or system resource status, the parameters of the graph neural network model are readjusted to dynamically update the embedding vectors of the task nodes and resource allocation decisions. S8. By continuously applying the joint optimization strategy of reinforcement learning and graph neural network, after the task is completed, the resource allocation strategy optimized by the reinforcement learning model is applied to the scheduling of subsequent tasks, thereby optimizing the allocation efficiency of storage and computing resources in the cloud computing environment.

2. According to claim 1, a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network is characterized in that: The S1 comprises the following sub-steps: S11. Collect each task T i The computational demand C(T i ), defined as task T i The total amount of computing resources required during the entire execution process; S12. Collect each task T i Storage requirement S(T i ), defined as task T i The total amount of storage space required during execution; S13, build the dependency matrix D between tasks, each task T i With task T j The dependence D(T i , T j ) is defined as: The dependencies between all tasks form a dependency matrix D, with a dimension of N×N, where N is the total number of tasks; S14, collection task priority P(T i ), the priority of the task P(T i ) is the task urgency U(T i ) and task dependency D(T i , T j ) and combined with resource allocation and task scheduling to calculate: Among them, U(T i ) is task T i The urgency of the task indicates the time sensitivity of the task, D(T i , T j ) is task T i With task T j The degree of dependency between them indicates the strength of the dependency relationship, W j For task T j The current weight of the task represents the overall resource requirements and its relative importance, R c (T i ) is task T i The amount of computing resources currently available, R s (T i ) is task T i The amount of storage resources currently available, C(T i ) is task T i The computational demand, S(T i ) is task T i The storage demand of , α1, β1 and γ1 are weight coefficients used to balance urgency, dependency and resource utilization, ∈ is a small constant to avoid the denominator being zero; S15. Collect resource status information related to the task, including available computing resources Rc and available storage resources Rc in the cloud computing environment. s ; S16. Build a cloud computing basic information set through S11-S15: I={C(T i ),S(T i ),D(T i ,T j ),P(T i ),R c ,R s }。 3. According to claim 1, a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network is characterized in that: The S2 comprises the following sub-steps: S21, each task T in the cloud computing task j Represented as task node v j , the computing resource nodes and storage resource nodes in the cloud computing environment are represented as computing nodes and storage nodes Build a graph structure of tasks and resources, where: Task Node v i Represents task T i and its corresponding computing and storage requirements; Compute Node Represents the kth computing resource node, including the amount of available computing resources Storage Node Indicates the first storage resource node, including the amount of available storage resources S22, based on the dependency relationship between tasks D(T i , T j ) at task node v i and task node v j Create an edge between ij , and define the edge weight w ij It represents the strength of the dependency between tasks. The dependency weight is defined as follows: w ij =D(T i ,T j ); Among them, D(T i , T j ) represents the task Ti and task T j The dependency between them constitutes the dependency edge set E between task nodes. T ; S23, at task node v i and compute nodes Create an edge between Edge Weight Reflects the intensity of task Ti's demand for computing resources: Among them, C(T i ) is task T i The computational requirements For computing nodes The amount of available computing resources constitutes the edge set E between the task node and the computing node C ; S24, at task node v i and storage nodes Create an edge between Its edge weight Reflection Task T i The intensity of demand for storage resources: Among them, S(T i ) is task T i storage requirements, For storage nodes The amount of available storage resources constitutes the edge set E between the task node and the storage node S ; S25. Construct a graph neural network model including task nodes, computing nodes, storage nodes and the edge information between them: G=(V,E T ∪E C ∪E S ): in, is a node set, including the task node v i , compute nodes and storage nodes E T ∪E C ∪E S is an edge set, including the dependency edge set E between task nodes T , the edge set E between the task node and the computing node C and the edge set E between the task node and the storage node S .

4. According to claim 1, a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network is characterized in that: The S3 comprises the following sub-steps: S31. Use the cloud computing basic information set to initialize the graph neural network model and construct the graph structure G = (V, E T ∪E C ∪E S ) is input into the graph neural network; S32. Initialize the feature vector of each node in the graph structure and set the initial feature vector For task node v i , the initial eigenvector is: For compute nodes Initial eigenvector: For storage nodes Initial eigenvector: S33. In the graph neural network model, the message passing mechanism is used to iteratively update the node feature vector: in, is the feature vector of node v after the first iteration, N(v) is the set of neighbor nodes of node v, φ(·) is the message function, combining the feature vectors of node v and its neighbor node u and the edge weight w vu Calculate the message, w vu is the edge weight between node v and node u, reflecting the strength of the relationship between nodes, and σ(·) is the activation function; S34, repeat step S33 until the number of iterations reaches the preset number of layers L or the feature vector converges, and finally obtain the embedding vector of each node S35, obtain the final embedding vectors of the task node and the resource node, the embedding vector of the task node Combined with Task T i The resource requirements, task priorities, and dependencies between tasks are calculated to embed the node vectors Reflects the computing resource nodes The current available computing resource status and its association with the task node, and the embedding vector of the storage node Reflects the storage resource node The current status of available storage resources and their association with task nodes.

5. According to claim 1, a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network is characterized in that: The S4 comprises the following sub-steps: S41, the historical task execution data H(T i ) and current resource usage U(R c , R s ) is used as the input of the reinforcement learning model, and the historical task execution data H(T i ) includes task T i The historical execution time, resource utilization rate and task execution success rate of the task, and the current resource usage U(R c , R s ) including the real-time availability of computing resources and storage resources; S42. Define the state s in the reinforcement learning model based on the input historical task execution data and current resource usage t ; S43, based on state s t and the embedding vector of the task node The reinforcement learning model generates action a for task resource allocation t : Among them, Q(s t , a; θ t ) indicates that the current policy function is in state s t and action a t The expected reward value under t is the current weight of the graph neural network model, w ik Represents the task node v i and its neighbor node v k The edge weight between i ) represents the task node v i The set of neighbor nodes of It is an evaluation function for matching computing resource requirements with available resources, used to measure the task T i On the current computing node The resource allocation efficiency on the ,λ1 and λ2 are the trade-off coefficients; S44. Evaluate each resource allocation result through the reward mechanism of the reinforcement learning model and define the reward function R(s t , a t ), the reward function is evaluated based on task execution time, resource utilization, and execution success rate: Among them, T exec (T i ) represents task T i The actual execution time, R c (T i ) and R s (T i ) represent tasks T i The computing resources and storage resources allocated, C(T i ) and S(T i ) are respectively tasks T i The computing demand and storage demand of the task are as follows: α2, α3, and α4 are the weight coefficients for evaluating the task execution time, resource utilization, and execution success rate, respectively; S45, according to the reward function R(s t , a t ), the reinforcement learning model dynamically adjusts the parameters θ in the graph neural network model, and the optimized policy function Gradually improve the efficiency of resource allocation for tasks: Among them, η1 is the learning rate, Q(s t , a t θ t ) is the expected reward of the policy function, w ij is the dependency weight between task nodes, which is used to balance the impact between tasks during the update process. is the policy function for the model parameters θ t The gradient of , which measures the impact of the current strategy on the parameters; S46, optimize the task resource allocation strategy through multiple iterations until the reward function R(s t , a t ) converges, the reinforcement learning model gradually optimizes the parameters in the graph neural network according to historical data and current resource usage to achieve the optimal match between tasks and resources.

6. According to claim 1, a method for intelligent scheduling of storage and computing resources for cloud computing tasks based on adaptive graph neural network is characterized in that: The S6 comprises the following sub-steps: S61, based on task priority P(T i ) and resource allocation decisions, and calculate each task node v in real time i Required computing resources R c (T i ) and storage resources R s (T i ), and allocate corresponding resources to task nodes through scheduling strategies; S62, according to the task priority P(T i ) for all task nodes v i Sorting: Task nodes with high priority will have priority in obtaining computing resources and storage resources, and will be allocated execution time based on the priority scheduling algorithm. The task scheduling order is: Among them, O(T) represents the task scheduling order, P(T i ) is task T i The priority of T exec (T i ) is task T i The estimated execution time of tasks with high priority and short execution time is guaranteed to be executed first; S63. Based on the task scheduling sequence O(T) and resource allocation results, reasonably arrange the task execution time T start (T i ) and T end (T i ), during the task execution process, monitor the execution progress and resource utilization of the task nodes in real time. If bottlenecks or sudden changes in computing resources or storage resources are detected during the task execution process, dynamically adjust the task execution order and resource allocation strategy; S64: After all tasks are completed, output the task execution log, including the start time T of each task. start (T i ), end time T end (T i ), the amount of computing resources and storage resources used, as well as the success rate of task execution and resource utilization.

Images