Cloud computing storage, calculation and training resource allocation method based on reinforcement learning and dynamic resource scheduling

Abstract

The invention discloses a cloud computing storage, calculation and training resource allocation method based on reinforcement learning and dynamic resource scheduling. The method comprises the following steps: S1, establishing a unified resource scheduling framework; s2, constructing a state space and an action space; s3, generating a state representation for predicting future resource demands; s4, initializing a reward function of key performance index weighted comprehensive evaluation; s5, generating an initial storage resource, computing resource and training resource allocation strategy; s6, continuously adjusting a reward function of the reinforcement learning model based on the self-supervised learning feedback signal; s7, dynamically adjusting a distribution scheme of the storage resources, the computing resources and the training resources through quick response of the resource scheduling framework; s8, optimizing resource allocation among a plurality of cloud computing nodes or platforms; s9, automatically identifying different stages of the current task through a self-supervised learning model; and S10, finally generating an optimal allocation scheme. According to the staged resource scheduling method, the condition of overload or insufficiency of resources is effectively avoided, and the task execution time is greatly shortened.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a cloud computing storage, computing and training resource allocation method based on reinforcement learning and dynamic resource scheduling. Background Art

[0002] With the rapid development of cloud computing, efficient allocation of storage, computing and training resources in cloud computing platforms has become the key. However, existing resource allocation technologies have significant defects when facing resource supply and demand imbalance, multi-task execution and dynamic demand changes. Traditional resource allocation methods are usually based on preset static allocation strategies or simple load balancing algorithms. In complex cloud computing environments, traditional methods cannot meet the diverse needs of tasks.

[0003] In the prior art, cloud computing resource allocation mainly adopts a centralized scheduling framework or a fixed allocation model based on historical data. The fixed allocation model formulates a resource allocation plan by analyzing historical workloads, but it cannot respond to sudden changes in system load in a timely manner. When there is a peak period of computing tasks or a sudden demand for training tasks, it is difficult for traditional models to quickly adjust resource configuration. In addition, the efficiency of collaborative allocation of multiple resources is low. Computing resources, storage resources, and training resources are often managed separately in different system modules, resulting in the system being unable to flexibly allocate resources according to task execution.

[0004] In order to improve the efficiency of resource allocation, some technical solutions have introduced rule-based dynamic load balancing strategies. However, dynamic load balancing strategies usually rely on fixed sets of rules and are difficult to adapt to changing resource requirements and diverse task scenarios. In computing-intensive and training optimization tasks, traditional methods find it difficult to flexibly schedule resources according to task stages. The lack of in-depth analysis of the task execution stage makes resource allocation in different task stages insufficiently adaptable, resulting in reduced resource utilization or system bottlenecks.

[0005] In summary, existing technologies have obvious shortcomings in dynamic resource scheduling, multi-resource collaboration, task stage identification and real-time feedback. As the complexity of cloud computing scenarios continues to increase, how to efficiently manage storage, computing and training resources and dynamically adjust resource allocation strategies according to the task stage are urgent issues to be addressed. Summary of the invention

[0006] One purpose of the present invention is to propose a cloud computing storage, computing and training resource allocation method based on reinforcement learning and dynamic resource scheduling. The phased resource scheduling method of the present invention effectively avoids resource overload or shortage and greatly shortens the task execution time.

[0007] A method for allocating cloud computing storage, computing and training resources based on reinforcement learning and dynamic resource scheduling according to an embodiment of the present invention includes the following steps:

[0008] S1. Build a multi-resource scheduling environment for the cloud computing platform, which includes storage resources, computing resources, and training resources, and establish a unified resource scheduling framework;

[0009] S2. Define an agent in the cloud computing platform, which is responsible for executing the self-supervised reinforcement learning strategy, and collects the workload, task execution and resource utilization data in the current cloud computing platform by monitoring the usage of storage resources, computing resources and training resources in the current system, and constructs the state space and action space;

[0010] S3, the agent uses the self-supervised learning model to self-train the collected historical resource usage data, automatically mines resource usage patterns and generates pseudo labels. The pseudo labels are used to represent the resource utilization rate, task completion time and system load balancing in the historical data, and generate a state representation to predict future resource demand;

[0011] S4. Based on the pseudo labels generated by self-supervised learning, the agent initializes the state space of the reinforcement learning model. The state space includes the current usage of storage resources, computing resources, and training resources. The action space includes the allocation strategy of storage resources, computing resources, and training resources. The reward function of the weighted comprehensive evaluation of key performance indicators is initialized.

[0012] S5, the agent generates the initial storage resource, computing resource and training resource allocation strategy by executing the reinforcement learning model, and allocates and adjusts resources in real time in the cloud computing platform through the resource scheduling framework to meet the current task requirements;

[0013] S6. During the resource allocation process, the agent provides real-time feedback on the system's resource utilization efficiency, task execution progress, and system load based on the resource scheduling framework, and continuously adjusts the reward function of the reinforcement learning model based on the self-supervised learning feedback signal;

[0014] S7. When the resource demand in the cloud computing platform changes dynamically, the agent quickly responds to dynamically adjust the allocation of storage resources, computing resources, and training resources through the resource scheduling framework based on the resource demand predicted by the self-supervised learning model;

[0015] S8, the agent optimizes resource allocation among multiple cloud computing nodes or platforms through a collaborative scheduling mechanism across multiple nodes and platforms, based on a cross-node resource scheduling pattern generated by self-supervised learning;

[0016] S9. The agent identifies the execution stage of each task and automatically identifies the different stages of the current task through the self-supervised learning model, including the task initialization stage, the computationally intensive stage, and the training optimization stage. It also dynamically adjusts the resource allocation strategy according to the resource requirements of different stages, so that each task can obtain the best resource allocation at different stages.

[0017] S10. Finally, the optimal storage resource, computing resource and training resource allocation plan is generated.

[0018] Optionally, the S1 includes the following steps:

[0019] S11. Define multiple resource types in the cloud computing platform, including computing resources C r , storage resources S r and training resources T r ;

[0020] S12. Establish resource status matrix:

[0021]

[0022] in, is the resource status matrix of the ith node at time t, Respectively represent the real-time usage of computing resources, storage resources, and training resources on node i;

[0023] S13, construct a resource utilization function U for evaluating the current utilization of each type of resource r (t):

[0024]

[0025] in, represents the actual usage of resource r on node i at time t, is the maximum available capacity of the corresponding resource on node i;

[0026] S14. Build a unified resource scheduling framework and define resource scheduling decision variables x i (t), where x i (t)∈{C r ,S r ,T r} represents the resource allocation plan at time t;

[0027] S15. Establish constraints for the resource scheduling framework, including resource usage constraints:

[0028]

[0029] Resource supply and demand balance constraints:

[0030]

[0031] Resource conflict avoidance constraints:

[0032]

[0033] S16. Initialize the state space S in the resource scheduling framework based on the resource state matrix and resource utilization function t and action space A t :

[0034]

[0035] A t ={a ij (t)|a ij (t)∈{C r ,S r ,T r}};

[0036] Where: S t Contains the resource status of all nodes and the workload of tasks, A t Indicates the resource allocation actions of each node.

[0037] S17. Define the weighted comprehensive evaluation function of key performance indicators:

[0038]

[0039] Among them, α r is the weight coefficient of resource r, satisfying ∑α r =1, W(t) is used to evaluate the utilization and scheduling efficiency of each resource;

[0040] S18. Build a cross-node collaborative scheduling model and define the cross-node resource allocation collaborative formula:

[0041]

[0042] in, It represents the ratio of resources allocated by node i to task j at time t. The denominator represents the total resources and total demand of all nodes and tasks.

[0043] Optionally, S2 includes the following steps:

[0044] S21. Define Agent A i , the agent is responsible for executing the self-supervised reinforcement learning strategy on node i and monitoring the storage resources of node i Computing resources and training resources usage of

[0045] S22. Define the local state space for the agent The structure is:

[0046]

[0047] in, Represents the state space of the local resources of the agent on node i, including the usage of computing resources, storage resources, and training resources of the current node, as well as the real-time workload of the task

[0048] S23. Define the local action space of the agent for:

[0049]

[0050] in, Describes the agent's resource allocation actions within node i, which is used for local resource management of each node;

[0051] S24. Construct a local reward function for each agent Used to measure the resource allocation efficiency of nodes:

[0052]

[0053] Among them, W i (t) is the resource utilization weighted function of node i, is the ideal resource utilization of node i, λ1 is the penalty coefficient;

[0054] S25, the agent in each node through the local state space and action space Execute self-supervised reinforcement learning strategies to adjust resource allocation plans within nodes.

[0055] Optionally, S3 includes the following steps:

[0056] S31, Agent A i Collect historical resource usage data on node i and build a historical resource usage matrix

[0057] S32, using time series self-supervision model based on historical resource matrix Extract resource usage patterns and generate resource usage pattern vectors The resource usage pattern vector represents the change trend of different resources on node i in the historical period:

[0058]

[0059] Where σ(·) is the activation function used to normalize the resource pattern vector, W r and X r is the model weight matrix, b r is the bias term, and tanh(·) is used to capture the nonlinear relationship in resource usage patterns;

[0060] S33, based on resource usage pattern vector The agent generates a pseudo-label matrix that reflects historical resource utilization and its impact on system performance Indicates resource utilization, task completion time, and system load balance in each time period;

[0061] S34, Agent based on pseudo-label matrix Use multivariate time series forecasting models to capture the continuation of historical resource usage patterns in the future and predict future resource demand:

[0062]

[0063] in, is the resource demand forecast after Δ time in the future, and α is the attenuation coefficient, which is used to reduce the impact of long-term data;

[0064] S35. Future resource requirements based on forecasts Intelligent generation of future state representation

[0065]

[0066] S36, the pseudo label matrix and future state representation Integrate into the reinforcement learning model to update the resource allocation plan:

[0067]

[0068] Among them, W(t) is the key performance indicator weighting function, λ2 is the penalty coefficient, Ideal resource utilization.

[0069] Optionally, S5 includes the following steps:

[0070] S51. The agent generates a resource allocation strategy matrix based on the reinforcement learning model in the initial stage The resource allocation strategy matrix represents the resource allocation plan of node i at time t:

[0071]

[0072] in, and They represent the proportion of computing resources, storage resources, and training resources allocated to node i at time t respectively;

[0073] S52, according to the current state space and action space The agent chooses the optimal action a at each time step t ij (t):

[0074]

[0075] in, The reward value for executing action a is used to guide the agent to choose a resource allocation plan;

[0076] S53, the agent adjusts the resource allocation of each node in real time and updates the resource allocation matrix by executing the reinforcement learning strategy To meet current mission requirements:

[0077]

[0078] Where η is the learning rate, represents the gradient of the reward function with respect to the current allocation scheme;

[0079] S54, the resource scheduling framework allocates resources on the cloud computing platform according to the resource allocation strategy generated by the intelligent agent, so that the nodes are in a resource coordination state:

[0080]

[0081] in, represents the proportion of computing resources allocated by node i to task j, is the sum of computing resources of all nodes at time t.

[0082] Optionally, the S6 includes the following steps:

[0083] S61. During the resource scheduling process, the agent collects the resource utilization efficiency of the system in real time Task execution progress P j (t) and system load state B i (t);

[0084] S62. Based on the collected real-time data, the intelligent agent generates feedback signals Used to describe the deviation of current resource usage from system performance:

[0085]

[0086] in, is the ideal resource efficiency on node i, γ is the penalty coefficient for task delay;

[0087] S63, the agent responds to the feedback signal Adjusting the Reward Function of a Reinforcement Learning Model Optimize resource allocation strategy:

[0088]

[0089] Among them, W(t) is the weighted comprehensive evaluation function of the key performance indicators, and λ3 is the feedback penalty coefficient, which is used to dynamically adjust the reward function;

[0090] S64, the agent recalculates the resource allocation strategy matrix at each time step t And update resource allocation in real time through the resource scheduling framework:

[0091]

[0092] Among them, η1 is the learning rate, is the gradient of the reward function, which is used to guide the optimization of the resource allocation strategy.

[0093] Optionally, the S9 includes the following steps:

[0094] S91. During the task execution process, the agent analyzes the real-time data of the task based on the self-supervised learning model. Construct a task stage classification function to identify the different stages of the task:

[0095]

[0096] Among them, f stage (t) is the prediction function of the task stage at time t, which is used to maximize the matching probability of the current stage. is the workload weight vector of task j at time t, e -α(τ-t) is the time decay factor, α controls the impact of distant period data on the current stage recognition, and the sum of squares in the denominator represents the volatility of workload data. is the average workload of the task in the time period, ∈ is a constant to prevent the denominator from being zero;

[0097] S92, the agent classifies according to the classification function f stage (t) The output recognition task is currently in the stage And generate the corresponding resource demand weight matrix

[0098] S93, based on the resource demand weight matrix of the current task stage, the agent updates the resource allocation strategy matrix To meet the needs of the current stage:

[0099]

[0100] in, represents the resource allocation strategy matrix of the current stage, ⊙ is the symbol of matrix multiplication, and η is the learning rate;

[0101] S94. When the task changes from one stage Transition to the next stage When the agent updates the stage prediction in real time and generates a new resource demand weight matrix

[0102] S95, the agent dynamically monitors the execution status of tasks at each stage through the resource scheduling framework to meet the resource requirements of each stage, while maintaining system load balance during resource transition;

[0103] S96. The intelligent agent dynamically adjusts the resource demand weights and reward functions based on the execution feedback of each stage and the self-supervised learning model to achieve efficient task execution and optimal resource utilization.

[0104] The beneficial effects of the present invention are:

[0105] (1) The present invention realizes adaptive optimization in the resource allocation process by combining self-supervised learning with reinforcement learning. The intelligent agent uses the self-supervised learning model to extract patterns from historical resource usage data and generate pseudo labels as training data to provide support for accurate prediction of future resource demand. Unlike traditional static allocation strategies, the intelligent agent of the present invention can dynamically adjust the allocation strategy of computing, storage and training resources during task execution, so that the system can respond in real time when facing sudden tasks or load changes, reducing resource allocation lag and waste. The reinforcement learning model continuously optimizes the reward function by introducing a real-time feedback mechanism to ensure that the efficiency of resource use and the stability of task execution are improved, thereby enhancing the intelligent level of resource scheduling.

[0106] (2) During the task execution process, the intelligent agent of the present invention automatically identifies the different stages of the task through a self-supervised learning model, including the initialization stage, the computationally intensive stage, and the training optimization stage. By performing differentiated analysis on the resource requirements of each stage, the system can dynamically generate a resource allocation strategy that matches the task stage, ensuring that each task obtains the required resource support at the critical stage of its life cycle. The traditional system lacks the ability to identify the task stage, resulting in inaccurate resource allocation at different stages of task execution. The phased resource scheduling method of the present invention effectively avoids resource overload or shortage, greatly shortens the task execution time, and significantly improves the load balancing effect of the system.

[0107] (3) In the present invention, the intelligent agent constructs a cross-node collaborative scheduling mechanism based on the resource scheduling framework. By analyzing the resource usage and task requirements of different nodes, the intelligent agent dynamically allocates resources among multiple nodes to optimize the global utilization of resources. The multi-stage resource requirement matrix between tasks is used to make resource scheduling not only limited to the optimization of a single node, but also to achieve efficient collaboration among multiple nodes. Compared with the isolated resource management mode between nodes in traditional methods, the cross-node scheduling model reduces resource competition and delay between nodes and improves the overall performance of the system. It is particularly outstanding in large-scale computing tasks and complex training task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0109] Figure 1 This is a flow chart of a cloud computing storage, computing and training resource allocation method based on reinforcement learning and dynamic resource scheduling proposed by the present invention;

[0110] Figure 2 This is a schematic diagram of the identification and dynamic resource allocation of the task execution phase in a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling proposed by the present invention. DETAILED DESCRIPTION

[0111] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0112] refer to Figure 1-Figure 2 A cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling includes the following steps:

[0113] S1. Build a multi-resource scheduling environment for the cloud computing platform, which includes storage resources, computing resources, and training resources, and establish a unified resource scheduling framework;

[0114] S2. Define an agent in the cloud computing platform, which is responsible for executing the self-supervised reinforcement learning strategy, and collects the workload, task execution and resource utilization data in the current cloud computing platform by monitoring the usage of storage resources, computing resources and training resources in the current system, and constructs the state space and action space;

[0115] S3, the agent uses the self-supervised learning model to self-train the collected historical resource usage data, automatically mines resource usage patterns and generates pseudo labels. The pseudo labels are used to represent the resource utilization rate, task completion time and system load balancing in the historical data, and generate a state representation to predict future resource demand;

[0116] S4. Based on the pseudo labels generated by self-supervised learning, the agent initializes the state space of the reinforcement learning model. The state space includes the current usage of storage resources, computing resources, and training resources. The action space includes the allocation strategy of storage resources, computing resources, and training resources. The reward function of the weighted comprehensive evaluation of key performance indicators is initialized.

[0117] S5, the agent generates the initial storage resource, computing resource and training resource allocation strategy by executing the reinforcement learning model, and allocates and adjusts resources in real time in the cloud computing platform through the resource scheduling framework to meet the current task requirements;

[0118] S6. During the resource allocation process, the agent provides real-time feedback on the system's resource utilization efficiency, task execution progress, and system load based on the resource scheduling framework, and continuously adjusts the reward function of the reinforcement learning model based on the self-supervised learning feedback signal;

[0119] S7. When the resource demand in the cloud computing platform changes dynamically, the agent quickly responds to dynamically adjust the allocation of storage resources, computing resources, and training resources through the resource scheduling framework based on the resource demand predicted by the self-supervised learning model;

[0120] S8, the agent optimizes resource allocation among multiple cloud computing nodes or platforms through a collaborative scheduling mechanism across multiple nodes and platforms, based on a cross-node resource scheduling pattern generated by self-supervised learning;

[0121] S9. The agent identifies the execution stage of each task and automatically identifies the different stages of the current task through the self-supervised learning model, including the task initialization stage, the computationally intensive stage, and the training optimization stage. It also dynamically adjusts the resource allocation strategy according to the resource requirements of different stages, so that each task can obtain the best resource allocation at different stages.

[0122] S10. Finally, the optimal storage resource, computing resource and training resource allocation plan is generated.

[0123] In this implementation, S1 includes the following steps:

[0124] S11. Define multiple resource types in the cloud computing platform, including computing resources C r , storage resources S r and training resources T r ;

[0125] S12. Establish resource status matrix:

[0126]

[0127] in, is the resource status matrix of the ith node at time t, Respectively represent the real-time usage of computing resources, storage resources, and training resources on node i;

[0128] S13, construct a resource utilization function U for evaluating the current utilization of each type of resource r (t):

[0129]

[0130] in, represents the actual usage of resource r on node i at time t, is the maximum available capacity of the corresponding resource on node i;

[0131] S14. Build a unified resource scheduling framework and define resource scheduling decision variables x i (t), where x i (t)∈{C r ,S r ,T r} represents the resource allocation plan at time t;

[0132] S15. Establish constraints for the resource scheduling framework, including resource usage constraints:

[0133]

[0134] Resource supply and demand balance constraints:

[0135]

[0136] Resource conflict avoidance constraints:

[0137]

[0138] S16. Initialize the state space S in the resource scheduling framework based on the resource state matrix and resource utilization function t and action space A t :

[0139]

[0140] A t ={a ij (t)|a ij (t)∈{C r ,S r,T r}};

[0141] Where: S t Contains the resource status of all nodes and the workload of tasks, A t Indicates the resource allocation actions of each node.

[0142] S17. Define the weighted comprehensive evaluation function of key performance indicators:

[0143]

[0144] Among them, α r is the weight coefficient of resource r, satisfying ∑α r =1, W(t) is used to evaluate the utilization and scheduling efficiency of each resource;

[0145] S18. Build a cross-node collaborative scheduling model and define the cross-node resource allocation collaborative formula:

[0146]

[0147] in, It represents the ratio of resources allocated by node i to task j at time t. The denominator represents the total resources and total demand of all nodes and tasks.

[0148] In this implementation, S2 includes the following steps:

[0149] S21. Define Agent A i , the agent is responsible for executing the self-supervised reinforcement learning strategy on node i and monitoring the storage resources of node i Computing resources and training resources usage of

[0150] S22. Define the local state space for the agent The structure is:

[0151]

[0152] in, Represents the state space of the local resources of the agent on node i, including the usage of computing resources, storage resources, and training resources of the current node, as well as the real-time workload of the task

[0153] S23. Define the local action space of the agent for:

[0154]

[0155] in, Describes the agent's resource allocation actions within node i, which is used for local resource management of each node;

[0156] S24. Construct a local reward function for each agent Used to measure the resource allocation efficiency of nodes:

[0157]

[0158] Among them, W i (t) is the resource utilization weighted function of node i, is the ideal resource utilization of node i, λ1 is the penalty coefficient;

[0159] S25, the agent in each node through the local state space and action space Execute self-supervised reinforcement learning strategies to adjust resource allocation plans within nodes.

[0160] In this implementation, S3 includes the following steps:

[0161] S31, Agent A i Collect historical resource usage data on node i and build a historical resource usage matrix

[0162] S32, using time series self-supervision model based on historical resource matrix Extract resource usage patterns and generate resource usage pattern vectors The resource usage pattern vector represents the change trend of different resources on node i in the historical period:

[0163]

[0164] Where σ(·) is the activation function used to normalize the resource pattern vector, W r and X r is the model weight matrix, b r is the bias term, and tanh(·) is used to capture the nonlinear relationship in resource usage patterns;

[0165] S33, based on resource usage pattern vector The agent generates a pseudo-label matrix that reflects historical resource utilization and its impact on system performance Indicates resource utilization, task completion time, and system load balance in each time period;

[0166] S34, Agent based on pseudo-label matrix Use multivariate time series forecasting models to capture the continuation of historical resource usage patterns in the future and predict future resource demand:

[0167]

[0168] in, is the resource demand forecast after Δ time in the future, and α is the attenuation coefficient, which is used to reduce the impact of long-term data;

[0169] S35. Future resource requirements based on forecasts Intelligent generation of future state representation

[0170]

[0171] S36, the pseudo label matrix and future state representation Integrate into the reinforcement learning model to update the resource allocation plan:

[0172]

[0173] Among them, W(t) is the key performance indicator weighting function, λ2 is the penalty coefficient, Ideal resource utilization.

[0174] In this implementation, S5 includes the following steps:

[0175] S51. The agent generates a resource allocation strategy matrix based on the reinforcement learning model in the initial stage The resource allocation strategy matrix represents the resource allocation plan of node i at time t:

[0176]

[0177] in, and They represent the proportion of computing resources, storage resources, and training resources allocated to node i at time t respectively;

[0178] S52, according to the current state space and action space The agent chooses the optimal action a at each time step t ij (t):

[0179]

[0180] in, The reward value for executing action a is used to guide the agent to choose a resource allocation plan;

[0181] S53, the agent adjusts the resource allocation of each node in real time and updates the resource allocation matrix by executing the reinforcement learning strategy To meet current mission requirements:

[0182]

[0183] Where η is the learning rate, represents the gradient of the reward function with respect to the current allocation scheme;

[0184] S54, the resource scheduling framework allocates resources on the cloud computing platform according to the resource allocation strategy generated by the intelligent agent, so that the nodes are in a resource coordination state:

[0185]

[0186] in, represents the proportion of computing resources allocated by node i to task j, is the sum of computing resources of all nodes at time t.

[0187] In this implementation, S6 includes the following steps:

[0188] S61. During the resource scheduling process, the agent collects the resource utilization efficiency of the system in real time Task execution progress P j (t) and system load state B i (t);

[0189] S62. Based on the collected real-time data, the intelligent agent generates feedback signals Used to describe the deviation of current resource usage from system performance:

[0190]

[0191] in, is the ideal resource efficiency on node i, γ is the penalty coefficient for task delay;

[0192] S63, the agent responds to the feedback signal Adjusting the Reward Function of a Reinforcement Learning Model Optimize resource allocation strategy:

[0193]

[0194] Among them, W(t) is the weighted comprehensive evaluation function of the key performance indicators, and λ3 is the feedback penalty coefficient, which is used to dynamically adjust the reward function;

[0195] S64, the agent recalculates the resource allocation strategy matrix at each time step t And update resource allocation in real time through the resource scheduling framework:

[0196]

[0197] Among them, η1 is the learning rate, is the gradient of the reward function, which is used to guide the optimization of the resource allocation strategy.

[0198] In this implementation, S9 includes the following steps:

[0199] S91. During the task execution process, the agent analyzes the real-time data of the task based on the self-supervised learning model. Construct a task stage classification function to identify the different stages of the task:

[0200]

[0201] Among them, f stage (t) is the prediction function of the task stage at time t, which is used to maximize the matching probability of the current stage. is the workload weight vector of task j at time t, e -α(τ-t) is the time decay factor, α controls the impact of distant period data on the current stage recognition, and the sum of squares in the denominator represents the volatility of workload data. is the average workload of the task in the time period, ∈ is a constant to prevent the denominator from being zero;

[0202] S92, the agent classifies according to the classification function f stage (t) The output recognition task is currently in the stage And generate the corresponding resource demand weight matrix

[0203] S93, based on the resource demand weight matrix of the current task stage, the agent updates the resource allocation strategy matrix To meet the needs of the current stage:

[0204]

[0205] in, represents the resource allocation strategy matrix of the current stage, ⊙ is the symbol of matrix multiplication, and η is the learning rate;

[0206] S94. When the task changes from one stage Transition to the next stage When the agent updates the stage prediction in real time and generates a new resource demand weight matrix

[0207] S95, the agent dynamically monitors the execution status of tasks at each stage through the resource scheduling framework to meet the resource requirements of each stage, while maintaining system load balance during resource transition;

[0208] S96. The intelligent agent dynamically adjusts the resource demand weights and reward functions based on the execution feedback of each stage and the self-supervised learning model to achieve efficient task execution and optimal resource utilization.

[0209] Embodiment 1:

[0210] In order to verify the feasibility and effectiveness of the present invention in the allocation of cloud computing storage, computing and training resources, the experimenters designed a simulation scenario of a large-scale AI model training task and compared it with the traditional resource allocation method in detail. The experiment was conducted in a cluster environment of a cloud computing data center from August 1 to August 10, 2024, simulating the distributed training task of anti-fraud models carried out by financial institutions. The resource types in the experiment included computing resources (CPU and GPU), storage resources (database and cache), and training resources (training process of deep learning model).

[0211] Financial institutions need to complete the training of a real-time anti-fraud model in a short period of time. The model needs to process a large amount of transaction data to analyze the characteristics of each transaction in order to identify abnormal transaction behavior. The model training process includes three main stages: initialization stage (data loading and preprocessing), computationally intensive stage (feature extraction and model building) and training optimization stage (model training and parameter tuning). The resource allocation of the cloud computing platform is crucial because the requirements for computing, storage and training resources vary at different stages. The initialization stage mainly consumes storage resources, the computationally intensive stage requires a large amount of CPU and GPU resources, and the training optimization stage relies on the coordinated scheduling of computing resources and training resources.

[0212] The method of the present invention and the traditional resource allocation method were compared and tested in the above scenario. This experiment used 10 cloud computing nodes, each node was equipped with 32 CPU cores, 128GB of memory, 2 Tesla V100 GPUs, and 1TB of NVMe storage. The financial institution planned to complete the full training of the model within 10 days and required the stability of resource allocation even when the system load was high.

[0213] During the simulation process, the traditional method adopts a static resource allocation strategy, which pre-allocates the resources of each node before the task starts. Each node is allocated 32 CPU cores, 2 GPUs, and a fixed size of cache and storage. The resource utilization rate is high when loading data at the beginning of the task, but as the task enters the computationally intensive stage, some nodes are short of resources, while the resources of other nodes are idle, resulting in a decline in system performance.

[0214] In contrast, the method of the present invention adopts a dynamic resource scheduling mechanism that combines self-supervised learning with reinforcement learning. In the initialization phase of the task, the system generates a pseudo-label matrix by analyzing historical transaction data and estimates future resource requirements. During the training process, the intelligent agent monitors the usage of resources in real time and dynamically adjusts the resource allocation strategy for each stage. In the data loading phase, the system prioritizes allocating more storage resources to nodes while reducing the occupancy of computing resources. In the computing-intensive phase, the system concentrates GPU and CPU resources on model building nodes and optimizes the use of computing resources through cross-node scheduling. In the training optimization phase, training resources and computing resources are dynamically allocated according to the task progress and model performance of each node.

[0215] In order to verify the effectiveness of the method of the present invention, the experimenters compared the resource utilization, task completion time and system load balance of the method of the present invention and the traditional method in a 10-day training task. The experimental data is shown in Table 1 below:

[0216] Table 1 Experimental data of resource utilization, task completion time and system load balancing of the method of the present invention and the traditional method in a 10-day training task

[0217]

[0218] It can be seen from the experimental data in Table 1 above that the method of the present invention is superior to the traditional method in terms of resource utilization, system load balancing and task completion time. The dynamic resource scheduling mechanism of the present invention shortens the data loading time by 33%, reduces the time of the computationally intensive phase by 20%, and shortens the time of the training optimization phase by 10%. The improvement is mainly attributed to the intelligent agent's accurate identification of the task phase and its adaptive resource scheduling capabilities.

[0219] In addition, the method of the present invention significantly improves the resource utilization of the system, with an average utilization of 92%, while the traditional method is only 74%. Through cross-node collaborative scheduling, the method of the present invention also improves the system's load balancing index to 0.93, avoiding the problem of idle resources in some nodes. The peak utilization of the GPU is increased by 15%, making the system perform better when processing computationally intensive tasks.

[0220] The method of the present invention also reduces the cost of cloud computing training tasks, with the total training cost reduced by 17%. During the 10-day simulation process, the method of the present invention effectively reduced unnecessary resource waste and shortened the overall training time, enabling financial institutions to complete model development and deployment faster to support their businesses.

[0221] Through the verification of this embodiment, the dynamic resource scheduling method combining self-supervised learning and reinforcement learning of the present invention has demonstrated excellent performance and flexibility in cloud computing storage, computing and training resource allocation. The experimental results show that the method of the present invention not only improves resource utilization and system load balancing, but also shortens task execution time and reduces overall costs, providing a more intelligent and efficient solution for AI model training of financial institutions.

[0222] The present invention realizes adaptive optimization in the resource allocation process by combining self-supervised learning with reinforcement learning. The intelligent agent uses the self-supervised learning model to extract patterns from historical resource usage data and generate pseudo labels as training data to provide support for accurate prediction of future resource needs. Unlike traditional static allocation strategies, the intelligent agent of the present invention can dynamically adjust the allocation strategy of computing, storage and training resources during task execution, so that the system can respond in real time when facing sudden tasks or load changes, reducing resource allocation lag and waste. The reinforcement learning model continuously optimizes the reward function by introducing a real-time feedback mechanism to ensure that the efficiency of resource use and the stability of task execution are improved, thereby enhancing the intelligent level of resource scheduling.

[0223] During the task execution process of the present invention, the intelligent agent automatically identifies the different stages of the task through a self-supervised learning model, including the initialization stage, the computationally intensive stage and the training optimization stage. By performing differentiated analysis on the resource requirements of each stage, the system can dynamically generate a resource allocation strategy that matches the task stage to ensure that each task obtains the required resource support at the critical stage of its life cycle. The traditional system lacks the ability to identify the task stage, resulting in inaccurate resource allocation at different stages of task execution. The phased resource scheduling method of the present invention effectively avoids resource overload or shortage, greatly shortens the task execution time, and significantly improves the load balancing effect of the system.

[0224] In the present invention, the intelligent agent constructs a cross-node collaborative scheduling mechanism based on the resource scheduling framework. By analyzing the resource usage and task requirements of different nodes, the intelligent agent dynamically allocates resources among multiple nodes to optimize the global utilization of resources. The multi-stage resource requirement matrix between tasks is used to make resource scheduling not only limited to the optimization of a single node, but also realize efficient collaboration among multiple nodes. Compared with the isolated resource management mode between nodes in traditional methods, the cross-node scheduling model reduces resource competition and delay between nodes and improves the overall performance of the system. It performs particularly well in large-scale computing tasks and complex training task scenarios.

[0225] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling, characterized in that: The steps include: S1. Build a multi-resource scheduling environment for the cloud computing platform, which includes storage resources, computing resources, and training resources, and establish a unified resource scheduling framework; S2. Define an agent in the cloud computing platform, which is responsible for executing the self-supervised reinforcement learning strategy, and collects the workload, task execution and resource utilization data in the current cloud computing platform by monitoring the usage of storage resources, computing resources and training resources in the current system, and constructs the state space and action space; S3, the agent uses the self-supervised learning model to self-train the collected historical resource usage data, automatically mines resource usage patterns and generates pseudo labels. The pseudo labels are used to represent the resource utilization rate, task completion time and system load balancing in the historical data, and generate a state representation to predict future resource demand; S4. Based on the pseudo labels generated by self-supervised learning, the agent initializes the state space of the reinforcement learning model. The state space includes the current usage of storage resources, computing resources, and training resources. The action space includes the allocation strategy of storage resources, computing resources, and training resources. The reward function of the weighted comprehensive evaluation of key performance indicators is initialized. S5, the agent generates the initial storage resource, computing resource and training resource allocation strategy by executing the reinforcement learning model, and allocates and adjusts resources in real time in the cloud computing platform through the resource scheduling framework to meet the current task requirements; S6. During the resource allocation process, the agent provides real-time feedback on the system's resource utilization efficiency, task execution progress, and system load based on the resource scheduling framework, and continuously adjusts the reward function of the reinforcement learning model based on the self-supervised learning feedback signal; S7. When the resource demand in the cloud computing platform changes dynamically, the agent quickly responds to dynamically adjust the allocation of storage resources, computing resources, and training resources through the resource scheduling framework based on the resource demand predicted by the self-supervised learning model; S8, the agent optimizes resource allocation among multiple cloud computing nodes or platforms through a collaborative scheduling mechanism across multiple nodes and platforms, based on a cross-node resource scheduling pattern generated by self-supervised learning; S9. The agent identifies the execution stage of each task and automatically identifies the different stages of the current task through the self-supervised learning model, including the task initialization stage, the computationally intensive stage, and the training optimization stage. It also dynamically adjusts the resource allocation strategy according to the resource requirements of different stages, so that each task can obtain the best resource allocation at different stages. S10. Finally, the optimal storage resource, computing resource and training resource allocation plan is generated.

2. According to claim 1, a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling is characterized in that: The S1 comprises the following steps: S11. Define multiple resource types in the cloud computing platform, including computing resources C r , storage resources S r and training resources T r ; S12. Establish resource status matrix: in, is the resource status matrix of the ith node at time t, Respectively represent the real-time usage of computing resources, storage resources, and training resources on node i; S13, construct a resource utilization function U for evaluating the current utilization of each type of resource r (t): in, represents the actual usage of resource r on node i at time t, is the maximum available capacity of the corresponding resource on node i; S14. Build a unified resource scheduling framework and define resource scheduling decision variables x i (t), where x i (t)∈{C r ,S r ,T r } represents the resource allocation plan at time t; S15. Establish constraints for the resource scheduling framework, including resource usage constraints: Resource supply and demand balance constraints: Resource conflict avoidance constraints: S16. Initialize the state space S in the resource scheduling framework based on the resource state matrix and resource utilization function t and action space A t : A t ={a ij (t)∣a ij (t)∈{C r ,S r ,T r }}; Where: S t Contains the resource status of all nodes and the workload of tasks, A t Indicates the resource allocation actions of each node. S17. Define the weighted comprehensive evaluation function of key performance indicators: Among them, α r is the weight coefficient of resource r, satisfying ∑α r =1, W(t) is used to evaluate the utilization and scheduling efficiency of each resource; S18. Build a cross-node collaborative scheduling model and define the cross-node resource allocation collaborative formula: in, It represents the ratio of resources allocated by node i to task j at time t. The denominator represents the total resources and total demand of all nodes and tasks.

3. According to claim 1, a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling is characterized in that: The S2 comprises the following steps: S21. Define Agent A i , the agent is responsible for executing the self-supervised reinforcement learning strategy on node i and monitoring the storage resources of node i Computing resources and training resources usage of S22. Define the local state space for the agent The structure is: in, Represents the state space of the local resources of the agent on node i, including the usage of computing resources, storage resources, and training resources of the current node, as well as the real-time workload of the task S23. Define the local action space of the agent for: in, Describes the agent's resource allocation actions within node i, which is used for local resource management of each node; S24. Construct a local reward function for each agent Used to measure the resource allocation efficiency of nodes: Among them, W i (t) is the resource utilization weighted function of node i, is the ideal resource utilization of node i, λ1 is the penalty coefficient; S25, the agent in each node through the local state space and action space Execute self-supervised reinforcement learning strategies to adjust resource allocation plans within nodes.

4. According to claim 1, a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling is characterized in that: The S3 comprises the following steps: S31, Agent A i Collect historical resource usage data on node i and build a historical resource usage matrix S32, using time series self-supervision model based on historical resource matrix Extract resource usage patterns and generate resource usage pattern vectors The resource usage pattern vector represents the change trend of different resources on node i in the historical period: Where σ(·) is the activation function used to normalize the resource pattern vector, W r and X r is the model weight matrix, b r is the bias term, and tanh(·) is used to capture the nonlinear relationship in resource usage patterns; S33, based on resource usage pattern vector The agent generates a pseudo-label matrix that reflects historical resource utilization and its impact on system performance Indicates resource utilization, task completion time, and system load balance in each time period; S34, Agent based on pseudo-label matrix Use multivariate time series forecasting models to capture the continuation of historical resource usage patterns in the future and predict future resource demand: in, is the resource demand forecast after Δ time in the future, and α is the attenuation coefficient, which is used to reduce the impact of long-term data; S35. Future resource requirements based on forecasts Intelligent generation of future state representation S36, the pseudo label matrix and future state representation Integrate into the reinforcement learning model to update the resource allocation plan: Among them, W(t) is the key performance indicator weighting function, λ2 is the penalty coefficient, Ideal resource utilization.

5. According to claim 1, a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling is characterized in that: The S5 comprises the following steps: S51. The agent generates a resource allocation strategy matrix based on the reinforcement learning model in the initial stage The resource allocation strategy matrix represents the resource allocation plan of node i at time t: in, and They represent the proportion of computing resources, storage resources, and training resources allocated to node i at time t respectively; S52, according to the current state space and action space The agent chooses the optimal action a at each time step t ij (t): in, The reward value for executing action a is used to guide the agent to choose a resource allocation plan; S53, the agent adjusts the resource allocation of each node in real time and updates the resource allocation matrix by executing the reinforcement learning strategy To meet current mission requirements: Where η is the learning rate, represents the gradient of the reward function with respect to the current allocation scheme; S54, the resource scheduling framework allocates resources on the cloud computing platform according to the resource allocation strategy generated by the intelligent agent, so that the nodes are in a resource coordination state: in, represents the proportion of computing resources allocated by node i to task j, is the sum of computing resources of all nodes at time t.

6. According to claim 1, a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling is characterized in that: The S6 comprises the following steps: S61. During the resource scheduling process, the agent collects the resource utilization efficiency of the system in real time Task execution progress P j (t) and system load state B i (t); S62. Based on the collected real-time data, the intelligent agent generates feedback signals Used to describe the deviation of current resource usage from system performance: in, is the ideal resource efficiency on node i, γ is the penalty coefficient for task delay; S63, the agent responds to the feedback signal Adjusting the Reward Function of a Reinforcement Learning Model Optimize resource allocation strategy: Among them, W(t) is the weighted comprehensive evaluation function of the key performance indicators, and λ3 is the feedback penalty coefficient, which is used to dynamically adjust the reward function; S64, the agent recalculates the resource allocation strategy matrix at each time step t And update resource allocation in real time through the resource scheduling framework: Among them, η1 is the learning rate, is the gradient of the reward function, which is used to guide the optimization of the resource allocation strategy.

7. According to claim 1, a cloud computing storage and training resource allocation method based on reinforcement learning and dynamic resource scheduling is characterized in that: The S9 comprises the following steps: S91. During the task execution process, the agent analyzes the real-time data of the task based on the self-supervised learning model. Construct a task stage classification function to identify the different stages of the task: Among them, f stage (t) is the prediction function of the task stage at time t, which is used to maximize the matching probability of the current stage. is the workload weight vector of task j at time t, e -α(τ-t) is the time decay factor, α controls the impact of distant period data on the current stage recognition, and the sum of squares in the denominator represents the volatility of workload data. is the average workload of the task in the time period, ∈ is a constant to prevent the denominator from being zero; S92, the agent classifies according to the classification function f stage (t) The current stage of the output recognition task And generate the corresponding resource demand weight matrix S93, based on the resource demand weight matrix of the current task stage, the agent updates the resource allocation strategy matrix To meet the needs of the current stage: in, represents the resource allocation strategy matrix of the current stage, ⊙ is the symbol of matrix multiplication, and η is the learning rate; S94. When the task changes from one stage Transition to the next stage When the agent updates the stage prediction in real time and generates a new resource demand weight matrix S95, the agent dynamically monitors the execution status of tasks at each stage through the resource scheduling framework to meet the resource requirements of each stage, while maintaining system load balance during resource transition; S96. The intelligent agent dynamically adjusts the resource demand weights and reward functions based on the execution feedback of each stage and the self-supervised learning model to achieve efficient task execution and optimal resource utilization.

Images