Computing resource dynamic allocation method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence technology and financial science and technology, and discloses a computing resource dynamic allocation method, device and equipment and a storage medium, and the method comprises the steps: obtaining a pre-constructed resource scheduler cluster, and carrying out the initial environment modeling of the resource scheduler cluster, and obtaining an initial scheduler cluster; selecting a resource scheduling action in the initialized scheduler cluster by using a preset reinforcement learning model; performing model optimization on the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model; and calculating a resource scheduling strategy of the resource scheduler cluster according to the target scheduling model. The method can effectively improve the utilization rate of computing resources and the completion efficiency of computing tasks.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a method, device, electronic device and storage medium for dynamically allocating computing resources. Background Art

[0002] In the field of financial technology, financial institutions need to process a large amount of transaction data and customer requests. Apache Hadoop YARN (Yet Another Resource Negotiator) is the core component of resource management and scheduling, responsible for allocating computing resources to various applications in the cluster. However, in actual operation, YARN's resource allocation strategy is often based on static configuration, which is difficult to adapt to dynamically changing workloads, resulting in low resource utilization and low task execution efficiency.

[0003] The FIFO (First-In First-Out) and Capacity Scheduler scheduling strategies adopted by YARN (Yet Another Resource Negotiator) by default are simple and easy to use, but lack the ability to perceive and respond to real-time workload changes. In the scenario of multi-tenant shared clusters, this static allocation method often leads to resource competition and waste, affecting task execution efficiency and user experience. In the case of tight resources, low-priority or long-tail tasks may wait for a long time due to insufficient resources, affecting the overall task processing efficiency. In addition, unreasonable resource allocation may also lead to extended task execution time, increasing the uncertainty of financial job completion. Therefore, how to improve the utilization of computing resources and the processing efficiency of computing tasks has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a method and device for dynamically allocating computing resources and a computer-readable storage medium, the main purpose of which is to solve the problems of low computing resource utilization and low computing task processing efficiency.

[0005] In a first aspect, to achieve the above-mentioned purpose, the present invention provides a method for dynamically allocating computing resources, comprising:

[0006] Obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster;

[0007] Selecting resource scheduling actions from the initialization scheduler cluster using a preset reinforcement learning model;

[0008] Optimizing the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model;

[0009] The resource scheduling strategy of the resource scheduler cluster is calculated according to the target scheduling model.

[0010] In a second aspect, the present invention further provides a computing resource dynamic allocation device, the device comprising:

[0011] An initial environment modeling module is used to obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster;

[0012] A resource scheduling action selection module, used to select a resource scheduling action from the initialization scheduler cluster using a preset reinforcement learning model;

[0013] A model optimization module, used to optimize the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model;

[0014] The resource scheduling strategy generation module is used to calculate the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model.

[0015] In a third aspect, the present invention further provides an electronic device, the electronic device comprising:

[0016] at least one processor; and,

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned method for dynamically allocating computing resources.

[0019] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned method for dynamic allocation of computing resources.

[0020] In an embodiment of the present invention, a pre-built resource scheduler cluster is obtained, and an initial environment modeling is performed on the resource scheduler cluster to obtain an initialized scheduler cluster, which can reflect the real-time status of the resource scheduler cluster and the execution action of the resource scheduler cluster, thereby exploring an unknown action space; a preset reinforcement learning model is used to select a resource scheduling action in the initialized scheduler cluster, and the state change of the initialized scheduler cluster after the action is executed can be observed in real time, which is convenient for real-time adjustment of computing resources and improves the accuracy of subsequent resource scheduling strategies; the reinforcement learning model is optimized according to the resource scheduling action to obtain a target scheduling model, and the resource scheduling strategy of the resource scheduler cluster is calculated using the target scheduling model, which can intelligently schedule computing resources, enter a state where key tasks can be processed first, reduce the waiting time of computing tasks, speed up the execution progress of computing tasks, and effectively improve the utilization rate of computing resources and the processing efficiency of computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 is a schematic diagram of an application environment of a method for dynamically allocating computing resources in one embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a flow chart of a method for dynamically allocating computing resources provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a module of a computing resource dynamic allocation device provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device for dynamically allocating computing resources provided by an embodiment of the present invention;

[0026] Figure 5 It is another structural schematic diagram of an electronic device for dynamic allocation of computing resources provided by an embodiment of the present invention.

[0027] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0030] The embodiment of the present application provides a method for dynamically allocating computing resources, and the execution subject of the method for dynamically allocating computing resources includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the device provided in the embodiment of the present application. In other words, the method for dynamically allocating computing resources can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0031] The computing resource dynamic allocation method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can obtain a pre-built resource scheduler cluster through the client, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster; select resource scheduling actions in the initialized scheduler cluster using a preset reinforcement learning model; optimize the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model; calculate the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model, and feed the resource scheduling strategy back to the client. In the present invention, for dynamic allocation of computing resources, in an embodiment of the present invention, by obtaining a pre-built resource scheduler cluster, the initial environment modeling of the resource scheduler cluster is performed to obtain an initialization scheduler cluster, which can reflect the real-time state of the resource scheduler cluster and the execution action of the resource scheduler cluster, and then explore the unknown action space; by using a preset reinforcement learning model to select a resource scheduling action in the initialization scheduler cluster, the state change of the initialization scheduler cluster after the action is executed can be observed in real time, which is convenient for real-time adjustment of computing resources and improves the accuracy of subsequent resource scheduling strategies; according to the resource scheduling action, the reinforcement learning model is optimized to obtain a target scheduling model, and the resource scheduling strategy of the resource scheduler cluster is calculated using the target scheduling model, which can intelligently schedule computing resources, enter the state that can prioritize key tasks, reduce the waiting time of computing tasks, speed up the execution progress of computing tasks, and effectively improve the utilization rate of computing resources and the processing efficiency of computing tasks. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] Reference Figure 2 , which is a flow chart of a method for dynamic allocation of computing resources provided by an embodiment of the present invention.

[0033] In this embodiment, the method for dynamically allocating computing resources includes:

[0034] S1. Obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster.

[0035] In the embodiment of the present invention, the resource scheduler cluster is a YARN (Apache Hadoop YARN, Yet Another Resource Negotiator) cluster. As a core component of resource management and scheduling, the resource scheduler cluster is responsible for allocating computing resources to various applications in the cluster.

[0036] In detail, in the financial field, financial institutions need to process massive amounts of transaction data, including but not limited to stock transactions, bank transfers, credit card transactions, etc. This data needs to be processed and analyzed efficiently for risk management, fraud detection, customer behavior analysis, etc. For example, by using big data analysis and machine learning technology, financial institutions can more accurately predict market trends and customer behavior, thereby providing more personalized services and products.

[0037] Specifically, the resource scheduler cluster is an open source resource management platform for managing and scheduling resources in a computer cluster. YARN is part of the Hadoop ecosystem, but can also be used independently of Hadoop. The YARN cluster includes the esourceManager (RM) component and the NodeManager (NM) component. The esourceManager (RM) component is the central scheduler of the entire cluster, responsible for receiving client requests, allocating resources to applications, and monitoring the execution of applications. The NodeManager (NM) component runs on each node in the cluster and is responsible for managing the resources of a single node, such as CPU, memory, and disk space, and executing tasks assigned by the ResourceManager. Therefore, the YARN cluster can allocate computing resources and improve the overall task processing efficiency.

[0038] Furthermore, the initial environment modeling includes the state space, action space, and reward function of the resource scheduler cluster. The state space contains the resource usage of all nodes in the resource scheduler cluster, the list of currently running tasks, the resource requirements and priorities of each task, etc. The update frequency of the state should be fast enough to reflect the real-time status of the resource scheduler cluster; the action space defines all possible actions that YARN can take, such as allocating a specific amount of CPU and memory to a task, adjusting the priority of a task, pausing or restarting a task; the reward function is a key indicator for measuring the quality of resource scheduling strategies, and is usually designed to maximize resource utilization and task completion efficiency while minimizing resource waste and task delays.

[0039] In the embodiment of the present invention, the initial environment modeling of the resource scheduler cluster to obtain the initialized scheduler cluster includes:

[0040] Acquire status information of each node in the resource scheduler cluster, and construct a state space of the resource scheduler cluster according to the status information;

[0041] Defining execution actions of the resource scheduler cluster, and constructing an action space of the resource scheduler cluster according to the execution actions;

[0042] A preset reward function is obtained, and the resource scheduler cluster is initialized according to the state space, the action space and the reward function to obtain an initialized scheduler cluster.

[0043] In the embodiment of the present invention, the node status information includes the resource usage of all nodes, the list of currently running tasks, the resource requirements and priority of each task, etc., wherein the nodes in the resource scheduler cluster can be physical machines or virtual machines. The scale and configuration of the cluster can be adjusted according to demand. For example, a small cluster may have only a few machines, each of which runs multiple NodeManagers and multiple DataNodes responsible for storing data in HDFS (Hadoop Distributed File System), while a large cluster may have hundreds or thousands of machines, each of which runs a NodeManager and a DataNode.

[0044] Furthermore, the reward function represents the expectation of the reward obtained when the state of the resource scheduler cluster is transferred to another state, and can be calculated based on the state information of each node in the resource scheduler cluster. Specifically, the state of the resource scheduler cluster can be determined based on the actions of the resource scheduler cluster, for example, the resource usage after allocating resources to a specific task, the task execution progress and other cluster states.

[0045] In the embodiment of the present invention, the real-time status of the resource scheduler cluster and the execution actions of the resource scheduler cluster can be reflected through initial environment modeling, thereby exploring the unknown action space. The reward function can be used to calculate whether the state of the resource scheduler cluster is the optimal state, thereby adjusting the allocation of computing resources in real time, reducing resource idleness and competition, and significantly improving resource utilization.

[0046] S2. Select resource scheduling actions from the initialization scheduler cluster using a preset reinforcement learning model.

[0047] In an embodiment of the present invention, the reinforcement learning model may be a model algorithm such as Q-Learning, Deep Q-Network (DQN) or Actor-Critic, wherein Q-Learning is a model-free reinforcement learning algorithm, which guides the agent to select the optimal action in a specific state by learning an action-value function (Q-function) to maximize the cumulative reward, and then selects the currently considered best action from the initialized scheduler cluster, that is, the resource scheduling action; Deep Q-Network (DQN) is an algorithm that combines deep learning and reinforcement learning, which uses a deep neural network to approximate the Q function in Q-Learning and learn the best resource scheduling action in the initialized scheduler cluster; the Actor-Critic method is a reinforcement learning algorithm that combines policy iteration and value iteration, which consists of two parts: Actor and Critic, the Actor is responsible for selecting an action according to the current policy, in the Actor-Critic method, the Actor is usually a policy network, which accepts the state of the environment as input and outputs the probability distribution of taking each action in the state. The goal of this network is to maximize the cumulative reward, which is optimized through the policy gradient method. The Critic is responsible for evaluating the quality of the action selected by the Actor. The Critic can be a value network that evaluates the value of the current state (State Value, denoted as V(s)) or the value of taking an action in the current state (Action Value, denoted as Q(s,a)). The output of the Critic network is usually used to calculate the time difference error (TD Error), which reflects the difference between the actual return and the expected return. The Actor-Critic method does not need to learn a complete value function, but guides the Actor's policy update through the immediate feedback provided by the Critic. The Actor-Critic method combines the advantages of the policy-based method (Policy-based) and the value-based method (Value-based), allowing the algorithm to learn more effectively in complex environments.

[0048] In the embodiment of the present invention, the selecting of resource scheduling actions in the initialization scheduler cluster by using a preset reinforcement learning model includes:

[0049] Initializing model parameters of the reinforcement learning model to obtain an action selection model;

[0050] Selecting an execution action in the action space of the initialization scheduler cluster, and obtaining an execution state corresponding to the execution action;

[0051] The action selection model is used to calculate a model evaluation value of the execution state, and a resource scheduling action is determined according to the model evaluation value.

[0052] In an embodiment of the present invention, the initialized model parameters may be algorithm parameters such as learning rate, discount factor, exploration rate, etc., wherein the learning rate is an important parameter in the optimization algorithm, which determines the step size of the model parameter update during the gradient descent process, and the discount factor (usually expressed as γ) is used to determine the importance of future rewards relative to current rewards. A smaller discount factor means that the agent pays more attention to immediate rewards, while a larger discount factor indicates that the agent pays more attention to long-term rewards. The exploration rate is a parameter that controls the agent's exploration of the environment in reinforcement learning. A higher exploration rate can encourage the agent to explore more state-action pairs, which helps to discover better strategies, but too much exploration may lead to a slow learning process. A lower exploration rate may cause the agent to converge to a suboptimal strategy prematurely.

[0053] Furthermore, the ε-greedy strategy can be used to select one in the action space in the initialized scheduler cluster, where the ε-greedy strategy selects a random action with a probability of ε and selects the action with the largest model evaluation value of the current action selection model with a probability of 1-ε.

[0054] Among them, the ε-greedy strategy will randomly select a number r between 0 and 1 during execution. If r is less than ε, a random action will be selected. If r is less than ε, the execution action with the largest model evaluation value will be selected as the resource scheduling action.

[0055] In an embodiment of the present invention, after selecting an execution action, the execution action will be directly applied in the initialized scheduler cluster, such as allocating resources to a specific task, and then observing the cluster state changes after the execution action, including resource usage, task execution progress, etc., to obtain the execution state corresponding to the execution action.

[0056] Furthermore, the using the action selection model to calculate the model evaluation value of the execution state, and determining the resource scheduling action according to the model evaluation value, includes:

[0057] Calculating an initial evaluation value of the execution state using a model evaluation function in the action selection model;

[0058] Iteratively updating the execution state according to the initial evaluation value until the number of iterative updates meets a preset number threshold, thereby obtaining an updated state;

[0059] A model evaluation value corresponding to the update state is calculated, and an execution response corresponding to the model evaluation value is used as a resource scheduling action.

[0060] In an embodiment of the present invention, the action selection model includes a corresponding model evaluation function, for example, the action-value function (Q-function) in Q-Learning, the Q function in Deep Q-Network (DQN), and the Q(s,a) function of the Actor-Critic method, so as to calculate the evaluation value corresponding to each execution state.

[0061] In detail, the update iteration is to repeat the above-mentioned steps of selecting an execution action in the action space in the initialized scheduler cluster, obtaining the execution state corresponding to the execution action, recalculating the evaluation value after each iteration, until the number of iterative updates meets the preset number threshold, and obtaining the updated state, and making the model evaluation function converge to the optimal, and obtaining the model evaluation value corresponding to the updated state, and taking the execution corresponding to the model evaluation value as the resource scheduling action.

[0062] In the embodiment of the present invention, the resource scheduling action is the optimal execution action for the current state. Through the resource scheduling action, the state changes of the initialized scheduler cluster after the action is executed can be observed in real time, for example, resource usage, task execution progress, etc., so as to adjust the computing resources in real time and improve the accuracy of subsequent resource scheduling strategies.

[0063] S3. Optimize the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model.

[0064] In an embodiment of the present invention, model optimization is an important part of deep learning, which involves adjusting the hyperparameters of the model to improve the performance of the model. Model optimization can be to optimize the model parameters initialized by the reinforcement learning model. In DQN, it can be to update and optimize the weights of the neural network; in the Actor-Critic method, it can be to update and optimize the policy network and the value network.

[0065] In detail, model weights are parameters adjusted during model learning, representing the mapping relationship from input to output of the model. In a neural network, weights are the edges that connect neurons; hyperparameters are parameters set before model training. They control the learning process, but are not learned directly from training data. Hyperparameters can include learning rate: controlling the step size of model weight updates, batch size: the number of samples used in each gradient update, etc. By performing model optimization, the strategy of the reinforcement learning model can be updated to improve the accuracy of subsequent resource scheduling.

[0066] In the embodiment of the present invention, the step of optimizing the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model includes:

[0067] Monitoring the cluster state of the initialization scheduler cluster according to the resource scheduling action;

[0068] Calculating the instant reward of the cluster state using the reward function in the initialized scheduler cluster;

[0069] Parameters of the reinforcement learning model are optimized according to the immediate reward to obtain a target scheduling model.

[0070] In an embodiment of the present invention, the cluster state is the state after the initialized scheduler cluster executes the selected execution action. The reward function obtained after the initial environment modeling in the initialized scheduler cluster can calculate the instant reward between the cluster state and the initial state, that is, the instant reward can reflect the resource utilization and task completion efficiency, and at the same time, the resource waste and task delay. For example, the larger the instant reward, the greater the resource utilization and task completion efficiency, and the smaller the resource waste and task delay. The design of the reward function can maximize the resource utilization and task completion efficiency while minimizing resource waste and task delay.

[0071] Furthermore, the reinforcement learning model can be optimized by the gradient descent method until the immediate reward meets the preset immediate reward threshold.

[0072] In detail, gradient descent is an iterative algorithm used to minimize functions in machine learning and optimization theory. In machine learning, gradient descent is usually used to minimize the loss function, that is, the difference between the model prediction and the actual observation value. In the present invention, the loss function can represent the difference between the immediate reward and the immediate reward threshold.

[0073] In the embodiment of the present invention, the step of optimizing the parameters of the reinforcement learning model according to the instant reward to obtain the target scheduling model includes:

[0074] Constructing a loss function according to the instant reward and a preset instant reward threshold;

[0075] Calculating the gradient of the loss function with respect to model parameters in the reinforcement learning model;

[0076] The model parameters are updated according to the gradient until the loss function is no greater than a preset loss function threshold, thereby obtaining target model parameters;

[0077] The target model parameters are assigned to the model parameters to obtain a target scheduling model.

[0078] In the embodiment of the present invention, the difference between the immediate reward and the immediate reward threshold is used as a loss function to represent the goal of resource scheduling, that is, maximizing resource utilization and task completion efficiency while minimizing resource waste and task delay.

[0079] In detail, for each model parameter in the reinforcement learning model, the gradient of the loss function with respect to the parameter is calculated. The gradient is the derivative of the loss function at the current parameter value, which indicates the rate of change of the loss function in this direction. According to the gradient and the predetermined learning rate (step size), each model parameter is updated, and the steps of calculating the gradient and updating the model parameters are repeated to update the model parameters until the loss function is not greater than the preset loss function threshold, and the target model parameters are obtained. Each target model parameter is assigned to the corresponding model parameter to replace the initialized model parameter, and the model parameter corresponding to the maximum value of the instant function is obtained, thereby improving the efficiency and accuracy of resource scheduling.

[0080] Preferably, the AdaGrad algorithm can be used to perform gradient optimization on the parameters of the reinforcement learning model, wherein AdaGrad (Adaptive Gradient Algorithm) is an optimization algorithm for adaptive learning rate. AdaGrad dynamically adjusts the learning rate according to the historical gradient information of each parameter. The key to the AdaGrad algorithm is to adaptively adjust the learning rate of each parameter so that the model can more flexibly adapt to different parameter update requirements during the optimization process. Specifically, the AdaGrad algorithm adjusts the learning rate by tracking the sum of squared gradients of each parameter to achieve adaptive adjustment of each parameter.

[0081] In the embodiment of the present invention, a more accurate decision can be made through the target scheduling model, so that the resource utilization rate and average task completion time of the resource scheduling are predetermined indicators, and a more accurate resource scheduling strategy is generated.

[0082] S4. Calculate the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model.

[0083] In the embodiment of the present invention, the resource scheduling policy is a policy for resource scheduler cluster to allocate computing resources, for example, allocating resources to specific tasks, resource usage, allocating a specific amount of CPU and memory to a task, etc.

[0084] In the embodiment of the present invention, the step of calculating the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model includes:

[0085] Calculating the optimal execution action of the resource scheduler cluster using the target scheduling model;

[0086] A resource scheduling strategy of the resource scheduler cluster is generated according to the optimal execution action.

[0087] In the embodiment of the present invention, the optimal execution action is the resource scheduling action taken by the resource scheduler cluster. The resource scheduling action is deployed as a resource scheduling strategy to the production environment corresponding to the resource scheduler cluster to calculate resource scheduling in real time and in detail.

[0088] Preferably, the cluster environment and task requirements of the resource scheduler cluster may change over time, so the model needs to be updated regularly to adapt to the new environmental conditions. This may involve retraining the model, or fine-tuning the parameters of the existing model to improve the flexibility of resource scheduling strategy generation.

[0089] In the embodiments of the present invention, resource allocation can be adjusted according to real-time demand through resource scheduling strategies, resource idleness and competition are reduced, and resource utilization is significantly improved; efficient resource utilization reduces unnecessary resource consumption, which helps to control cloud computing costs, especially the impact on task processing efficiency in a pay-as-you-go cloud environment; at the same time, resources are intelligently scheduled through resource scheduling strategies, so that key tasks can be processed first, the waiting time of computing tasks is reduced, the execution progress of computing tasks is accelerated, and efficient completion of computing tasks is achieved, so that financial institutions can comprehensively consider data processing, customer service, risk management, product innovation, infrastructure operations, and computing resource architecture when processing computing tasks of large amounts of transaction data and customer requests, thereby ensuring efficient, stable and secure financial services.

[0090] In the embodiment of the present invention, by obtaining a pre-built resource scheduler cluster, the initial environment modeling of the resource scheduler cluster is performed to obtain an initialization scheduler cluster, which can reflect the real-time state of the resource scheduler cluster and the execution action of the resource scheduler cluster, and then explore the unknown action space; by using a preset reinforcement learning model to select a resource scheduling action in the initialization scheduler cluster, the state change of the initialization scheduler cluster after the action is executed can be observed in real time, which is convenient for real-time adjustment of computing resources and improves the accuracy of subsequent resource scheduling strategies; according to the resource scheduling action, the reinforcement learning model is optimized to obtain a target scheduling model, and the resource scheduling strategy of the resource scheduler cluster is calculated using the target scheduling model, which can intelligently schedule computing resources, enter the state that can prioritize key tasks, reduce the waiting time of computing tasks, speed up the execution progress of computing tasks, and effectively improve the computing resource utilization and the processing efficiency of computing tasks. It should be understood that the size of the sequence number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0091] In the embodiment of the present disclosure, a computing resource dynamic allocation device is provided, and the computing resource dynamic allocation device corresponds to the computing resource dynamic allocation method in the above embodiment. Figure 3 As shown, the computing resource dynamic allocation device 100 can be installed in an electronic device. According to the functions to be implemented, the computing resource dynamic allocation device 100 includes an initial environment modeling module 101, a resource scheduling action selection module 102, a model optimization module 103, and a resource scheduling strategy generation module 104. The functional modules are described in detail as follows:

[0092] The initial environment modeling module 101 is used to obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster;

[0093] The resource scheduling action selection module 102 is used to select resource scheduling actions in the initialization scheduler cluster using a preset reinforcement learning model;

[0094] The model optimization module 103 is used to optimize the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model;

[0095] The resource scheduling strategy generating module 104 is used to calculate the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model.

[0096] In one embodiment, the initial environment modeling module 101 is specifically used to:

[0097] Acquire status information of each node in the resource scheduler cluster, and construct a state space of the resource scheduler cluster according to the status information;

[0098] Defining execution actions of the resource scheduler cluster, and constructing an action space of the resource scheduler cluster according to the execution actions;

[0099] A preset reward function is obtained, and the resource scheduler cluster is initialized according to the state space, the action space and the reward function to obtain an initialized scheduler cluster.

[0100] In one embodiment, the resource scheduling action selection module 102 is used to:

[0101] Initializing model parameters of the reinforcement learning model to obtain an action selection model;

[0102] Selecting an execution action in the action space of the initialization scheduler cluster, and obtaining an execution state corresponding to the execution action;

[0103] The action selection model is used to calculate a model evaluation value of the execution state, and a resource scheduling action is determined according to the model evaluation value.

[0104] In one embodiment, the resource scheduling action selection module 102 is used to:

[0105] Calculating an initial evaluation value of the execution state using a model evaluation function in the action selection model;

[0106] Iteratively updating the execution state according to the initial evaluation value until the number of iterative updates meets a preset number threshold, thereby obtaining an updated state;

[0107] A model evaluation value corresponding to the update state is calculated, and an execution response corresponding to the model evaluation value is used as a resource scheduling action.

[0108] In one embodiment, the model optimization module 103 is used to:

[0109] Monitoring the cluster state of the initialization scheduler cluster according to the resource scheduling action;

[0110] Calculating the instant reward of the cluster state using the reward function in the initialized scheduler cluster;

[0111] Parameters of the reinforcement learning model are optimized according to the immediate reward to obtain a target scheduling model.

[0112] In one embodiment, the model optimization module 103 is used to:

[0113] Constructing a loss function according to the instant reward and a preset instant reward threshold;

[0114] Calculating the gradient of the loss function with respect to model parameters in the reinforcement learning model;

[0115] The model parameters are updated according to the gradient until the loss function is no greater than a preset loss function threshold, thereby obtaining target model parameters;

[0116] The target model parameters are assigned to the model parameters to obtain a target scheduling model.

[0117] In one embodiment, the resource scheduling strategy generation module 104 is used to:

[0118] Calculating the optimal execution action of the resource scheduler cluster using the target scheduling model;

[0119] A resource scheduling strategy of the resource scheduler cluster is generated according to the optimal execution action.

[0120] In the present invention, by obtaining a pre-built resource scheduler cluster, the initial environment modeling of the resource scheduler cluster is performed to obtain an initialization scheduler cluster, which can reflect the real-time state of the resource scheduler cluster and the execution action of the resource scheduler cluster, and then explore the unknown action space; by using a preset reinforcement learning model to select a resource scheduling action in the initialization scheduler cluster, the state change of the initialization scheduler cluster after the action is executed can be observed in real time, which is convenient for real-time adjustment of computing resources and improves the accuracy of subsequent resource scheduling strategies; according to the resource scheduling action, the reinforcement learning model is optimized to obtain a target scheduling model, and the resource scheduling strategy of the resource scheduler cluster is calculated using the target scheduling model, which can intelligently schedule computing resources, enter a state that can prioritize key tasks, reduce the waiting time of computing tasks, speed up the execution progress of computing tasks, and effectively improve the computing resource utilization and computing task processing efficiency. For the specific definition of the computing resource dynamic allocation device, please refer to the definition of the computing resource dynamic allocation method in the above text, which will not be repeated here. Each module in the above-mentioned computing resource dynamic allocation device can be implemented in whole or in part by software, hardware and their combination. The above modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0121] In an embodiment of the present invention, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps of the server side of the above-mentioned dynamic allocation method of computing resources are implemented.

[0122] In an embodiment of the present invention, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the above-mentioned dynamic allocation method of computing resources are implemented.

[0123] In an embodiment of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0124] Obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster;

[0125] Selecting resource scheduling actions from the initialization scheduler cluster using a preset reinforcement learning model;

[0126] Optimizing the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model;

[0127] The resource scheduling strategy of the resource scheduler cluster is calculated according to the target scheduling model.

[0128] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0129] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0130] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0131] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0132] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0133] The readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can achieve:

[0134] Obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster;

[0135] Selecting resource scheduling actions from the initialization scheduler cluster using a preset reinforcement learning model;

[0136] Optimizing the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model;

[0137] The resource scheduling strategy of the resource scheduler cluster is calculated according to the target scheduling model.

[0138] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0139] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0140] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0141] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0142] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0144] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0145] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

[0148] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A method for dynamically allocating computing resources, characterized in that: The method comprises: Obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster; Selecting resource scheduling actions from the initialization scheduler cluster using a preset reinforcement learning model; Optimizing the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model; The resource scheduling strategy of the resource scheduler cluster is calculated according to the target scheduling model.

2. A method for dynamically allocating computing resources as claimed in claim 1, characterized in that: The initial environment modeling of the resource scheduler cluster to obtain the initialized scheduler cluster includes: Acquire status information of each node in the resource scheduler cluster, and construct a state space of the resource scheduler cluster according to the status information; Defining execution actions of the resource scheduler cluster, and constructing an action space of the resource scheduler cluster according to the execution actions; A preset reward function is obtained, and the resource scheduler cluster is initialized according to the state space, the action space and the reward function to obtain an initialized scheduler cluster.

3. A method for dynamically allocating computing resources as claimed in claim 1, characterized in that: The selecting of resource scheduling actions in the initialization scheduler cluster by using a preset reinforcement learning model includes: Initializing model parameters of the reinforcement learning model to obtain an action selection model; Selecting an execution action in the action space of the initialization scheduler cluster, and obtaining an execution state corresponding to the execution action; The action selection model is used to calculate a model evaluation value of the execution state, and a resource scheduling action is determined according to the model evaluation value.

4. A method for dynamically allocating computing resources as claimed in claim 3, characterized in that: The step of calculating the model evaluation value of the execution state by using the action selection model, and determining the resource scheduling action according to the model evaluation value includes: Calculating an initial evaluation value of the execution state using a model evaluation function in the action selection model; Iteratively updating the execution state according to the initial evaluation value until the number of iterative updates meets a preset number threshold, thereby obtaining an updated state; A model evaluation value corresponding to the update state is calculated, and an execution response corresponding to the model evaluation value is used as a resource scheduling action.

5. A method for dynamically allocating computing resources as claimed in claim 1, characterized in that: The step of optimizing the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model includes: Monitoring the cluster state of the initialization scheduler cluster according to the resource scheduling action; Calculating the instant reward of the cluster state using the reward function in the initialized scheduler cluster; Parameters of the reinforcement learning model are optimized according to the immediate reward to obtain a target scheduling model.

6. A method for dynamically allocating computing resources as claimed in claim 5, characterized in that: The step of optimizing parameters of the reinforcement learning model according to the instant reward to obtain a target scheduling model includes: Constructing a loss function according to the instant reward and a preset instant reward threshold; Calculating the gradient of the loss function with respect to model parameters in the reinforcement learning model; The model parameters are updated according to the gradient until the loss function is no greater than a preset loss function threshold, thereby obtaining target model parameters; The target model parameters are assigned to the model parameters to obtain a target scheduling model.

7. A method for dynamically allocating computing resources as claimed in claim 1, characterized in that: The calculating the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model includes: Calculating the optimal execution action of the resource scheduler cluster using the target scheduling model; A resource scheduling strategy of the resource scheduler cluster is generated according to the optimal execution action.

8. A computing resource dynamic allocation device, characterized in that: The device comprises: An initial environment modeling module is used to obtain a pre-built resource scheduler cluster, perform initial environment modeling on the resource scheduler cluster, and obtain an initialized scheduler cluster; A resource scheduling action selection module, used to select a resource scheduling action from the initialization scheduler cluster using a preset reinforcement learning model; A model optimization module, used to optimize the reinforcement learning model according to the resource scheduling action to obtain a target scheduling model; The resource scheduling strategy generation module is used to calculate the resource scheduling strategy of the resource scheduler cluster according to the target scheduling model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for dynamically allocating computing resources as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for dynamically allocating computing resources as described in any one of claims 1 to 7 is implemented.

Images