Distributed optimization strategy method, device and equipment based on reinforcement learning

Abstract

The invention relates to the technical field of distributed computing, and discloses a distributed optimization strategy method, device and equipment based on reinforcement learning and a storage medium, the method comprises the following steps: evaluating computing resources of each computing node in a distributed computing system to obtain an evaluation result, and executing a distributed computing task based on a distributed computing task execution list and the evaluation result. An initial task allocation result is generated, and the initial task allocation result comprises tasks in the distributed computing task execution list and target computing nodes matched with the tasks; estimating a first expected cumulative return for executing the task based on the initial task allocation result; adjusting an initial task allocation result based on available nodes in the computing nodes; estimating a second expected cumulative return for executing the task based on the adjusted initial task allocation result; determining a target task allocation result based on the first expected cumulative return and the second expected cumulative return; and performing task allocation and execution based on the target task allocation result. The problem of insufficient computing resources in the distributed computing field is solved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a distributed optimization strategy method, device and equipment based on reinforcement learning. Background Art

[0002] With the widespread application of Large Language Models (LLM), the demand for computing resources in distributed computing systems has increased dramatically, especially when processing the training of large-scale deep learning models, which places extremely high demands on computing power. The current distributed computing field involves the mixed use of homogeneous resources and heterogeneous resources. Homogeneous resources can be multi-core central processing units (CPUs), and heterogeneous resources can be graphics processing units (GPUs), tensor processing units (TPUs), or field programmable gate arrays (FPGAs). How to efficiently schedule these resources has become the key to solving computing bottlenecks. Summary of the invention

[0003] In view of this, the present invention provides a distributed optimization strategy method, device and equipment based on reinforcement learning to solve the problem of insufficient computing resources in the field of distributed computing.

[0004] In a first aspect, the present invention provides a distributed optimization strategy method based on reinforcement learning, the method comprising:

[0005] Evaluate the computing resources of each computing node in the distributed computing system to obtain an evaluation result, generate an initial task allocation result based on the distributed computing task execution list and the evaluation result, the initial task allocation result includes the tasks in the distributed computing task execution list and the target computing nodes matching the tasks; estimate the first expected cumulative return of executing the tasks based on the initial task allocation result; adjust the initial task allocation result based on the available nodes in the computing nodes; estimate the second expected cumulative return of executing the tasks based on the adjusted initial task allocation result; determine the target task allocation result based on the first expected cumulative return and the second expected cumulative return; and allocate and execute the tasks based on the target task allocation result.

[0006] In an optional implementation, generating the initial task allocation result based on the distributed computing task execution list and the evaluation result includes: generating the initial task allocation result by the following formula:

[0007]

[0008] Among them, P(T i ,C j ) represents the task T in the distributed computing task execution list i And with T i Matching target computing node C j , R(C j ) means C j Resource utilization, L(C j ) means C j The current load, U(T i ) indicates T i The execution priority of , α represents the first preset weight, β represents the second preset weight, and γ represents the weight coefficient.

[0009] In an optional implementation, determining the target task allocation result based on the first expected cumulative reward and the second expected cumulative reward includes: determining the maximum value of the first expected cumulative reward and the second expected cumulative reward, and obtaining the target task allocation result corresponding to the maximum value.

[0010] In an optional implementation, determining the target task allocation result includes: determining the target task allocation result by the following formula:

[0011]

[0012] Among them, C k represents any node among the available nodes, T i represents the tasks in the distributed computing task execution list, C j Indicates the execution T determined in the available nodes i The target computing node, Q(C j ,T i ) represents the first expected cumulative return or the second expected cumulative return, represents the maximum value.

[0013] In an optional implementation, adjusting the initial task allocation result based on the available nodes in the computing nodes; and estimating the second expected cumulative reward of executing the task based on the adjusted initial task allocation result, comprises: calculating the second expected cumulative reward by the following formula:

[0014]

[0015] Among them, s t Used to characterize the adjusted initial task allocation result, a tIt is used to characterize the task allocation process of allocating the task to the target computing node, Q(s t ,a t ) is used to characterize the t Conduct a t The second expected cumulative return is obtained as Used to represent immediate returns, a represents the learning rate, b represents the discount factor, Represents the initial task allocation result s based on subsequent adjustments t+1 The maximum second expected cumulative reward of performing possible task assignment procedures.

[0016] In an optional embodiment, determining the immediate reward includes: taking a weighted average based on the initial task allocation result, the remaining computing resources of each of the computing nodes, the completion time of executing the task, and the overload penalty of the target computing node to determine the immediate reward.

[0017] In the second aspect, the present invention provides a distributed optimization strategy device based on reinforcement learning, which includes: a pre-allocation module, which is used to evaluate the computing resources of each computing node in the distributed computing system, obtain an evaluation result, and generate an initial task allocation result based on the distributed computing task execution list and the evaluation result, wherein the initial task allocation result includes the tasks in the distributed computing task execution list and the target computing nodes matching the tasks; an allocation adjustment module, which is used to estimate the first expected cumulative reward of executing the task based on the initial task allocation result; adjust the initial task allocation result based on the available nodes in the computing nodes; estimate the second expected cumulative reward of executing the task based on the adjusted initial task allocation result; determine the target task allocation result based on the first expected cumulative reward and the second expected cumulative reward; and a scheduling module, which is used to allocate and execute the task based on the target task allocation result.

[0018] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the distributed optimization strategy method based on reinforcement learning according to the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the distributed optimization strategy method based on reinforcement learning according to the above-mentioned first aspect or any corresponding embodiment thereof.

[0020] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the distributed optimization strategy method based on reinforcement learning according to the above-mentioned first aspect or any corresponding embodiment thereof.

[0021] By evaluating the computing resources of each computing node in the distributed computing system and generating the initial task allocation results, nodes with more sufficient resources can be better allocated to high-priority distributed computing tasks, and balanced scheduling of computing resources can be achieved. Based on the first expected cumulative return and multiple second expected cumulative returns, the allocation of computing resources can be adjusted to determine the target task allocation results. This can optimize the task scheduling strategy, solve the problem of insufficient computing resources in the distributed computing field, and ensure high resource utilization and task execution efficiency. It not only simplifies the management and optimization of distributed computing for developers, but also provides a new solution for dealing with increasingly complex application scenarios and the training of large language models, and promotes technological progress in the field of deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of a distributed optimization strategy method based on reinforcement learning provided by an embodiment of the present invention is shown;

[0024] Figure 2 A schematic diagram of the distributed optimization strategy system architecture based on reinforcement learning is shown;

[0025] Figure 3 Another method flow diagram of a distributed optimization strategy based on reinforcement learning provided by an embodiment of the present invention is shown;

[0026] Figure 4 A schematic diagram of the structure of a distributed optimization strategy device based on reinforcement learning provided by an embodiment of the present invention is shown;

[0027] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] Q-Learning is a model-free reinforcement learning algorithm that can play an important role in task scheduling. It learns the optimal scheduling strategy in the system through continuous interaction with the environment, and dynamically adjusts the matching of each task and node without relying on the prior model of tasks and nodes. Especially when facing homogeneous and heterogeneous resources, Q-Learning can optimize the scheduling strategy based on parameters such as the load of computing nodes, resource utilization, and the urgency of tasks, thereby balancing the computing burden of each node and avoiding resource waste or task processing delays.

[0030] According to an embodiment of the present invention, an embodiment of a distributed optimization strategy method based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] In this embodiment, a distributed optimization strategy method based on reinforcement learning is provided, which can be used for computing nodes in a distributed computing system, such as a physical server, a virtual machine hosted by a physical server, or a container. Figure 1 A schematic diagram of the process of a distributed optimization strategy method based on reinforcement learning provided by an embodiment of the present invention is shown, Figure 1 As shown, the process includes the following steps:

[0032] Step S101, evaluate the computing resources of each computing node in the distributed computing system to obtain an evaluation result, and generate an initial task allocation result based on the distributed computing task execution list and the evaluation result. The initial task allocation result includes the tasks in the distributed computing task execution list and the target computing nodes that match the tasks.

[0033] In a large-scale distributed computing system, computing nodes include a variety of homogeneous or heterogeneous resources, including hardware resources such as CPU, GPU, TPU, and FPGA. In order to achieve efficient use of computing resources, the resource status of each computing node can be evaluated.

[0034] In this step, you can get detailed information about each computing node in the distributed computing system, including CPU performance, memory size, storage capacity, or network bandwidth, etc. Quantify the detailed information of the computing nodes, assign a weight to each resource indicator, and score each computing node according to the actual situation to get the evaluation result.

[0035] Perform a detailed analysis of the distributed computing tasks that need to be executed to determine the computing resource requirements of each task, such as whether it is CPU-intensive, memory-intensive, or input-output-intensive. Match the task requirements with the resource evaluation results of the computing nodes to determine the most suitable computing node for executing the task.

[0036] Based on the evaluation results and task requirements of the computing nodes, as well as the urgency and importance of the tasks, the tasks in the distributed computing task execution list are comprehensively sorted to generate the initial task allocation results.

[0037] Step S102, based on the initial task allocation result, estimate the first expected cumulative reward of executing the task; based on the available nodes in each computing node, adjust the initial task allocation result; based on the adjusted initial task allocation result, estimate the second expected cumulative reward of executing the task; based on the first expected cumulative reward and the second expected cumulative reward, determine the target task allocation result.

[0038] In this step, the first expected cumulative reward can be estimated based on the prediction of factors such as task execution time, resource consumption, and success probability. For the target task, available nodes that can execute the target task are obtained, and the target task is executed based on the available nodes. The second expected cumulative reward of executing the target task is estimated, and each available node calculates the second expected cumulative reward to obtain multiple second expected cumulative rewards.

[0039] Based on the first expected cumulative reward and multiple second expected cumulative rewards, the expected cumulative rewards of different nodes executing the target task can be estimated, and the expected cumulative rewards are increased, and the initial task allocation result is adjusted to obtain the target task allocation result.

[0040] Step S103: assigning and executing tasks based on the target task assignment result.

[0041] Figure 2 The following is a schematic diagram of the system architecture of the distributed optimization strategy based on reinforcement learning. Figure 2As shown, there are multiple computing nodes, and the evaluation results of the computing resources of the first computing node 21, the evaluation results of the computing resources of the second computing node 22, and the evaluation results of the computing resources of the Nth computing node 23 are obtained. The task scheduler (TaskScheduler) 24 can allocate the tasks in the distributed computing task execution list to the corresponding computing nodes one by one based on the target task allocation results, schedule the corresponding computing nodes, and calculate the tasks in the task execution list. The computing nodes can exchange information based on the network communication layer 25.

[0042] The distributed optimization strategy method based on reinforcement learning provided in this embodiment generates initial task allocation results by evaluating the computing resources of each computing node in the distributed computing system, so as to better allocate nodes with relatively sufficient resources to high-priority distributed computing tasks, realize balanced scheduling of computing resources, adjust the allocation of computing resources based on the first expected cumulative return and multiple second expected cumulative returns, determine the target task allocation result, optimize the task scheduling strategy, solve the problem of insufficient computing resources in the distributed computing field, and ensure high resource utilization and task execution efficiency. It not only simplifies the management and optimization of distributed computing by developers, but also provides a new solution for processing increasingly complex application scenarios and the training of large language models, and promotes technological progress in the field of deep learning.

[0043] The initial task allocation result can be determined based on the resource utilization and current load of each computing node in the distributed computing system. Determining the evaluation result of the computing resources of the computing nodes can better allocate nodes with more sufficient resources to high-priority tasks and achieve balanced scheduling of computing resources.

[0044] In some optional implementations, generating an initial task allocation result based on the distributed computing task execution list and the evaluation result includes:

[0045] The initial task allocation result is generated by the following formula:

[0046]

[0047] Among them, P(T i ,C j ) represents the task T in the distributed computing task execution list i And with T i Matching target computing node C j , R(C j ) means C j Resource utilization, L(C j ) means C j The current load, U(T i ) indicates T iThe execution priority of , α represents the first preset weight, β represents the second preset weight, and γ represents the weight coefficient.

[0048] In this embodiment, P(T i ,C j ) is used to guide task scheduling. j ) means C j The resource utilization rate, for example, is the current occupancy ratio of resources such as CPU or memory. The larger the resource utilization rate, the fewer available resources the computing node has. The task allocation priority of the current node can be reduced. The inverse of the resource utilization rate can be taken during the initial task allocation process to perform the initial task allocation.

[0049] L(C j ) means C j The current load, that is, the number of tasks or pressure value currently being processed by the target computing node. Similar to the aforementioned resource utilization, the higher the current load of the target computing node, the lower the priority of assigning tasks to the target computing node. The inverse of the current load can also be taken during the initial task allocation process to perform initial task allocation.

[0050] The higher the resource utilization and current load, the more likely it is that the target computing node is close to full capacity. Therefore, the priority of task allocation to the target computing node is lowered. By taking the inverse, we can ensure that tasks are allocated to nodes with light loads first.

[0051] U(T i ) indicates T i The execution priority of U(T i ) is used to characterize the urgency or importance of a service and has nothing to do with resource utilization and node load. i ) can be defined by business logic, and tasks with high execution priority will be executed faster. The urgency of certain business tasks cannot be ignored, such as real-time processing tasks, important system tasks, and user-defined priority tasks. Urgent tasks can get higher priority, even exceeding the influence of resource and load weights.

[0052] α represents the first preset weight, which is used to adjust the impact of resource utilization on the priority of task allocation. β represents the second preset weight, which is used to adjust the impact of the current load on the priority of task allocation. By adjusting the first preset weight and the second preset weight, flexible scheduling of tasks in different environments can be achieved. For example, if the first preset weight is set to a larger value, in the process of initial task allocation, it is more inclined to select nodes with lower resource utilization for priority task allocation. If the second preset weight is set to a smaller value, in the process of initial task allocation, it is more inclined to allocate tasks to nodes with lower loads.

[0053] γ represents the weight coefficient, which is used to balance the impact of task urgency in the distributed computing task execution list. The specific value of the weight coefficient can be adjusted according to business requirements and the operation status of the distributed system.

[0054] In this way, by comprehensively considering the resource usage, current load and task urgency of each computing node, the priority of task allocation in each distributed computing task execution list is dynamically calculated, thereby ensuring that the system can maximize the use of available resources and reasonably disperse tasks during the task scheduling process, avoiding load imbalance problems caused by excessive concentration of tasks on a few nodes, and at the same time ensuring that urgent tasks can be processed first.

[0055] In some optional implementations, determining the target task allocation result based on the first expected cumulative reward and the second expected cumulative reward includes: determining the maximum value of the first expected cumulative reward and the second expected cumulative reward, and obtaining the target task allocation result corresponding to the maximum value.

[0056] In this embodiment, the expected cumulative return represents the expected benefit that can be obtained by assigning a task to a certain node in the current state. By comparing the expected returns of different computing nodes for a certain task, it is possible to determine which computing node can obtain the greatest benefit by executing the task. By adjusting the initial task allocation result multiple times and optimizing the initial task allocation process, the number of second expected cumulative returns obtained is multiple. Among the first expected cumulative return and multiple second expected cumulative returns, the maximum value of the expected cumulative return is determined, and the task allocation result corresponding to the maximum value is the target task allocation result. Based on the target task allocation result, the task is assigned to the computing node with the largest expected cumulative return.

[0057] In some optional implementations, determining the target task allocation result includes:

[0058] The target task allocation result is determined by the following formula:

[0059]

[0060] Among them, C k represents any node among the available nodes, T i Represents the tasks in the distributed computing task execution list, C j Indicates the execution T determined in the available nodes i The target computing node, Q(C j ,T i ) represents the first expected cumulative return or the second expected cumulative return, that is, T i Assigned to C k The expected cumulative return is represents the maximum value, that is, for all available nodes, Ti The maximum expected cumulative reward from assigning to all available nodes.

[0061] In this embodiment, T i Assigned to the target computing node C j , including: calculation of T i The expected cumulative return Q(C j ,T i ); determine the Q(C j ,T i ) The target computing node C with the largest value j , and the task T i Assigned to C j . Q(C j ,T i ) is used to evaluate the expected cumulative return of taking an action in a specific state. The state is used to characterize the current utilization of distributed computing system resources and the status of the distributed computing task execution list, and the action is to assign the task to a computing node. k ,T i ) indicates that task T i Assigned to Node C k expected returns.

[0062] The distributed computing system calculates the task T in the distributed computing task execution list according to the current state. i Assign to any node C among all available nodes k The expected cumulative return of each available node is compared, and the target computing node C with the largest expected cumulative return is selected. j , T i Assigned to C j After the task is executed, the distributed computing system updates the expected cumulative reward based on the actual execution result, which can be the task completion time or resource consumption, etc. This can optimize future scheduling strategies.

[0063] By selecting the target computing node with the largest expected cumulative return, we can ensure that tasks are assigned to the most appropriate resources, thereby improving resource utilization. The distributed computing system can dynamically adjust task allocation based on real-time resource status and task requirements to adapt to load changes. It can optimize the allocation of tasks and resources, reduce task waiting time and execution time, and improve the overall distributed computing system performance.

[0064] After obtaining the Q value, the system needs to actually assign the task to the computing node. k ,T i ) indicates that task T i Assigned to compute node Ck The expected cumulative returns that can be brought.

[0065] The following is an exemplary description of this implementation. For task T1, there are three available computing nodes C1, C2 and C3. After calculation, the Q values ​​are as follows: Q(C1, T1) = 0.75, Q(C2, T1) = 0.85, Q(C3, T1) = 0.65. According to the formula, Task T1 may be assigned to the target computing node C2 for execution.

[0066] Through the above task allocation strategy, the distributed computing system can intelligently allocate tasks to the most suitable target computing nodes, achieve optimal utilization of computing resources and efficient execution of tasks. It has self-learning and adaptive capabilities, and can dynamically adjust the scheduling strategy according to the changes in the state of the distributed computing system, thereby improving the performance and stability of the distributed system.

[0067] The use of resource utilization evaluation, reinforcement learning algorithm optimization, and real-time task scheduling strategies can effectively improve the resource utilization and task execution efficiency of distributed systems. By dynamically adapting to different computing loads and task requirements, it can provide a more flexible and efficient task scheduling solution for large-scale distributed systems, thereby achieving continuous optimization of the overall performance of distributed computing systems.

[0068] In some optional implementations, adjusting the initial task allocation result based on the available nodes in the computing nodes; and estimating the second expected cumulative reward of executing the task based on the adjusted initial task allocation result includes:

[0069] The second expected cumulative return is calculated using the following formula:

[0070]

[0071] Among them, s t Used to characterize the adjusted initial task allocation result, a t It is used to characterize the task allocation process of allocating tasks to target computing nodes, Q(s t ,a t ) is used to characterize the t Conduct a t The second expected cumulative return is Used to represent immediate returns, a represents the learning rate, b represents the discount factor, Represents the initial task allocation result s based on subsequent adjustments t+1 The maximum second expected cumulative reward of performing possible task assignment procedures.

[0072] Q(s t ,a t ) means in state st Next, take action a t The expected cumulative return that can be brought. By continuously updating the Q value, the distributed computing system can optimize the scheduling strategy and find the optimal task allocation path. t Indicates the current state of the system, for example, the resource utilization of the node. t It represents the action of assigning tasks, for example, assigning tasks to a certain node. Q-learning gradually learns the best task scheduling strategy by updating the Q value.

[0073] In this implementation, in order to further optimize the task scheduling strategy, the reinforcement learning algorithm Q-learning is used. In this method, the state of each computing node is represented by resource utilization and current load, and the action is the decision to schedule a task to a certain node. The system continuously optimizes the scheduling strategy through the reinforcement learning algorithm, thereby adjusting the task allocation in real time during operation to ensure optimal resource utilization and task execution efficiency.

[0074] Q-learning is a model-free reinforcement learning algorithm that is mainly used to solve the optimal strategy in the Markov decision process (MDP). In the task scheduling scenario, the Q-learning algorithm estimates the expected cumulative reward (Q value) of each state after taking a specific action by learning on the state-action pair. Through repeated updates and learning, the distributed computing system gradually optimizes the task scheduling strategy so that it can make the best scheduling decisions under different states.

[0075] In task scheduling, the state s t Including, the resource utilization R(C j )、Load L(C j ), task priority U(T i ) and task queues. This information reflects the health and availability of the nodes, helping the distributed computing system determine which nodes are more suitable for receiving new tasks.

[0076] An action is a task scheduling decision made by a distributed computing system in a certain state. In Q-learning, an action a t That is, task T i Assigned to a specific computing node C j This decision is made based on the current state and the Q-values ​​of each possible action.

[0077] Status t is the initial task allocation result P(T i ,C j ). In the Q-learning reinforcement learning algorithm, the state s tRepresents the current task T i and node C j Combination of t It is task T i Assigned to node C at a certain moment j The state can be regarded as the initial task allocation result P(T i ,C j ) is further dynamical evolution. State s t Directly with the initial task allocation result P(T i ,C j ) is associated with the task allocation strategy through continuous learning of Q-learning. This combination can ensure that urgent tasks are given priority while effectively utilizing resources in the system.

[0078] Action a t That is, task T i Assign to a node C j This can be adjusted based on the initial task allocation results so that high-priority tasks are allocated to nodes with better resources first.

[0079] Q(s t ,a t ) is used to indicate that in state s t Next, select action a t The Q value of , which is the expected cumulative return of the current estimate. Used to represent immediate rewards, indicating that the distributed computing system takes action a t After receiving the immediate feedback. The definition of values ​​includes task priority, node resource usage, and task execution effect.

[0080] a represents the learning rate, ranging from 0 to 1. The learning rate determines the speed of Q value update. The larger the learning rate, the faster the distributed computing system learns new experience. b represents the discount factor, ranging from 0 to 1. The discount factor is used to measure the present value of future returns. The closer the discount factor is to 1, the greater the importance of future returns. Regarding the selection of learning rate and discount factor, a higher learning rate enables the distributed computing system to adapt to new information faster, but may cause policy instability; a lower learning rate can ensure policy stability, but the learning speed is slower. Choosing an appropriate learning rate can balance the learning speed and policy stability. The choice of discount factor reflects the importance of the distributed computing system to future returns. A higher discount factor means that the distributed computing system will pay more attention to future returns, while a lower discount factor pays more attention to current returns.

[0081] max a Q(s t+1,a′) means in the next state s t+1 The maximum Q value among all possible actions under this condition. This is not a prediction of the future, but a cumulative learning on different state-action pairs to gradually find the optimal action and update the Q value in the current state.

[0082] In this way, past experience is relied upon to gradually optimize future action selection. a′ Q(s t+1 ,a′) represents the maximum expected reward of all possible actions we can take in the next state. This term is estimated by constantly trying and updating the Q value, rather than predicting the future. Although s t+1 is the next state, but we do not really know its result in advance, but estimate the possible future returns through the discount factor b. This discount is to measure the importance of future returns, and the ultimate goal is to allow the distributed computing system to make a trade-off between long-term and short-term benefits.

[0083] In an optional implementation, determining the immediate reward includes: determining the immediate reward by taking a weighted average based on the initial task allocation result, the remaining computing resources of each computing node, the completion time of the execution task, and the overload penalty of the target computing node.

[0084] In this embodiment, the immediate return can be determined by the following formula:

[0085] Node overload penalty

[0086] Among them, x1, x2, x3 and x4 are adjustable weights used to balance the impact of different factors on the immediate return. x1 is used to balance the impact of the initial task allocation result on the immediate return; x2 is used to balance the impact of the remaining computing resources of each computing node on the immediate return. The higher the resource utilization, the higher the immediate return; x3 is used to balance the impact of the task completion time on the immediate return. The faster the task is completed, the higher the immediate return; x4 is used to balance the impact of excessive computing node load on the immediate return. By balancing the load of each computing node, the immediate return increases when some computing nodes are avoided from being overloaded. If there is a computing node overload, the immediate return decreases.

[0087] Although maximizing the Q value can optimize the execution of a single task, it is necessary to prevent some nodes from being overloaded. A load penalty term can be added to the Q value calculation. In order to avoid local optimal solutions, the distributed computing system can adopt an ∈-greedy strategy to explore other nodes with a certain probability.

[0088] By setting up immediate rewards, the distributed computing system not only considers the priority of tasks during scheduling, but also balances the resource consumption of tasks and the resource load of nodes, thereby avoiding excessive concentration of tasks on certain nodes. Balanced utilization of computing resources can be achieved. By repeatedly applying this update process, the system can gradually learn the optimal task scheduling strategy, making each task assignment as close to the optimal solution as possible.

[0089] Through the above Q-value update process, the distributed computing system can gradually learn the optimal task scheduling strategy under various conditions. This strategy will help the distributed computing system to adjust task allocation in real time during actual operation to ensure optimal resource utilization and task execution efficiency. This Q-learning-based method is particularly suitable for complex distributed environments because it can adaptively optimize scheduling strategies based on dynamic changes in the environment.

[0090] Figure 3 FIG. 2 shows another method flow diagram of a distributed optimization strategy based on reinforcement learning provided by an embodiment of the present invention. Figure 3 As shown, the distributed optimization strategy method based on reinforcement learning includes: step S301, obtaining resource utilization information and load information of the computing node; step S302, according to P(T i ,C j ) to generate the initial task allocation result; Step S303, determine whether there is a task with a high allocation priority, if so, go to step S304, if not, go to step S305; Step S304, give priority to scheduling tasks with high allocation priority, and update the status after the task is completed; Step S305, allocate regular tasks.

[0091] In this embodiment, a device for a distributed optimization strategy based on reinforcement learning is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0092] This embodiment provides a device for a distributed optimization strategy based on reinforcement learning. Figure 4 The schematic diagram of the structure of the device of the distributed optimization strategy based on reinforcement learning provided by the embodiment of the present invention is shown as follows: Figure 4 As shown, including:

[0093] The pre-allocation module 401 is used to evaluate the computing resources of each computing node in the distributed computing system, obtain the evaluation results, and generate the initial task allocation results based on the distributed computing task execution list and the evaluation results. The initial task allocation results include the tasks in the distributed computing task execution list and the target computing nodes that match the tasks.

[0094] The allocation adjustment module 402 is used to estimate the first expected cumulative reward of executing the task based on the initial task allocation result; adjust the initial task allocation result based on the available nodes in the computing node; estimate the second expected cumulative reward of executing the task based on the adjusted initial task allocation result; and determine the target task allocation result based on the first expected cumulative reward and the second expected cumulative reward.

[0095] The scheduling module 403 is used to allocate and execute tasks based on the target task allocation result.

[0096] In some optional embodiments, the pre-allocation module includes:

[0097] The pre-allocation unit is used to generate an initial task allocation result based on the distributed computing task execution list and evaluation results, including:

[0098] The initial task allocation result is generated by the following formula:

[0099]

[0100] Among them, P(T i ,C j ) represents the task T in the distributed computing task execution list i And with T i Matching target computing node C j , R(C j ) means C j Resource utilization, L(C j ) means C j The current load, U(T i ) indicates T i The execution priority of , α represents the first preset weight, β represents the second preset weight, and γ represents the weight coefficient.

[0101] In some optional implementations, the allocation adjustment module includes:

[0102] The first allocation adjustment unit is used to determine the target task allocation result based on the first expected cumulative return and the second expected cumulative return, including: determining the maximum value of the first expected cumulative return and the second expected cumulative return, and obtaining the target task allocation result corresponding to the maximum value.

[0103] In some optional implementations, the allocation adjustment module further includes:

[0104] The second allocation adjustment unit is used to determine the target task allocation result, including:

[0105] The target task allocation result is determined by the following formula:

[0106]

[0107] Among them, C k represents any node among the available nodes, T i Represents the tasks in the distributed computing task execution list, C j Indicates the execution T determined in the available nodes i The target computing node, Q(C j ,T i ) represents the first expected cumulative return or the second expected cumulative return, Indicates the maximum value.

[0108] In some optional implementations, the allocation adjustment module further includes:

[0109] The third allocation adjustment unit is used to adjust the initial task allocation result based on the available nodes in the computing nodes; and estimate the second expected cumulative return of executing the task based on the adjusted initial task allocation result, including:

[0110] The second expected cumulative return is calculated using the following formula:

[0111]

[0112] Among them, s t Used to characterize the adjusted initial task allocation result, a t It is used to characterize the task allocation process of allocating tasks to target computing nodes, Q(s t ,a t ) is used to characterize the t Conduct a t The second expected cumulative return is Used to represent immediate returns, a represents the learning rate, b represents the discount factor, Represents the initial task allocation result s based on subsequent adjustments t+1 The maximum second expected cumulative reward of performing possible task assignment procedures.

[0113] In some optional implementations, the third allocation adjustment unit includes:

[0114] The third allocation adjustment subunit, used to determine the immediate return, includes:

[0115] Based on the initial task allocation results, the remaining computing resources of each computing node, the completion time of the execution task and the overload penalty of the target computing node, a weighted average is taken to determine the immediate reward.

[0116] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0117] The device of the distributed optimization strategy based on reinforcement learning in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0118] The device of the distributed optimization strategy based on reinforcement learning provided by the embodiment of the present invention can realize dynamic scheduling of resources and task priority allocation through the adaptability of reinforcement learning. It solves the dynamic scheduling problem in distributed computing, especially for the training requirements of large-scale deep learning models. Through Q-Learning automatic learning and optimization of task scheduling strategies, the distributed computing system can adaptively adjust the allocation of computing resources and improve the efficiency of task execution.

[0119] The embodiment of the present invention also provides a computer device having the above Figure 4 The apparatus of the distributed optimization strategy based on reinforcement learning is shown.

[0120] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0121] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0122] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0123] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0125] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0126] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0127] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0128] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0129] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A distributed optimization strategy method based on reinforcement learning, characterized in that: include: Evaluate the computing resources of each computing node in the distributed computing system to obtain an evaluation result, and generate an initial task allocation result based on the distributed computing task execution list and the evaluation result, wherein the initial task allocation result includes the tasks in the distributed computing task execution list and the target computing nodes matching the tasks; Based on the initial task allocation result, estimating a first expected cumulative reward of executing the task; Adjusting the initial task allocation result based on available nodes in the computing nodes; Based on the adjusted initial task allocation result, estimating a second expected cumulative reward of executing the task; based on the first expected cumulative reward and the second expected cumulative reward, determining a target task allocation result; Based on the target task allocation result, the task is allocated and executed.

2. The distributed optimization strategy method based on reinforcement learning according to claim 1, characterized in that: The generating of the initial task allocation result based on the distributed computing task execution list and the evaluation result includes: The initial task allocation result is generated by the following formula: Among them, P(T i ,C j ) represents the task T in the distributed computing task execution list i And with T i Matching target computing node C j , R(C j ) means C j Resource utilization, L(C j ) means C j The current load, U(T i ) indicates T i The execution priority of , α represents the first preset weight, β represents the second preset weight, and γ represents the weight coefficient.

3. The distributed optimization strategy method based on reinforcement learning according to claim 1 or 2, characterized in that: The determining the target task allocation result based on the first expected cumulative reward and the second expected cumulative reward includes: Determine a maximum value between the first expected cumulative reward and the second expected cumulative reward, and obtain the target task allocation result corresponding to the maximum value.

4. The distributed optimization strategy method based on reinforcement learning according to claim 3 is characterized in that: The step of determining the target task allocation result includes: The target task allocation result is determined by the following formula: Among them, C k represents any node among the available nodes, T i represents the tasks in the distributed computing task execution list, C j Indicates the execution T determined in the available nodes i The target computing node, Q(C j ,T i ) represents the first expected cumulative return or the second expected cumulative return, represents the maximum value.

5. The distributed optimization strategy method based on reinforcement learning according to claim 1, characterized in that: adjusting the initial task allocation result based on available nodes in the computing nodes; Based on the adjusted initial task allocation result, estimating a second expected cumulative reward of executing the task, including: The second expected cumulative return is calculated by the following formula: Among them, s t Used to characterize the adjusted initial task allocation result, a t It is used to characterize the task allocation process of allocating the task to the target computing node, Q(s t ,a t ) is used to characterize the t Conduct a t The second expected cumulative return is obtained as Used to represent immediate returns, a represents the learning rate, b represents the discount factor, Represents the initial task allocation result s based on subsequent adjustments t+1 The maximum second expected cumulative reward of performing the possible task assignment process.

6. The distributed optimization strategy method based on reinforcement learning according to claim 1, characterized in that: Determining the immediate return includes: The immediate reward is determined by taking a weighted average based on the initial task allocation result, the remaining computing resources of each computing node, the completion time of executing the task, and the overload penalty of the target computing node.

7. A distributed optimization strategy device based on reinforcement learning, characterized in that: include: A pre-allocation module is used to evaluate the computing resources of each computing node in the distributed computing system, obtain an evaluation result, and generate an initial task allocation result based on the distributed computing task execution list and the evaluation result, wherein the initial task allocation result includes the tasks in the distributed computing task execution list and the target computing nodes matching the tasks; an allocation adjustment module, configured to estimate a first expected cumulative reward for executing the task based on the initial task allocation result; Adjusting the initial task allocation result based on available nodes in the computing nodes; Based on the adjusted initial task allocation result, estimating a second expected cumulative reward of executing the task; based on the first expected cumulative reward and the second expected cumulative reward, determining a target task allocation result; The scheduling module is used to allocate and execute the tasks based on the target task allocation result.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the distributed optimization strategy method based on reinforcement learning described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the distributed optimization strategy method based on reinforcement learning according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the distributed optimization strategy method based on reinforcement learning according to any one of claims 1 to 6.

Images