Resource scheduling method and device and distributed system
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-26
Smart Images
Figure CN122086546A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of resource scheduling technology, and in particular to a resource scheduling method, apparatus and distributed system. Background Technology
[0002] In current distributed systems, the resource scheduling strategy for tasks is mainly handled by dedicated resource scheduling nodes / functional modules.
[0003] Based on the current resource status of all nodes in the system and the resource occupancy of currently running tasks, the resource scheduling node / functional module allocates the necessary computing resources (including central processing unit (CPU), memory, storage and other computer hardware resources) to new tasks or tasks in execution.
[0004] However, dedicated resource scheduling nodes / functional modules are prone to becoming single points of failure and have high communication latency in large-scale clusters. Summary of the Invention
[0005] To address the above problems, the present invention provides a resource scheduling method, apparatus, and distributed system.
[0006] This invention provides a resource scheduling method, applicable to any task agent in a distributed system, comprising: When a user-submitted task is bound, obtain the available resource information of the nodes in the distributed system; Based on the execution requirements of the task and the available resource information of the nodes in the distributed system, determine the resource allocation strategy for the task; Resource requests are initiated to the nodes of the distributed system according to the resource allocation strategy in order to execute the task.
[0007] According to a resource scheduling method provided by the present invention, each node in the distributed system is deployed with a node agent, and each node agent is used to monitor resource changes of the node it belongs to. The step of obtaining the available resource information of the nodes in the distributed system includes: Broadcast a resource request message to each of the node agents, and receive a resource quote message returned by each of the node agents based on the resource request message; Obtain the available resource information of each node from the resource quote message returned by each node agent.
[0008] According to a resource scheduling method provided by the present invention, the available resource information includes resource commitment and timeout constraint information, and the execution requirements include resource demand, task priority and resource bidding strategy; The step of determining the resource allocation strategy for the task based on the task's execution requirements and the available resource information of the nodes in the distributed system includes: Based on the resource demand, task priority, resource bidding strategy, resource commitment amount and timeout constraint information of each node, multiple candidate resource allocation strategies and task execution prediction results corresponding to each candidate resource allocation strategy are generated. Based on the task execution prediction results corresponding to each of the candidate resource allocation strategies, the resource allocation strategy for the task is determined from the plurality of candidate resource allocation strategies.
[0009] According to a resource scheduling method provided by the present invention, the task execution prediction result includes task delay, resource matching degree, energy consumption cost, and conflict penalty; The step of determining the resource allocation strategy for the task from the plurality of candidate resource allocation strategies based on the task execution prediction result corresponding to each candidate resource allocation strategy includes: For each candidate resource allocation strategy, the benefit value of the candidate resource allocation strategy is calculated based on the task delay, resource matching degree, energy consumption cost and conflict penalty corresponding to the candidate resource allocation strategy. The candidate resource allocation strategy with the highest return value shall be used as the resource allocation strategy for the task.
[0010] According to a resource scheduling method provided by the present invention, the benefit value of the candidate resource allocation strategy is calculated according to the following formula: Ri = α·T1 + β·T2 - γ·T3 - δ·T4 Where Ri is the revenue value of the i-th candidate resource allocation strategy, T1 is the task delay, T2 is the resource matching degree, T3 is the energy consumption cost, T4 is the conflict penalty, and α, β, γ and δ are all revenue weight parameters.
[0011] According to a resource scheduling method provided by the present invention, the method further includes: If the task is completed or times out, the revenue weight parameter is updated based on the task's delay, resource matching degree, and energy consumption cost.
[0012] According to a resource scheduling method provided by the present invention, the method further includes: If the resource allocation strategy for the new task submitted by the user conflicts with the resource allocation strategy for the task, the task priority in the execution requirements of the task will be adjusted according to a preset strategy. Based on the updated execution requirements of the task and the available resource information of the nodes in the distributed system, the resource allocation strategy for the task is re-determined; According to the redefined resource allocation strategy for the task, resource requests are initiated to the nodes of the distributed system to continue executing the task.
[0013] According to a resource scheduling method provided by the present invention, the method further includes: If a faulty node occurs in the distributed system and the faulty node provides resources for the task, obtain the available resource information of other nodes in the distributed system besides the faulty node. Based on the execution requirements of the task and the available resource information of other nodes in the distributed system excluding the faulty node, the resource allocation strategy for the task is re-determined. Resource requests are initiated to the nodes of the distributed system according to the redefined resource allocation strategy for the task, in order to continue executing the task.
[0014] The present invention also provides a resource scheduling device, applicable to any task agent in a distributed system, the device comprising: The acquisition module is used to acquire available resource information of the nodes in the distributed system when a user-submitted task is bound to it. The determination module is used to determine the resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system. The processing module is used to initiate resource requests to the nodes of the distributed system according to the resource allocation strategy in order to execute the task.
[0015] The present invention also provides a distributed system, including multiple nodes, multiple task agents, and agent management service; The intelligent agent management service is used to configure a corresponding task intelligent agent for the task submitted by the user, and to bind the task to the task intelligent agent; The task agent is configured to, when bound to a user-submitted task, obtain available resource information of the nodes in the distributed system; determine the resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system; and initiate resource requests to the nodes in the distributed system according to the resource allocation strategy to execute the task.
[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the resource scheduling methods described above.
[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the resource scheduling method as described above.
[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the resource scheduling methods described above.
[0019] The resource scheduling method, apparatus, and distributed system provided by this invention bind a task agent to a user-submitted task, utilize the task agent to obtain the available resource information of the nodes in the distributed system, and determine the resource allocation strategy for the task based on the task's execution requirements and the available resource information of the nodes in the distributed system. The task is then executed by initiating resource requests to the nodes in the distributed system according to the resource allocation strategy. This eliminates the need to set up a dedicated resource scheduling node / functional module in the distributed system, thereby avoiding single points of failure. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0021] Figure 1 This is a schematic diagram of the structure of a distributed system provided in an embodiment of the present invention.
[0022] Figure 2 This is a schematic diagram of the architecture of the task intelligence agent provided in an embodiment of the present invention.
[0023] Figure 3 This is one of the flowcharts illustrating the resource scheduling method provided in this embodiment of the invention.
[0024] Figure 4 This is the second flowchart of the resource scheduling method provided in the embodiments of the present invention.
[0025] Figure 5 This is the third flowchart of the resource scheduling method provided in this embodiment of the invention.
[0026] Figure 6 This is the fourth flowchart of the resource scheduling method provided in this embodiment of the invention.
[0027] Figure 7 This is a schematic diagram of the resource scheduling device provided in an embodiment of the present invention.
[0028] Figure 8This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0030] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0031] The following is combined Figures 1 to 8 The present invention describes the resource scheduling method, apparatus, and distributed system.
[0032] Figure 1 This is a schematic diagram of the structure of the distributed system provided in the embodiments of the present invention, such as... Figure 1 As shown, the distributed system includes multiple nodes, multiple task agents, and agent management services; The intelligent agent management service is used to configure a corresponding task intelligent agent for the task submitted by the user, and to bind the task to the task intelligent agent; The task agent is configured to, when bound to a user-submitted task, obtain available resource information of the nodes in the distributed system; determine the resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system; and initiate resource requests to the nodes in the distributed system according to the resource allocation strategy to execute the task.
[0033] In addition, see Figure 1 The distributed system also includes a job submission service, and each node has a node agent configured.
[0034] Specifically, a node can be a physical or virtual, long-term existing resource unit.
[0035] The job submission service can be a highly simplified version of the traditional scheduler, responsible for receiving user-submitted tasks and providing a basic communication framework.
[0036] The agent management service supports the creation, destruction, registration, discovery, and lifecycle management of task agents.
[0037] Node agents (such as node agent 1, node agent 2, node agent M, etc.) are the "representatives" of resource providers, continuously monitoring the resource status of their respective nodes (CPU, memory, disk input / output (IO), network, load, temperature, etc.), and representing the "interests" of their nodes (such as avoiding overload, maximizing utilization, and obtaining task rewards).
[0038] Node agents are created when the distributed system starts. Each node (e.g., node 1, node 2, node M, etc.) creates its own node agent upon startup and registers it with the agent management service, announcing the existence of its corresponding node agent. The binding relationship between nodes and node agents (e.g., associating the node agent's identifier (ID) with the node's ID) is determined during registration and remains unchanged throughout the node's lifecycle. Node agents can perceive the state of their respective nodes and decide whether to accept new tasks, report load, and participate in resource pricing, etc.
[0039] Task agents (such as Task Agent 1, Task Agent 2, Task Agent N, etc.) are resource users. They continuously initiate and adjust node resource requests, representing the "interests" of the bound task (such as quickly obtaining the required resources, meeting the deadline, and minimizing costs).
[0040] Task agents are dynamically generated on demand when a user submits a task, are uniquely bound to the task, and are destroyed when the task ends. Users submit task descriptions (including the task's directed acyclic graph, resource requirements, constraints, etc.) to the distributed system's job submission service through a task submission portal (such as a command-line interface (CLI) tool, a World Wide Web (Web) application programming interface (API), or a job scheduler client). The job submission service then notifies the agent management service to generate the task agent.
[0041] It should be understood that, Figure 1 The structure shown is only a schematic diagram of a distributed system; the distributed system may also include components such as... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.
[0042] Figure 2 This is a schematic diagram of the architecture of the task intelligence agent provided in an embodiment of the present invention. See also... Figure 2 Please refer to Figure 2 The architecture of the task intelligence agents (task intelligence agent A and task intelligence agent B) consists of four layers: resource perception layer, game decision layer, collaborative optimization layer, and execution feedback layer.
[0043] The resource awareness layer can collect the resource status of nodes (such as distributed resources, CPU / Graphics Processing Unit (GPU) / bandwidth utilization, memory, network, disk, task queue length, etc.) and the attributes of the bound tasks (priority, deadline, resource requirements).
[0044] The game decision layer generates strategies based on the acquired information (partially observable Markov decision process (POMDP) model), competes for resources through non-cooperative games, and optimizes strategy weights using Q-learning.
[0045] The collaborative optimization layer employs a federated parameter synchronization mechanism (parameter synchronization), exchanging policy gradients through lightweight communication (such as the Gossip protocol) to avoid central node bottlenecks and achieve global convergence. Figure 2 Task agent A and task agent B interact through a collaborative optimization layer.
[0046] The execution feedback layer dynamically adjusts the resource allocation results, introduces a time penalty factor to compensate for timed-out tasks, and updates task priorities in real time through Temporal-Difference Learning (TD-learning).
[0047] The role or function of the task agent can be referred to in relation to the resource scheduling method described below, and will not be repeated here.
[0048] The distributed system provided by this invention binds a task agent to the task submitted by the user, uses the task agent to obtain the available resource information of the nodes in the distributed system, and determines the resource allocation strategy of the task according to the execution requirements of the task and the available resource information of the nodes in the distributed system. The system then initiates resource requests to the nodes in the distributed system according to the resource allocation strategy to execute the task. This eliminates the need to set up a dedicated resource scheduling node / functional module in the distributed system, thereby avoiding single points of failure.
[0049] Figure 3 This is one of the flowcharts illustrating the resource scheduling method provided in this embodiment of the invention, such as... Figure 3 As shown, the method is applied to any task agent in a distributed system, including steps 301 to 303.
[0050] Step 301: If a user-submitted task is bound, obtain the available resource information of the nodes in the distributed system.
[0051] Specifically, the tasks submitted by users can be artificial intelligence (AI), real-time video processing, etc. Each task is bound to a unique task agent, which carries the task attributes of the corresponding task (obtained through the resource awareness layer). The task attributes include at least one of the following: task type (e.g., CPU / GPU intensive), resource requirements (e.g., number of CPU cores, memory, bandwidth), quality of service (QoS) requirements (e.g., deadline), dynamic attributes, initial priority weight, and historical execution records.
[0052] Each node agent is used to monitor resource changes on its node (such as at least one of CPU / GPU utilization, remaining memory, and network bandwidth). The task agent can obtain information about the available resources of each node by interacting with the node agents corresponding to each node in the distributed system.
[0053] Each node in the distributed system is equipped with a node agent, and each node agent is used to monitor resource changes on its node; obtaining the available resource information of the nodes in the distributed system includes: Broadcast a resource request message to each of the node agents, and receive a resource quote message returned by each of the node agents based on the resource request message; Obtain the available resource information of each node from the resource quote message returned by each node agent.
[0054] That is, the task agent broadcasts a resource request message to each node agent, receives a resource quote message returned by each node agent based on the resource request message, and obtains the available resource information of each node from the resource quote message returned by each node agent.
[0055] Step 302: Determine the resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system.
[0056] Specifically, the available resource information includes the resource commitment amount and timeout constraint information, and the execution requirements include the resource demand amount, task priority, and resource bidding strategy.
[0057] Specifically, the execution requirements include at least one of resource demand, task priority, and resource bidding strategy. The resource bidding strategy refers to the bidding weight of the task agent for scarce computing resources.
[0058] Optionally, determining the resource allocation strategy for the task based on its execution requirements and the available resource information of the nodes in the distributed system includes: Based on the resource demand, task priority, resource bidding strategy, resource commitment amount and timeout constraint information of each node, multiple candidate resource allocation strategies and task execution prediction results corresponding to each candidate resource allocation strategy are generated. Based on the task execution prediction results corresponding to each of the candidate resource allocation strategies, the resource allocation strategy for the task is determined from the plurality of candidate resource allocation strategies.
[0059] In practical applications, the task agent generates multiple candidate resource allocation strategies and corresponding task execution prediction results based on resource demand, task priority, resource bidding strategy, resource commitment and timeout constraint information of each node. Based on the task execution prediction results of each candidate resource allocation strategy, the task resource allocation strategy is determined from all candidate resource allocation strategies.
[0060] Specifically, the task execution prediction result refers to the outcome of executing the corresponding task if resources are obtained according to the candidate resource allocation strategy, including task delay, resource matching degree, energy consumption cost, and conflict penalty. Among them, the conflict penalty refers to the penalty value added by Fenny for multiple task agents competing for the same resource and failing to be allocated the resource due to timeout.
[0061] Optionally, determining the resource allocation strategy for the task from the plurality of candidate resource allocation strategies based on the task execution prediction result corresponding to each candidate resource allocation strategy includes: For each candidate resource allocation strategy, the benefit value of the candidate resource allocation strategy is calculated based on the task delay, resource matching degree, energy consumption cost and conflict penalty corresponding to the candidate resource allocation strategy. The candidate resource allocation strategy with the highest return value shall be used as the resource allocation strategy for the task.
[0062] In practical applications, the task agent calculates the reward value of each candidate resource allocation strategy based on the task delay, resource matching degree, energy consumption cost, and conflict penalty corresponding to each candidate resource allocation strategy, and selects the candidate resource allocation strategy with the maximum reward value as the resource allocation strategy for the task.
[0063] Optionally, the payoff value of the candidate resource allocation strategy is calculated according to the following formula: Ri = α·T1 + β·T2 - γ·T3 - δ·T4 Where Ri is the revenue value of the i-th candidate resource allocation strategy, T1 is the task delay, T2 is the resource matching degree, T3 is the energy consumption cost, T4 is the conflict penalty, and α, β, γ and δ are all revenue weight parameters.
[0064] The profit weight parameters α, β, γ, and δ are dynamically adjusted through Q-learning and stored in the local Q table.
[0065] Step 303: Initiate resource requests to the nodes of the distributed system according to the resource allocation strategy to execute the task.
[0066] After obtaining the resource allocation strategy, resource requests can be initiated to the nodes of the distributed system in order to execute tasks.
[0067] The resource scheduling method provided by this invention binds a task agent to the task submitted by the user, uses the task agent to obtain the available resource information of the nodes in the distributed system, and determines the resource allocation strategy of the task according to the execution requirements of the task and the available resource information of the nodes in the distributed system. The method then initiates a resource request to the nodes in the distributed system according to the resource allocation strategy to execute the task. This eliminates the need to set up a dedicated resource scheduling node / functional module in the distributed system, thereby avoiding single points of failure.
[0068] exist Figure 3 Based on the above, see Figure 4 , Figure 4 This is a second flowchart illustrating the resource scheduling method provided in this embodiment of the invention, including: Step 401: If the task is completed or timed out, update the revenue weight parameter according to the task delay, resource matching degree and energy consumption cost.
[0069] In practical applications, after a task is completed or times out, the task agent collects the task execution results, including task delay, resource matching degree, and energy consumption cost, and updates the benefit weight parameters α, β, γ, and δ stored in the local Q table using Q-learning. This enables any agent in the distributed system to choose the strategy that maximizes long-term benefits when determining the resource allocation strategy for the corresponding task.
[0070] The Q-learning update satisfies the following formula: Q(s,a)←Q(s,a)+η[R+λ (s′,a′)-Q(s,a)] Where η is the learning rate, λ is the discount factor, R is the immediate reward, Q() is Q-learning, and s and a are the parameters being updated. (s′,a′) is the estimate of the largest Q value among all parameters a′ under parameter s′.
[0071] Since each task is bound to a corresponding task agent, and each task agent acts as a participant in the resource game, in this embodiment of the invention, the gradient descent method can be used to gradually converge each task agent to Nash equilibrium, thereby avoiding excessive competition for resources.
[0072] To reduce communication overhead, node agents can interact with each other and with task agents via the Gossip protocol.
[0073] Any task agent can ensure a balance between local optimization and global stability by synchronizing parameters with other task agents in a federated manner, thus avoiding policy oscillations (such as multiple tasks competing for the same resource at the same time).
[0074] Federal-style parameters synchronously satisfy the following formula: θi←θi+ϵ·1 / N∑j∈Neighbors(θj-θi) Where θi is the policy parameter of task agent i, ϵ is the collaborative learning rate, N is the number of neighboring task agents of task agent i, Neighbors represents the set of neighboring task agents, and θj is the policy parameter of neighboring task agent j in the set of neighboring task agents.
[0075] Because resource conflicts may exist between tasks, therefore, in Figure 4 Based on the above, see Figure 5 , Figure 5 This is the third flowchart of the resource scheduling method provided in the embodiment of the present invention. The resource scheduling method provided in the embodiment of the present invention also includes steps 501 to 503.
[0076] Step 501: If the resource allocation strategy of the new task submitted by the user conflicts with the resource allocation strategy of the task, the task priority in the execution requirements of the task shall be adjusted according to the preset strategy.
[0077] The preset strategy can be to use TD-learning to adjust task weights in real time. Understandably, the higher the task weight, the higher the priority.
[0078] TD-learning adjusts task weights in real time according to the following formula: wt←wt-1+ζ⋅(Ractual-Rpredicted) Where ζ is the learning rate, Ractual is the actual benefit feedback, Rpredicted is the predicted beneficial feedback, and wt is the task weight.
[0079] Step 502: Based on the updated execution requirements of the task and the available resource information of the nodes in the distributed system, redetermine the resource allocation strategy for the task.
[0080] Step 503: In accordance with the redefined resource allocation strategy for the task, initiate resource requests to the nodes of the distributed system to continue executing the task.
[0081] In this embodiment of the invention, the node agent allocates resources to each task according to the resource allocation strategy of each task received. If two tasks have resource conflicts, the node agent can allow the timed-out task to take priority to seize the corresponding resources, such as the vehicle network emergency task seizing bandwidth.
[0082] Since a node failure in a distributed system will affect the tasks executed by that node, therefore, in Figure 4 Based on the above, see Figure 6 , Figure 6 This is the fourth flowchart of the resource scheduling method provided in the embodiment of the present invention. The resource scheduling method provided in the embodiment of the present invention also includes steps 601 to 603.
[0083] Step 601: If a faulty node occurs in the distributed system and the faulty node provides resources for the task, obtain the available resource information of other nodes in the distributed system besides the faulty node.
[0084] Step 602: Based on the execution requirements of the task and the available resource information of other nodes in the distributed system excluding the faulty node, redetermine the resource allocation strategy for the task; Step 603: Initiate resource requests to the nodes of the distributed system according to the redefined resource allocation strategy for the task, in order to continue executing the task.
[0085] The resource scheduling device provided by the present invention is described below. The resource scheduling device described below and the resource scheduling method described above can be referred to in correspondence.
[0086] Figure 7 This is a schematic diagram of the resource scheduling device provided in this embodiment of the invention. It should be noted that the resource scheduling device provided in this embodiment has the same basic principle and technical effects as the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. See also... Figure 7 The resource scheduling device 700 is applied to any task agent in a distributed system, including: The acquisition module 701 is used to acquire the available resource information of the nodes of the distributed system when a user-submitted task is bound to it. The determination module 702 is used to determine the resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system. The processing module 703 is used to initiate resource requests to the nodes of the distributed system according to the resource allocation strategy in order to execute the task.
[0087] The resource scheduling device provided by this invention binds a task agent to the task submitted by the user, uses the task agent to obtain the available resource information of the nodes in the distributed system, and determines the resource allocation strategy of the task according to the execution requirements of the task and the available resource information of the nodes in the distributed system. The device then initiates a resource request to the nodes in the distributed system according to the resource allocation strategy to execute the task. This eliminates the need to set up a dedicated resource scheduling node / functional module in the distributed system, thereby avoiding single points of failure.
[0088] Optionally, each node in the distributed system is equipped with a node agent, and each node agent is used to monitor resource changes on its node. The acquisition module 701 is specifically used for: Broadcast a resource request message to each of the node agents, and receive a resource quote message returned by each of the node agents based on the resource request message; Obtain the available resource information of each node from the resource quote message returned by each node agent.
[0089] Optionally, the available resource information includes the resource commitment amount and timeout constraint information, and the execution requirements include the resource demand amount, task priority, and resource bidding strategy; The determining module 702 is specifically used for: Based on the resource demand, task priority, resource bidding strategy, resource commitment amount and timeout constraint information of each node, multiple candidate resource allocation strategies and task execution prediction results corresponding to each candidate resource allocation strategy are generated. Based on the task execution prediction results corresponding to each of the candidate resource allocation strategies, the resource allocation strategy for the task is determined from the plurality of candidate resource allocation strategies.
[0090] Optionally, the task execution prediction results include task latency, resource matching degree, energy consumption cost, and conflict penalty; The determining module 702 is specifically used for: For each candidate resource allocation strategy, the benefit value of the candidate resource allocation strategy is calculated based on the task delay, resource matching degree, energy consumption cost and conflict penalty corresponding to the candidate resource allocation strategy. The candidate resource allocation strategy with the highest return value shall be used as the resource allocation strategy for the task.
[0091] Optionally, the payoff value of the candidate resource allocation strategy is calculated according to the following formula: Ri = α·T1 + β·T2 - γ·T3 - δ·T4 Where Ri is the revenue value of the i-th candidate resource allocation strategy, T1 is the task delay, T2 is the resource matching degree, T3 is the energy consumption cost, T4 is the conflict penalty, and α, β, γ and δ are all revenue weight parameters.
[0092] Optionally, the determining module 702 is further configured to: If the task is completed or times out, the revenue weight parameter is updated based on the task's delay, resource matching degree, and energy consumption cost.
[0093] Optionally, the determining module 702 is further configured to: If the resource allocation strategy for the new task submitted by the user conflicts with the resource allocation strategy for the task, the task priority in the execution requirements of the task will be adjusted according to a preset strategy. Based on the updated execution requirements of the task and the available resource information of the nodes in the distributed system, the resource allocation strategy for the task is re-determined; The processing module 703 is further configured to: According to the redefined resource allocation strategy for the task, resource requests are initiated to the nodes of the distributed system to continue executing the task.
[0094] Optionally, the determining module 702 is further configured to: If a faulty node occurs in the distributed system and the faulty node provides resources for the task, obtain the available resource information of other nodes in the distributed system besides the faulty node. Based on the execution requirements of the task and the available resource information of other nodes in the distributed system excluding the faulty node, the resource allocation strategy for the task is re-determined. The processing module 703 is further configured to: Resource requests are initiated to the nodes of the distributed system according to the redefined resource allocation strategy for the task, in order to continue executing the task.
[0095] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can invoke logical instructions in the memory 830 to execute a resource scheduling method. This method is applied to any task agent in a distributed system and includes: obtaining available resource information of the nodes in the distributed system when a user-submitted task is bound to it; determining a resource allocation strategy for the task based on the task's execution requirements and the available resource information of the nodes in the distributed system; and initiating a resource request to the nodes in the distributed system according to the resource allocation strategy to execute the task.
[0096] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the resource scheduling method provided by the above methods. This method is applied to any task agent in a distributed system and includes: obtaining available resource information of the nodes in the distributed system when a user-submitted task is bound to it; determining a resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system; and initiating a resource request to the nodes in the distributed system according to the resource allocation strategy to execute the task.
[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the resource scheduling method provided by the above methods. This method is applied to any task agent in a distributed system and includes: obtaining available resource information of nodes in the distributed system when a user-submitted task is bound to it; determining a resource allocation strategy for the task based on the task's execution requirements and the available resource information of the nodes in the distributed system; and initiating a resource request to the nodes in the distributed system according to the resource allocation strategy to execute the task.
[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A resource scheduling method, characterized in that, Any task-oriented intelligent agent applied in a distributed system, including: When a user-submitted task is bound, obtain the available resource information of the nodes in the distributed system; Based on the execution requirements of the task and the available resource information of the nodes in the distributed system, determine the resource allocation strategy for the task; Resource requests are initiated to the nodes of the distributed system according to the resource allocation strategy in order to execute the task.
2. The resource scheduling method according to claim 1, characterized in that, Each node in the distributed system is equipped with a node agent, and each node agent is used to monitor resource changes on its node. The step of obtaining the available resource information of the nodes in the distributed system includes: Broadcast a resource request message to each of the node agents, and receive a resource quote message returned by each of the node agents based on the resource request message; Obtain the available resource information of each node from the resource quote message returned by each node agent.
3. The resource scheduling method according to claim 1, characterized in that, The available resource information includes the resource commitment amount and timeout constraint information, and the execution requirements include the resource demand amount, task priority, and resource bidding strategy. The step of determining the resource allocation strategy for the task based on the task's execution requirements and the available resource information of the nodes in the distributed system includes: Based on the resource demand, task priority, resource bidding strategy, resource commitment amount and timeout constraint information of each node, multiple candidate resource allocation strategies and task execution prediction results corresponding to each candidate resource allocation strategy are generated. Based on the task execution prediction results corresponding to each of the candidate resource allocation strategies, the resource allocation strategy for the task is determined from the plurality of candidate resource allocation strategies.
4. The resource scheduling method according to claim 3, characterized in that, The task execution prediction results include task delay, resource matching degree, energy consumption cost, and conflict penalty; The step of determining the resource allocation strategy for the task from the plurality of candidate resource allocation strategies based on the task execution prediction result corresponding to each candidate resource allocation strategy includes: For each candidate resource allocation strategy, the benefit value of the candidate resource allocation strategy is calculated based on the task delay, resource matching degree, energy consumption cost and conflict penalty corresponding to the candidate resource allocation strategy. The candidate resource allocation strategy with the highest return value shall be used as the resource allocation strategy for the task.
5. The resource scheduling method according to claim 4, characterized in that, The payoff of the candidate resource allocation strategy is calculated using the following formula: Ri = α·T1 + β·T2 - γ·T3 - δ·T4 Where Ri is the revenue value of the i-th candidate resource allocation strategy, T1 is the task delay, T2 is the resource matching degree, T3 is the energy consumption cost, T4 is the conflict penalty, and α, β, γ and δ are all revenue weight parameters.
6. The resource scheduling method according to claim 5, characterized in that, The method further includes: If the task is completed or times out, the revenue weight parameter is updated based on the task's delay, resource matching degree, and energy consumption cost.
7. The resource scheduling method according to claim 1, characterized in that, The method further includes: If the resource allocation strategy for the new task submitted by the user conflicts with the resource allocation strategy for the task, the task priority in the execution requirements of the task will be adjusted according to a preset strategy. Based on the updated execution requirements of the task and the available resource information of the nodes in the distributed system, the resource allocation strategy for the task is re-determined; According to the redefined resource allocation strategy for the task, resource requests are initiated to the nodes of the distributed system to continue executing the task.
8. The resource scheduling method according to claim 1, characterized in that, The method further includes: If a faulty node occurs in the distributed system and the faulty node provides resources for the task, obtain the available resource information of other nodes in the distributed system besides the faulty node. Based on the execution requirements of the task and the available resource information of other nodes in the distributed system excluding the faulty node, the resource allocation strategy for the task is re-determined. Resource requests are initiated to the nodes of the distributed system according to the redefined resource allocation strategy for the task, in order to continue executing the task.
9. A resource scheduling device, characterized in that, The device, applicable to any task agent in a distributed system, comprises: The acquisition module is used to acquire available resource information of the nodes in the distributed system when a user-submitted task is bound to it. The determination module is used to determine the resource allocation strategy for the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system. The processing module is used to initiate resource requests to the nodes of the distributed system according to the resource allocation strategy in order to execute the task.
10. A distributed system, characterized in that, This includes multiple nodes, multiple task agents, and agent management services; The intelligent agent management service is used to configure a corresponding task intelligent agent for the task submitted by the user, and to bind the task to the task intelligent agent; The task agent is used to obtain the available resource information of the nodes in the distributed system when it is bound to a user-submitted task; and to determine the resource allocation strategy of the task based on the execution requirements of the task and the available resource information of the nodes in the distributed system. Resource requests are initiated to the nodes of the distributed system according to the resource allocation strategy in order to execute the task.