A task optimization configuration method based on historical resource records

Abstract

The present invention relates to the technical field of centralized configuration of distributed systems, and in particular to a method for optimizing task configuration based on historical resource records, the steps of which are as follows: obtaining a task configuration request, the task configuration request including the resource requirements of a task to be executed; obtaining historical resource records of each task node, and predicting the current task acceptance capacity value of each task node according to the historical resource records; and allocating task execution nodes to the task to be executed based on the resource requirements of the task to be executed and the current task acceptance capacity value of each task node. The present invention calculates the task acceptance capacity value of the task node based on the historical resource records, so that the configurator can estimate the current task acceptance capacity value of each task node in the absence of real-time resource information of the node, effectively reducing the dependence on the real-time resource information of the node, and realizing task configuration under the condition of asymmetric node resource information, which has stronger robustness and applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of centralized configuration of distributed systems, and in particular to a task optimization configuration method based on historical resource records. Background Art

[0002] Distributed systems are computer systems built on the Internet. They are increasingly used on the Internet. Each server or node in a distributed system relies on the Internet to communicate with each other. There are a large number of tasks that need to be executed in a distributed system. The demand for efficient configuration of a large number of tasks poses a great challenge to the distributed system.

[0003] Currently, in distributed systems, various tasks are executed on a single machine, and the nodes can communicate and monitor each other. The centralized configuration relies on the central node or configurator to grasp the real-time resource information of each computing node; but in some scenarios, the central node or configurator may not be able to obtain the real-time resource status information of each node, or due to network delays, bandwidth limitations, etc., the cost of obtaining resource information of all nodes is too high. The problem of asymmetric node resource information makes it impossible to perform task configuration normally or the cost of task configuration is too high.

[0004] Based on this, how to realize task configuration under the condition of asymmetric node resource information is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a method for optimizing task configuration based on historical resource records. By calculating the task acceptance capacity value of task nodes based on historical resource records, the configurator can estimate the current task acceptance capacity value of each task node in the absence of real-time resource information of the node, effectively reducing the dependence on the real-time resource information of the node, thereby solving the technical problem of how to realize task configuration in the case of asymmetric node resource information.

[0006] The present invention is implemented by the following technical solution: A method for optimizing task configuration based on historical resource records, comprising the following steps:

[0007] Obtaining a task configuration request, wherein the task configuration request includes resource requirements for the task to be executed;

[0008] Obtaining historical resource records of each task node, and predicting the current task acceptance capacity value of each task node based on the historical resource records, wherein the historical resource records include historical task data, task execution time, and resource consumption;

[0009] Based on the resource requirements of the tasks to be executed and the current task acceptance capacity value of each task node, a task execution node is allocated to the tasks to be executed.

[0010] According to a preferred embodiment, after determining the resource requirements of the task to be executed, the method further comprises: determining the weights of various types of resources in the resource requirements.

[0011] According to a preferred implementation, the calculation expression of the weight of each type of resource in the resource demand is as follows:

[0012]

[0013] In the above formula, All represent indexes, where is the index of the current resource, is the index used to traverse the resource, The index is The resource weight at the time of execution indicates the resource weight of the task to be executed The intensity of demand, Indicates that the index is The resource demand of the tasks to be executed at the time It represents the total demand of all resources for the tasks to be executed. Indicates the total number of resource types. Indicates that the index is The resource demand of the tasks to be executed at that time.

[0014] According to a preferred implementation, the specific process of predicting the current task acceptance capacity value of each node includes:

[0015] According to the historical resource records of each task node in the distributed system, calculate the acceptance of each task node for various types of resources in the resource demand;

[0016] Based on the weights of various resources in the resource requirements and the acceptance of each task node to various resources, the task acceptance capacity value of each task node is calculated.

[0017] According to a preferred implementation, the calculation expression of the acceptance degree of each type of resource is as follows:

[0018]

[0019] In the above formula, Indicates the task node's resource acceptance, Represents all resources of the task node The total amount, Indicates all tasks being executed on the task node. The total consumption, Indicates the total number of tasks being executed on the task node. Indicates the task About Resources The consumption of is the task decay coefficient, which means that the impact of the task on resources gradually weakens over time.

[0020] According to a preferred embodiment, the calculation expression of the task decay coefficient is as follows:

[0021]

[0022] In the above formula, Indicates that the index is The resource requirements of the task at that time, is the single-task extinction coefficient.

[0023] According to a preferred embodiment, the calculation expression of the single-task decay coefficient is as follows:

[0024]

[0025] In the above formula, is the growth rate control parameter, which indicates the growth degree of resource demand of the task during its life cycle. Indicates the current time. Indicates the task Start time, The preset maximum life cycle of the task.

[0026] According to a preferred implementation, the calculation expression of the task acceptance capability value of each task node is as follows:

[0027]

[0028] In the above formula, Indicates the task acceptance capacity value of the task node.

[0029] According to a preferred implementation, the specific process of assigning a task execution node to the task to be executed is as follows:

[0030] Determine a task node with the largest task acceptance capability value from each task node, and use the task node with the largest task acceptance capability value as the task execution node for the task to be executed;

[0031] The tasks to be executed are assigned to the task execution nodes for execution.

[0032] According to a preferred implementation, the specific process of determining the task node with the largest task acceptance capability value from each task node is as follows:

[0033] Traverse the task node set, compare the task acceptance capability values ​​of all task nodes in the task node set, and determine the task node with the largest task acceptance capability value in the task node set. The expression is as follows:

[0034]

[0035] In the above formula, Indicates the task node with the largest task acceptance capability value. Represents a task node The task acceptance capacity value of Represents a collection of task nodes.

[0036] The technical solution of a task optimization configuration method based on historical resource records provided by the present invention has at least the following advantages and beneficial effects:

[0037] (1) The present invention introduces the calculation of task decay coefficient and resource acceptance based on historical resource records, so that the configurator can obtain the current task acceptance capacity value of each task node by estimating the value in the absence of real-time resource information of the node, effectively reducing the dependence on the real-time resource information of the node, and realizing task configuration in the case of asymmetric node resource information, which has stronger robustness and applicability;

[0038] (2) This method does not rely on real-time synchronization, avoiding the overhead caused by high-frequency communication and real-time status updates. It only needs to extract relevant data from the stored historical resource records. Compared with task configuration based on real-time resource information, the computational complexity is lower and there is no need to frequently obtain and process data, which reduces the computational and communication burdens.

[0039] (3) This method calculates the acceptance and weight of various types of resources, and can perform differentiated analysis based on the intensity of task demand for different resources, making it more suitable for distributed systems with multiple resource configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a method for optimizing task configuration based on historical resource records provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0042] Example 1

[0043] Figure 1 A flowchart of a method for optimizing task configuration based on historical resource records provided by an embodiment of the present invention includes the following steps:

[0044] S101: acquiring a task configuration request initiated by a task provider and analyzing relevant task information carried in the task configuration request, wherein the task configuration request includes resource requirements of the task to be executed.

[0045] The task optimization configuration method provided in an embodiment of the present invention can be applied to a task configurator or a master node in a distributed system. The task configurator or the master node can be located on a server or on a virtual machine of the server. The task node that provides resources to the task provider can be a physical server or a virtual machine on a physical server.

[0046] When performing task configuration, the embodiment of the present invention targets each task node, and predicts the resource usage of each task node in the previous cycle, so as to allocate corresponding resources to the current task provider. In each cycle, the historical resource records of each task node will be stored in the task configurator or the master node to facilitate subsequent query and use. The process of obtaining historical resource records belongs to the prior art, and the embodiment of the present invention will not be described in detail for this process.

[0047] S102, obtaining historical resource records of each task node, and predicting the current task acceptance capacity value of each task node according to the historical resource records, wherein the historical resource records include historical task data, task execution time, and resource consumption.

[0048] In each cycle, the historical resource records of each task node will be saved. Therefore, when it is necessary to determine the task acceptance capacity value of the task node in the current cycle, the task acceptance capacity value of the node can be predicted based on the stored historical resource records of each task node, and the predicted task acceptance capacity value can be used as the theoretical load capacity of the task node in the current cycle.

[0049] S103: Allocate a task execution node to the task to be executed based on the resource demand of the task to be executed and the current task acceptance capacity value of each task node.

[0050] After the task acceptance capability value of each task node is determined, the task execution node can be determined according to the task acceptance capability value in the current cycle, so as to allocate resources to the tasks to be executed on the task execution node.

[0051] Because in the embodiment of the present invention, by calculating the task acceptance capacity value of the task node based on historical resource records, the configurator can estimate the current task acceptance capacity value of each task node in the absence of real-time resource information of the node, effectively reducing the dependence on the real-time resource information of the node, and realizing task configuration in the case of asymmetric node resource information, which has stronger robustness and applicability.

[0052] Example 2

[0053] Tasks have different requirements for different resources. In order to accurately determine the task acceptance capacity value of each task node, so as to allocate the best task execution node for the task to be executed, based on Example 1, in this embodiment of the present invention, after determining the resource requirements of the task to be executed, the method also includes determining the weights of various types of resources in the resource requirements. It should be noted that the greater the intensity of the task's demand for a resource, the higher the weight of the resource.

[0054] This embodiment determines the weight of each resource type according to the demand of the task to be executed for each resource. The higher the proportion of the resource demand in all resource demands, the higher the weight of the resource, which also indicates that the demand of the task to be executed for the resource is more important.

[0055] In one implementation of this embodiment, the calculation expression of the weight of each type of resource in the resource demand is as follows:

[0056]

[0057] In the above formula, All represent indexes, where is the index of the current resource, is the index used to traverse the resource, The index is The resource weight at the time of execution indicates the resource weight of the task to be executed The intensity of demand, Indicates that the index is The resource demand of the tasks to be executed at the time It represents the total demand of all resources for the tasks to be executed. Indicates the total number of resource types. Indicates that the index is The resource demand of the tasks to be executed at that time.

[0058] It should be noted that this method of calculating weights for various types of resources can more accurately reflect the demand intensity of various types of resources in resource demand, and is helpful to find the most suitable task execution node from various task nodes for the task to be executed.

[0059] Example 3

[0060] This embodiment is based on the technical solutions provided in Example 1 and Example 2, and further explains the prediction of the task acceptance capability value:

[0061] In order to cope with the situation of asymmetric node resources, the present invention introduces a task decay coefficient and a resource receiving end to calculate the task acceptance capacity value of each task node; in this embodiment, the specific process of predicting the current task acceptance capacity value of each node includes:

[0062] S1021. Calculate the acceptance of each task node for each type of resource in the resource demand based on the historical resource records of each task node in the distributed system.

[0063] This embodiment determines the acceptance of each task node to each resource in the resource demand according to the historical task data, task execution time and resource consumption included in the historical resource record; the calculation expression of the acceptance of each resource is as follows:

[0064]

[0065] In the above formula, Indicates the task node's resource acceptance, The higher the value, the lower the utilization rate of the resource and the smaller the load. Indicates resource types, such as CPU, memory, storage, network, etc. Represents all resources of the task node The total amount, such as the total number of CPU cores, total memory size, total storage capacity, etc. Indicates all tasks being executed on the task node. The total consumption, The higher the value, the higher the resource load of the task node. Indicates the total number of tasks being executed on the task node. Indicates the task About Resources The consumption of is the task decay coefficient, which means that the impact of the task on resources gradually weakens over time.

[0066] The task decay coefficient indicates that the impact of the historical tasks of the task node on the resources gradually weakens over time. In this embodiment, the calculation expression of the task decay coefficient is as follows:

[0067]

[0068]

[0069] In the above formula, Indicates that the index is The resource requirements of the task at that time, is the single-task decay coefficient, is the growth rate control parameter, which indicates the growth degree of resource demand of the task during its life cycle. Indicates the current time. Indicates the task Start time, The maximum life cycle of a task is preset. Tasks that exceed the maximum life cycle of the task can be considered to have ended and will no longer affect the resources of the task node.

[0070] Since in the embodiment of the present invention, by introducing the calculation of task decay coefficient and resource acceptance, this method does not rely on real-time synchronization, avoids the overhead caused by high-frequency communication and real-time status update, and only needs to extract relevant data from the stored historical resource records. Compared with task configuration based on real-time resource information, the calculation complexity is lower, and there is no need to frequently obtain and process data, which reduces the calculation and communication burden. S1022. Based on the weights of various resources in the resource requirements and the acceptance of each task node to various resources, calculate the task acceptance capacity value of each task node.

[0071] For each task node, the task configurator comprehensively considers the resource weight and the remaining resource availability of the task node to calculate the task acceptance capacity value of each task node; the calculation expression of the task acceptance capacity value of each task node is as follows:

[0072]

[0073] In the above formula, Indicates the task acceptance capacity value of the task node.

[0074] After determining the task acceptance capacity value of each task node, the best task execution node that can meet the resource requirements of the task to be executed can be determined based on the acceptance capacity value. The determined best task execution node takes the weights of various resources into consideration and can therefore more accurately reflect the demand intensity of various resources in the resource requirements. When the task to be executed is a certain type of resource-intensive task, such as a CPU-intensive task or an IO-intensive task, this method can fully meet the configuration requirements of this type of task.

[0075] Example 4

[0076] This embodiment further explains the allocation of task execution nodes based on the technical solution provided in Embodiment 4:

[0077] In this embodiment, the allocation criterion of the task execution node is to allocate the task node with the largest task acceptance capability value, and the various resource requirements of the task to be executed are met through the task node with the largest task acceptance capability value; the specific process of allocating the task execution node to the task to be executed is as follows:

[0078] S1031 . Determine a task node with the largest task acceptance capability value from among the task nodes, and use the task node with the largest task acceptance capability value as a task execution node for the task to be executed.

[0079] The shortcomings of the task nodes can be solved by using methods such as loop traversal, divide and conquer, randomized maximum value search algorithm, sorting method, etc. to determine the task node with the largest task acceptance capacity value from each task node; in this embodiment, the loop traversal method is used to determine the task node with the largest task acceptance capacity value from each task node, and the specific process is as follows:

[0080] Traverse the task node set, compare the task acceptance capability values ​​of all task nodes in the task node set, and determine the task node with the largest task acceptance capability value in the task node set. The expression is as follows:

[0081]

[0082] In the above formula, Indicates the task node with the largest task acceptance capability value. Represents a task node The task acceptance capacity value of S1032: Allocate the to-be-executed tasks to the task execution nodes for execution.

[0083] It should be noted that if there is a need to further improve the efficiency of searching for task execution nodes, other more efficient methods may also be used, and no specific limitation is made here.

[0084] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A task optimization configuration method based on historical resource records, characterized in that: The following steps are involved: Obtaining a task configuration request, wherein the task configuration request includes resource requirements for the task to be executed; Obtaining historical resource records of each task node, and predicting the current task acceptance capacity value of each task node based on the historical resource records, wherein the historical resource records include historical task data, task execution time, and resource consumption; Allocating a task execution node to the task to be executed based on the resource requirements of the task to be executed and the current task acceptance capacity value of each task node; The specific process of predicting the current task acceptance capacity value of each node includes: According to the historical resource records of each task node in the distributed system, calculate the acceptance of each task node for various types of resources in the resource demand; The calculation expressions of the acceptance of various types of resources are as follows: In the above formula, Indicates the task node's resource acceptance, Represents all resources of the task node The total amount, Indicates all tasks being executed on the task node. The total consumption, Indicates the total number of tasks being executed on the task node. Indicates the task About Resources The consumption of is the task decay coefficient, which indicates that the impact of the task on resources gradually weakens over time; The calculation expression of the task decay coefficient is as follows: In the above formula, Indicates that the index is The amount of resources required by the task at that time, is the single-task extinction coefficient; The calculation expression of the single task extinction coefficient is as follows: In the above formula, is the growth rate control parameter, which indicates the growth degree of resource demand of the task during its life cycle. Indicates the current time. Indicates the task Start time, The maximum life cycle of the preset task; Based on the weights of various resources in the resource requirements and the acceptance of each task node to various resources, the task acceptance capacity value of each task node is calculated; The calculation expression of the task acceptance capability value of each task node is as follows: In the above formula, Indicates the task acceptance capability value of the task node.

2. The task optimization configuration method based on historical resource records as claimed in claim 1, characterized in that: The calculation expressions of the weights of various resources in the resource requirements are as follows: In the above formula, All represent indexes, where is the index of the current resource, is the index used to traverse the resource, The index is The resource weight at the time of execution indicates the resource weight of the task to be executed The intensity of demand, Indicates that the index is The resource demand of the tasks to be executed at the time It represents the total demand of all resources for the tasks to be executed. Indicates the total number of resource types. Indicates that the index is The resource demand of the tasks to be executed at that time.

3. The task optimization configuration method based on historical resource records as claimed in claim 2, characterized in that: The specific process of assigning task execution nodes to the tasks to be executed is as follows: Determine a task node with the largest task acceptance capability value from each task node, and use the task node with the largest task acceptance capability value as the task execution node for the task to be executed; The tasks to be executed are assigned to the task execution nodes for execution.

4. The task optimization configuration method based on historical resource records as claimed in claim 3, characterized in that: The specific process of determining the task node with the largest task acceptance capacity value from each task node is as follows: Traverse the task node set, compare the task acceptance capability values ​​of all task nodes in the task node set, and determine the task node with the largest task acceptance capability value in the task node set. The expression is as follows: In the above formula, Indicates the task node with the largest task acceptance capability value. Represents a task node The task acceptance capacity value of Represents a collection of task nodes.

Images