Distributed task resource scheduling system and method

Abstract

The invention provides a distributed task resource scheduling system and method, and the system comprises a resource management unit which is used for dividing resource nodes into conventional nodes and special nodes according to the important emergency degree; the task management unit is used for adding attribute identifiers for the tasks according to the relationship among the distributed tasks; determining the priority of the task according to the attribute identifier, and putting the task into a waiting queue for queuing based on the priority; the attribute identifier comprises a character string and a label, the character string comprises a task type, a priority and a system task number, and the label comprises an affiliation relationship and a dependency relationship; and the scheduling unit is used for carrying out node resource configuration according to the priority of the task and creating a contract. According to the method, through fine resource node division and task attribute identification, emergency task priority scheduling and accurate management of complex relations between tasks are realized, and by combining an advanced allocation algorithm and a dynamic scheduling strategy, the resource utilization rate and the task execution efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed task resource scheduling system and method. Background Art

[0002] Resource scheduling technology is currently in a stage of rapid development, driven by fields such as cloud computing, artificial intelligence, and big data. Modern application scenarios have an increasing demand for computing resources, so efficient resource scheduling has become crucial.

[0003] However, resource scheduling technology still faces some challenges. For example, for resource scheduling of distributed tasks, the traditional solution is to split the task into multiple subtasks and perform resource scheduling for each subtask separately. The problem with this method is that the resource scheduling of each subtask is asynchronous, and it cannot guarantee that the resources of all subtasks can be successfully scheduled at the same time, which will lead to very low resource utilization; the same is true for the subtasks of the same task, some subtasks obtain resources first, while other subtasks obtain resources later or cannot obtain resources; in addition, due to the complex dependencies between tasks and network communication delays, resource scheduling also faces some challenges in node load balancing. Therefore, it is particularly important to develop a new resource scheduling technology to solve the problems of distributed tasks with dependencies, asynchronous resource scheduling of each subtask of the same task, and node load balancing. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a distributed task resource scheduling system and method. The present invention realizes efficient scheduling of distributed task resources by finely dividing resource nodes and task priorities, ensures priority processing of urgent tasks through intelligent resource allocation, and accurately manages complex relationships between tasks to avoid conflicts. Real-time monitoring and data recording functions improve system stability and optimization capabilities, providing reliable support for large-scale distributed computing.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a distributed task resource scheduling system, comprising:

[0007] A resource management unit, used to divide resource nodes into regular nodes and dedicated nodes according to their importance and urgency;

[0008] The task management unit is used to add attribute identifiers to tasks according to the relationship between distributed tasks; determine the priority of the task according to the attribute identifier, and put the task into a waiting queue based on the priority; the attribute identifier includes a string and a label, the string includes the task type, priority and system task number, and the label includes the subordinate relationship and the dependent relationship;

[0009] The scheduling unit is used to configure node resources according to the priority of the task and create contracts;

[0010] A monitoring unit is used to monitor the operating status of the resource management unit, the task management unit and the scheduling unit in real time and record it in the information life cycle management unit;

[0011] The information lifecycle management unit is used to receive the operating status data sent by the monitoring unit.

[0012] Preferably, the conventional nodes are working nodes for general urgent or important tasks, and the dedicated nodes are working nodes for urgent or important tasks.

[0013] Preferably, the task types contained in the character string include computation-intensive tasks, memory-intensive tasks, bandwidth-intensive tasks, and I / O-intensive tasks;

[0014] The priorities include highest, second highest, medium, second lowest, and lowest.

[0015] Preferably, the scheduling unit includes a task requirement calculation module, a node RES calculation module and a matching module;

[0016] The task requirement calculation module is used to calculate the resource requirement type and required quantity of the task, determine the task type according to the task attribute identifier, and determine the node allocation type according to the task type;

[0017] The node RES calculation module is used to obtain the load formula coefficient according to the task type and calculate the computing power resource value of all nodes in the entire domain of nodes applicable to the task;

[0018] The matching module is used to assign tasks to the node with the largest computing power resource value according to the dynamic scheduling strategy, create a contract and record it in the information lifecycle management unit.

[0019] Preferably, the load formula is specifically:

[0020] Res=W1*K+W2*R+W3*B+W4*I

[0021] Among them, Res is the computing resource value, W1 is the first weight, K is the average CPU or GPU load, W2 is the second weight, R is the physical memory value, W3 is the third weight, B is the bandwidth, W4 is the fourth weight, and I is the I / O occupancy.

[0022] Preferably, the dynamic scheduling strategy includes:

[0023] Obtain the initial priority of the task according to the task's label, and obtain its subordinate or dependent relationship through the task's label;

[0024] According to the subordination or dependency of the task, check the priority of the tasks related to the task. If the priority of the tasks with subordination or dependency to the task does not match the priority of the task, adjust the priority of the task to a reasonable level and use the priority at this time as the final priority; otherwise, use the initial priority as the final priority for task scheduling.

[0025] Preferably, the contract is an execution protocol established by the scheduling unit when selecting appropriate resources for the task through an allocation algorithm based on task priority and node resources, which includes task execution parameters, resource allocation rules and conditions required for successful execution.

[0026] In a second aspect, the present invention provides a distributed task resource scheduling method, comprising:

[0027] According to the importance and urgency, resource nodes are divided into regular nodes and dedicated nodes;

[0028] According to the relationship between distributed tasks, attribute identifiers are added to the tasks; the priority of the tasks is determined according to the attribute identifiers, and the tasks are queued in a waiting queue based on the priority; the attribute identifiers include a string and a label, the string includes the task type, priority and system task number, and the label includes the subordination and dependency relationships;

[0029] Configure node resources according to task priority and create contracts;

[0030] Monitor the operating status of the resource management unit, task management unit and scheduling unit in real time, and record it in the information lifecycle management unit.

[0031] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a distributed task resource scheduling method described in the second aspect.

[0032] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a distributed task resource scheduling method described in the second aspect are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The distributed task resource scheduling system of the present invention can flexibly respond to task requirements of different urgency levels through refined resource node division (conventional nodes and dedicated nodes), ensuring that urgent and important tasks are given priority in obtaining resources, thereby improving the overall task execution efficiency and response speed. Secondly, the task management unit achieves accurate management of complex relationships between tasks by adding rich attribute identifiers to tasks, including task type, priority, system task number, and subordinate and dependent relationships, thereby avoiding conflicts and delays in task execution. The scheduling unit adopts advanced allocation algorithms, combined with dynamic scheduling strategies, to ensure that tasks are executed on the most suitable nodes, further improving resource utilization and task execution efficiency. In addition, the system's real-time monitoring and information lifecycle management functions not only ensure the stable operation of the system, but also provide data support for the continuous optimization of the system.

[0035] (2) The allocation algorithm adopted by the scheduling unit provided by the present invention will calculate the computing power resource value of each node in real time when selecting nodes for tasks, and preferentially allocate tasks to nodes with the highest computing power resource value. This means that the system will tend to select nodes that are currently idle or lightly loaded, ensuring that tasks are allocated to nodes with the maximum computing power and the minimum load, thereby achieving optimization and balance of resource utilization. By dynamically calculating the computing power resource value of nodes, the system can effectively avoid excessive resource consumption or excessive idleness of certain nodes, and achieve load balancing and efficient resource management among nodes.

[0036] (3) The task management unit provided by the present invention adjusts sibling or dependent tasks to the same priority by setting a unique identifier (string) and label for each task. For each subtask of the same task, the priority is unified through the identifier, so that tasks with related relationships can be processed in the same scheduling level. This method ensures that tasks with dependent relationships in an asynchronous environment are executed in order of priority, avoiding the disorder of task execution and confusion of resource allocation, thereby effectively solving the timing problems and resource competition conflicts in asynchronous resource scheduling, and achieving stability and efficiency of resource scheduling.

[0037] In summary, the present invention effectively improves the intelligence and automation level of distributed task resource scheduling, and provides strong technical support for large-scale distributed computing environments.

[0038] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0040] Figure 1 A schematic diagram of a distributed task resource scheduling system provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of resource scheduling of a distributed task resource scheduling system provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a scheduling unit provided in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the operation of a monitoring unit provided in an embodiment of the present invention;

[0044] Figure 5 A distributed task resource scheduling method is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] Technical term explanation:

[0046] 1. Resource scheduling: In the field of computing resource allocation, resource scheduling refers to the process of rationally arranging and allocating computing resources according to task requirements and computing resources (such as CPU / GPU, memory, etc.) to maximize system performance and efficiency. This includes task scheduling, resource management, scheduling strategies, and performance optimization.

[0047] 2. Container orchestration technology: In the field of computing resource scheduling, container orchestration technology refers to the method of using containerization technology (such as Docker, Kubernetes, etc.) to effectively manage and schedule computing resources. Through container orchestration technology, containerized applications can be automatically deployed, expanded, and managed, and resource allocation can be dynamically adjusted according to system load conditions to achieve efficient resource utilization and flexible application deployment. This technology can improve the reliability, scalability, and maintainability of the system, and is one of the key technologies commonly used in cloud computing and large-scale distributed systems.

[0048] 3.Kubernetes: Kubernetes (often referred to as K8s) is an open source container orchestration and management platform for automating the deployment, scaling, and operation of application containers. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation (CNCF). Kubernetes provides a powerful infrastructure that helps users effectively manage containerized applications with complex network requirements. The main features of Kubernetes include: automated deployment and scaling, self-healing, service discovery and load balancing, storage orchestration, custom configuration, and automated rolling updates and rollbacks. In short, Kubernetes provides users with an efficient, reliable, and scalable way to manage containerized applications, and has become one of the preferred platforms for cloud-native application deployment and management.

[0049] 4. Node load balancing: In the field of computing resource allocation, node load balancing refers to the process of evenly distributing computing tasks or workloads among different computing nodes (servers, virtual machines, etc.) in a distributed system. This means that the system will dynamically allocate tasks to the most suitable nodes based on the load conditions and performance indicators of each node to achieve optimal resource utilization and system performance.

[0050] 5. Subordinate and dependent relationships: A task and another task belong to the same large task, that is, the two tasks are brother tasks and are both subtasks of the large task; there is a subordinate relationship between the subtask and the large task; in a distributed system, if there is a mutual dependence or sequential relationship between one task and another, it indicates that the two tasks have a dependent relationship.

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] Embodiment 1

[0053] like Figure 1 As shown, this embodiment discloses a distributed task resource scheduling system, including:

[0054] A resource management unit, used to divide resource nodes into regular nodes and dedicated nodes according to their importance and urgency;

[0055] The task management unit is used to add attribute identifiers to tasks according to the relationship between distributed tasks; determine the priority of the task according to the attribute identifier, and put the task into a waiting queue based on the priority; the attribute identifier includes a string and a label, the string includes the task type, priority and system task number, and the label includes the subordinate relationship and the dependent relationship;

[0056] The scheduling unit is used to configure node resources according to the priority of the task and create contracts;

[0057] A monitoring unit is used to monitor the operating status of the resource management unit, the task management unit and the scheduling unit in real time and record it in the information life cycle management unit;

[0058] The information lifecycle management unit is used to receive the operating status data sent by the monitoring unit.

[0059] like Figure 2 As shown, this embodiment integrates all tasks into a task set, wherein the tasks in the task set are queued according to priority, and then all resources are integrated into a resource set, and finally resources are scheduled through a scheduling unit, which selects a suitable node through an allocation algorithm and assigns the task to it for execution and creates a contract.

[0060] As a specific implementation method, the contract is a dynamic agreement mechanism based on task and resource selection. In the task scheduling process, after each task is assigned to resources, a "contract" between the task and the resource will be created. The contract represents the agreement that the task will select the specified resources to complete its execution requirements under the arrangement of the scheduling unit. The contract is formed by the scheduling unit based on the tasks in the task priority queue and the resources in the resource set through the allocation algorithm. This contract, as a binding relationship between tasks and resources, contains the parameters of task execution (such as execution time, resource requirements, etc.), resource allocation rules (such as priority allocation, etc.) and conditions required for successful execution (such as completion standards, performance requirements, etc., which are specific requirements of a certain task), and further includes the task number and the node number. The existence of the contract ensures the reliability and traceability of task scheduling, and provides real-time management of resource usage and task execution process.

[0061] The task management unit aggregates all tasks together to facilitate unified management of all tasks.

[0062] The task management unit sets a unique identifier for each task, first measures its task type and priority, assigns the task a string to indicate the type, priority and system task number of the task, and assigns the task a label after obtaining the subordination and dependency of the task, so that it can indicate the subordination and dependency of the task. The system task number is an identifier for each task to reflect the order of precedence.

[0063] The unique identifier of a task is set as follows:

[0064] The string consists of task type (1-3 digits) + priority (4-6 digits) + task code (7-10 digits);

[0065] 1-3 bits indicate the type (001: compute-intensive task, 002: memory-intensive task, 003: bandwidth-intensive task, 004: I / O-intensive task)

[0066] 4-6 bits can be set as priority (001: highest level, 002: second highest level, 003: middle level, 004: second lowest level, 005: lowest level)

[0067] 7-10 digits are the numbers assigned by the system to each task.

[0068] In addition to the string, each task should also be given a label (indicating the subordinate relationship and dependency relationship). Tasks with the same label are sibling tasks / distributed tasks with dependencies. Among them, the label corresponding to the distributed task with dependencies can also highlight the front-and-back dependency relationship with other tasks. It should be understood that the front-and-back relationship can be represented by the size of the number, which is set by those skilled in the art according to actual conditions.

[0069] This embodiment provides strong support for efficient management and accurate tracking of tasks by setting a unique identification for the task. This identification method not only facilitates the system to quickly identify task characteristics, intelligently allocate resources, and optimize load balancing, but also effectively manages complex dependencies between tasks to ensure the order and accuracy of task execution. At the same time, it improves the scalability and flexibility of the system, lays a solid foundation for the efficient and coordinated execution of distributed tasks, and helps to improve the overall system operation efficiency and task completion quality.

[0070] After setting a unique identifier for each task, assign it to the corresponding waiting queue. First, obtain the initial priority of the task based on the task label, and then obtain its subordination or dependency through the task label. After obtaining the subordination or dependency of the task, check the priority of the tasks related to the task.

[0071] If the priority of a task that has a dependency relationship with this task does not match the priority of this task (that is, some tasks must be executed after this task is completed, and if the priority of this task is lower, it does not match), then adjust the priority of the task to a reasonable level, for example, raise the priority of the task to a level higher than that of its dependent tasks, and make the priority at this time the final priority.

[0072] If the priority of a task that has a sibling relationship with this task does not match the priority of this task (that is, most of the sibling tasks of this task have higher priorities, so other sibling tasks may be completed at a similar time, but this task may not be completed due to a delay in allocating resources), then the priority of the task is adjusted to a reasonable level, for example, the priority of the task is adjusted to the same level as its sibling tasks, and the priority at this time is set as the final priority.

[0073] If the priority of the task that is subordinate to the task does not match the priority of the task, that is, the priority of the subordinate task (such as a subtask or branch task) does not match the priority of the main task, resulting in uneven resource allocation or chaotic execution order, the priority of the task should be adjusted to a reasonable level to ensure the coordinated execution of the entire task system. For example, if the priority of the main task is low, and the priority of its subordinate tasks is generally high, it may affect the progress of the overall task. At this time, the priority of the main task should be raised to a level similar to or higher than that of its subordinate tasks. The adjusted priority is the final priority of the task. This ensures that resources can be reasonably allocated and avoids execution conflicts between tasks.

[0074] After each task has its final priority level determined, it will be placed in the corresponding priority waiting queue in turn. After the task is queued, it will enter the scheduling unit to allocate resources for the task.

[0075] At the same time, the monitoring unit also monitors it in real time and records it in the information lifecycle management unit, so that emergency tasks can be handled in a timely manner to avoid occupying available resources later.

[0076] The resource management unit aggregates the resources of all nodes (puts all resources in a resource pool) and manages them in a unified manner to achieve efficient utilization of resources.

[0077] like Figure 2 As shown, in order to ensure the smooth execution of certain important tasks, all nodes are divided into regular nodes and dedicated nodes. Therefore, the resource collection will include two types of nodes: regular nodes and dedicated nodes.

[0078] Conventional nodes are used to run general tasks (tasks with general priority or tasks that are not sensitive to time, etc.);

[0079] Dedicated nodes are used to run certain tasks (tasks with higher priority or are time-sensitive, or have dependencies on other high-priority tasks, etc.), which strictly guarantees the implementation of certain urgent or important tasks.

[0080] When the task is finished waiting in the queue, it enters the scheduling unit to allocate resources for the task.

[0081] At the same time, the monitoring unit performs real-time monitoring and updates (node ​​occupancy, release, etc.) and records them in the information lifecycle management unit to facilitate efficient management of all nodes and avoid waste of certain resources.

[0082] like Figure 3 As shown, the scheduling unit is divided into three modules: task requirement calculation module, node RES calculation module, and matching module.

[0083] The function of the task requirement calculation module is to first calculate the resource requirement type and quantity of the task when a task is scheduled for resource matching to facilitate the subsequent resource matching and determine which resource-intensive task the task is;

[0084] The function of the node RES calculation module is to allocate the task to a regular node or a dedicated node according to the emergency type / priority level of the task, and then calculate the RES (computing resource value) of all nodes in the regular node (or dedicated node);

[0085] The function of the matching module is to select the node with the largest RES value, assign the task to the node and create a contract;

[0086] When the task queue ends, that is, when matching resources for it, first determine the task type and priority (urgency / importance level) of the task according to the task identifier and calculate the resource requirement type and quantity of the task.

[0087] Divide the node allocation type of the task (regular node or dedicated node) according to the priority of the task;

[0088] After the node types are divided, the coefficient of the computing resource value RES (load formula) is allocated to the task according to the task type; then the allocation algorithm (load formula) is called to calculate the RES value of each node in the entire domain (all nodes that meet the requirements).

[0089] Among them, the compliance requirement means that after dividing whether regular nodes or dedicated nodes should be allocated, all nodes in the schedulable state (with available computing resources) within the scope of the node type participate in the calculation of the RES computing power resource value.

[0090] It should be understood that there are many allocation algorithms, but it is difficult to select a suitable allocation algorithm.

[0091] For example, the problem with the random allocation algorithm is that it cannot guarantee the fairness and efficiency of task allocation, which may cause some computing nodes to be overloaded while other nodes are underloaded. The random allocation algorithm is the simplest computing power allocation algorithm, which randomly allocates tasks to different computing nodes. The advantage of this algorithm is that it is simple to implement and is suitable for small-scale distributed systems; the polling allocation algorithm allocates tasks to different computing nodes in a fixed order. The advantage of this algorithm is that it can distribute the task load more evenly and avoid overloading some nodes. However, the polling allocation algorithm cannot be dynamically adjusted according to the actual load of the node, which may cause the computing power of some nodes to be underutilized; the weighted allocation algorithm allocates tasks according to the actual computing power of the computing node. By assigning different weights to different computing nodes, nodes with stronger computing power can take on more tasks. The advantage of the weighted allocation algorithm is that it can allocate tasks according to the actual capabilities of the nodes, thereby improving the computing efficiency of the distributed system.

[0092] To this end, the present invention sets a weighted distribution algorithm - load formula, which assigns different weight coefficients to different task types. The implementation method is to save a coefficient table in the system, and different tasks correspond to different coefficients.

[0093] The calculation formula of computing power resource value RES is as follows:

[0094] Res=W1*K+W2*R+W3*B+W4*I

[0095] Among them, Res is the computing resource value, W1 is the first weight, K is the CPU / GPU load average, W2 is the second weight, R is the physical memory value, W3 is the third weight, B (bandwidth) is the bandwidth, W4 is the fourth weight, and I (I / O) is the I / O occupancy.

[0096] In order to better improve the utilization of computing resources, the present invention improves some previous algorithms through the load formula - giving weights to the CPU / GPU load average, physical memory value, bandwidth and I / O occupancy.

[0097] Different weight values ​​are set for the average cpuload / gpuload, available physical memory, bandwidth and I / O occupancy according to the requirements of different types of tasks. This is because the resource requirements of different tasks have different emphases. Therefore, in the load formula, the weights of the average CPU / GPU load, physical memory value, bandwidth and I / O occupancy are set and can be dynamically adjusted according to actual conditions. This improvement makes linear load computing power scheduling more responsive to actual task needs and improves the utilization rate of computing power resources.

[0098] The weight coefficients of different types of tasks are shown in Table 1:

[0099] Table 1 - Weight coefficients for different types of tasks

[0100]

[0101] Not only that, the rationality basis is also given for the setting of the coefficients in the table.

[0102] For CPU / GPU-Intensive tasks, which have high requirements for memory and CPU / GPU resources and relatively low requirements for bandwidth and I / O occupancy, their W1 and W2 coefficients are set to 0.5 and 0.3. Because compute-intensive tasks mainly consume CPU / GPU resources, they also require sufficient memory to store and process the data and temporary variables required for calculation. If the memory is insufficient, it may lead to frequent memory swaps, thereby reducing the execution efficiency of the task. Relatively speaking, compute-intensive tasks have low requirements for bandwidth and I / O occupancy. Because these tasks mainly rely on the CPU / GPU for calculations, data transmission and I / O operations are relatively less. Therefore, the W3 and W4 coefficients are set to 0.1 and 0.1.

[0103] For memory-intensive tasks that require a large amount of memory resources to run and process data, the W2 coefficient is set to 0.5. Memory-intensive tasks usually involve a large amount of data processing and computing operations, requiring the storage and operation of large-scale data sets in memory, and usually frequently access and operate data in memory without involving a large amount of computing or I / O operations. These tasks have a high demand for system memory and require sufficient memory resources to store and process data to ensure the performance and efficiency of the application. The performance of memory-intensive applications is affected by the system memory capacity and memory bandwidth, and memory resources need to be reasonably managed and optimized to meet the memory requirements of the application. In addition, although memory-intensive tasks mainly rely on memory access and operation, when processing large amounts of data, data needs to be frequently read and written from memory. This requires a high-speed data transmission channel, that is, bandwidth, to ensure that data can be quickly transmitted between memory and processor to avoid bottlenecks that affect the execution efficiency of tasks. Therefore, for memory-intensive tasks, high bandwidth is very important. Therefore, the W1, W3, and W4 coefficients are set to 0.1, 0.3, and 0.1.

[0104] For bandwidth-intensive / bandwidth throttling tasks, we mainly focus on the bandwidth requirements of the system's external communications. We need sufficient network bandwidth resources to support the characteristics of data transmission and communication operations. We set its W3 coefficient to 0.4. Relatively speaking, bandwidth-intensive tasks have lower requirements for CPU / GPU, memory, and I / O occupancy. Although some computing operations and memory accesses may be involved in the data transmission process, they are usually not the bottleneck of the task. Therefore, for such tasks, the main focus is on optimizing the utilization of bandwidth to improve the speed and efficiency of data transmission. Therefore, we set the W1, W2, and W4 coefficients to 0.2, 0.2, and 0.2, respectively.

[0105] For I / O-intensive tasks, which consume very little CPU / GPU and spend most of their time waiting for IO operations to complete (because the speed of I / O is much lower than the speed of CPU / GPU and memory), the W4 coefficient is set to 0.4. Secondly, in the operation of I / O-intensive task systems, most of the time the CPU / GPU is waiting for I / O (hard disk / memory) read / write operations, and the CPU / GPU loading is not high at this time. In addition, higher bandwidth and sufficient memory can effectively improve the execution efficiency and performance of I / O-intensive tasks. Therefore, the W1, W2, and W3 coefficients are set to 0.2, 0.2, and 0.2.

[0106] After obtaining the RES computing power resources of all nodes in the domain, the matching module is called to select the node with the largest RES value to assign the task to it for execution and create a contract to be recorded in the information lifecycle management unit.

[0107] In this embodiment, the task demand calculation module, node RES calculation module and matching module of the scheduling unit work together to significantly improve the efficiency and accuracy of task scheduling and resource allocation. The task demand calculation module accurately identifies the task resource requirements and provides a basis for resource matching; the node RES calculation module intelligently divides the node type according to the task priority and type, and calculates the node computing power resource value to ensure reasonable resource allocation; the matching module selects the best from the best, assigns the task to the node with the strongest computing power, and guarantees execution by creating a contract. This systematic design effectively avoids idle and overloaded resources, and greatly improves the utilization rate of computing power resources and the efficiency of task execution.

[0108] The monitoring unit refers to a unit that plays a monitoring function in the entire resource allocation. The monitoring unit is directly related to the resource management unit, task management unit, scheduling unit and information lifecycle management unit. It monitors the status of resources, tasks and contracts of the first three in real time and records them in the information lifecycle management unit.

[0109] like Figure 4 As shown, by monitoring resources, tasks, contracts, etc. through the monitoring unit, recording the release and recovery of resources, the type and amount of available resources of the node, the running status of the node, the unique identification of the task, the real-time status of the task (creation, execution and cancellation), the creation and destruction of the contract, etc., efficient utilization of resources can be achieved.

[0110] The so-called information lifecycle management unit is an important part of the entire resource allocation system. Its main function is to record information changes related to the resource management unit, task management unit, scheduling unit and monitoring unit. By recording the status of resources, the execution of tasks and the fulfillment of contracts in real time, the information lifecycle management unit can timely record the operation status of the system, especially key data and events.

[0111] The information lifecycle management unit is responsible for maintaining the interactive information between various units in the system to ensure smooth and efficient operation of the system. By collecting, storing and analyzing the monitored data, the information lifecycle management unit can help the system achieve reasonable allocation of resources, effective execution of tasks and accurate fulfillment of contracts. At the same time, the information lifecycle management unit can also provide important references for system optimization and help the system make better decisions through historical data analysis and trend prediction.

[0112] The information lifecycle management unit plays an important role in the entire resource allocation system. It promotes the coordinated operation and continuous optimization of the system by monitoring and recording key information.

[0113] Embodiment 2

[0114] like Figure 5 As shown, this embodiment discloses a distributed task resource scheduling method, comprising the following steps:

[0115] S1: Divide resource nodes into regular nodes and dedicated nodes according to their importance and urgency;

[0116] S2: adding attribute identifiers to tasks according to the relationship between distributed tasks; determining the priority of tasks according to the attribute identifiers, and placing tasks in a waiting queue based on the priority; the attribute identifiers include a string and a label, the string includes the task type, priority and system task number, and the label includes the subordination and dependency relationships;

[0117] S3: Configure node resources according to task priority and create a contract;

[0118] S4: Monitor the operating status of the resource management unit, task management unit and scheduling unit in real time, and record it in the information lifecycle management unit.

[0119] Embodiment 3

[0120] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the distributed task resource scheduling method described in the above-mentioned embodiment 2 are implemented.

[0121] Embodiment 4

[0122] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in a distributed task resource scheduling method as described in the above-mentioned embodiment 2 are implemented.

[0123] The steps or modules involved in the above embodiments 2 to 4 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed task resource scheduling system, characterized in that: include: A resource management unit, used to divide resource nodes into regular nodes and dedicated nodes according to their importance and urgency; The task management unit is used to add attribute identifiers to tasks according to the relationship between distributed tasks; determine the priority of the task according to the attribute identifier, and put the task into a waiting queue based on the priority; The attribute identifier includes a character string and a label, the character string includes the task type, priority and system task number, and the label includes the subordination and dependency relationship; The scheduling unit is used to configure node resources according to the priority of the task and create contracts; A monitoring unit is used to monitor the operating status of the resource management unit, the task management unit and the scheduling unit in real time and record it in the information life cycle management unit; The information lifecycle management unit is used to receive the operating status data sent by the monitoring unit.

2. A distributed task resource scheduling system as claimed in claim 1, characterized in that: The conventional nodes are working nodes for general urgent or important tasks, and the dedicated nodes are working nodes for urgent or important tasks.

3. A distributed task resource scheduling system as claimed in claim 1, characterized in that: The task types included in the character string include computation-intensive tasks, memory-intensive tasks, bandwidth-intensive tasks, and I / O-intensive tasks; The priorities include highest, second highest, medium, second lowest, and lowest.

4. A distributed task resource scheduling system as claimed in claim 1, characterized in that: The scheduling unit includes a task requirement calculation module, a node RES calculation module and a matching module; The task requirement calculation module is used to calculate the resource requirement type and required quantity of the task, determine the task type according to the task attribute identifier, and determine the node allocation type according to the task type; The node RES calculation module is used to obtain the load formula coefficient according to the task type and calculate the computing power resource value of all nodes in the entire domain of nodes applicable to the task; The matching module is used to assign tasks to the node with the largest computing power resource value according to the dynamic scheduling strategy, create a contract and record it in the information lifecycle management unit.

5. A distributed task resource scheduling system as claimed in claim 4, characterized in that: The load formula is specifically: Res=W1*K+W2*R+W3*B+W4*I Among them, Res is the computing resource value, W1 is the first weight, K is the average CPU or GPU load, W2 is the second weight, R is the physical memory value, W3 is the third weight, B is the bandwidth, W4 is the fourth weight, and I is the I / O occupancy.

6. A distributed task resource scheduling system as claimed in claim 4, characterized in that: The dynamic scheduling strategy includes: Obtain the initial priority of the task according to the task's label, and obtain its subordinate or dependent relationship through the task's label; According to the subordination or dependency of the task, check the priority of the tasks related to the task. If the priority of the tasks with subordination or dependency to the task does not match the priority of the task, adjust the priority of the task to a reasonable level and use the priority at this time as the final priority; otherwise, use the initial priority as the final priority for task scheduling.

7. A distributed task resource scheduling system as claimed in claim 1, characterized in that: The contract is an execution protocol established by the scheduling unit when selecting appropriate resources for the task through an allocation algorithm based on task priority and node resources, and includes task execution parameters, resource allocation rules, and conditions required for successful execution.

8. A distributed task resource scheduling method, characterized in that: include: According to the importance and urgency, resource nodes are divided into regular nodes and dedicated nodes; Add attribute identifiers to tasks based on the relationship between distributed tasks; Determine the priority of the task according to the attribute identifier, and put the task into the waiting queue based on the priority; The attribute identifier includes a character string and a label, the character string includes the task type, priority and system task number, and the label includes the subordination and dependency relationship; Configure node resources according to task priority and create contracts; Monitor the operating status of the resource management unit, task management unit and scheduling unit in real time, and record it in the information lifecycle management unit.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the steps in the distributed task resource scheduling method as claimed in claim 8 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the distributed task resource scheduling method as claimed in claim 8 are implemented.

Images