A Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm

Abstract

The present invention discloses a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which is applied to a Hadoop resource allocation center. The cluster task scheduling method comprises: an idle computing node applies to a management node for a task execution request; the management node receives the task request and queries whether there is a historical record of running the task request; the historical records are compared and analyzed, and if the task configuration meets the resource requirements, the task is calculated in this computing node; otherwise, the computing node is re-matched with a new task request; and computing tasks can be dynamically allocated in cluster task scheduling according to the resource usage of different servers and the differences in server performance, thereby achieving efficient completion of computing tasks and improving CPU and resource utilization efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of cluster task scheduling, and in particular relates to a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm. Background Art

[0002] One of the core applications of the Hadoop platform is job scheduling, which assigns computing tasks to compute nodes on demand. This allocation method directly impacts the resource utilization and overall performance of the Hadoop platform. Typically, the platform uses a queue-based, first-come, first-served approach for scheduling. However, this scheduling approach lacks interactivity and cannot effectively allocate tasks on demand to maximize resource utilization. Improper job scheduling algorithms can lead to an imbalanced distribution of cluster resources and tasks, impacting cluster efficiency. To optimize job scheduling algorithms and address their challenges, Yahoo and Facebook have developed the Capacity Scheduler and Fair Scheduler algorithms, respectively, to address these issues. These algorithms rely on administrators to manually configure cluster compute nodes based on task resource consumption and cluster node performance. While these approaches address the challenges of first-come, first-served scheduling to some extent, they still fall short of achieving a fully automated solution. Therefore, research on the Hadoop resource scheduling framework (Yarn) and distributed file system (HDFS) to address the shortcomings of existing resource scheduling algorithms is crucial for improving the overall performance and resource utilization of cloud computing tasks.

[0003] The primary application of job scheduling algorithms is to rationally allocate the multiple tasks within each job according to their needs, ensuring efficient execution on each assigned node. Currently, existing job scheduling algorithms have certain flaws. For example, queue-based scheduling algorithms are generally used when cluster nodes have similar configurations and are running the same jobs and tasks. They are only suitable for specific scenarios where the jobs and tasks are identical or substantially similar. While computing power scheduling algorithms can improve system utilization and automatically allocate tasks based on job execution efficiency, they require manual pre-configuration of hardware conditions and cannot achieve automated machine allocation. Existing scheduling methods have not yet fundamentally addressed these issues.

[0004] Chinese patent application No. 201710113656.8 discloses a distributed task scheduling system and method for improving the efficiency and reliability of task scheduling execution. The system includes: a task submission cluster, a task scheduling cluster, and a task execution cluster; the task submission cluster includes multiple task client nodes, the task scheduling cluster includes multiple task scheduling nodes, and the task execution cluster includes multiple task execution nodes; the task client nodes in the task submission cluster are used to submit tasks to the task scheduling cluster; the task scheduling nodes in the task scheduling cluster are used to receive tasks, generate task assignment information for the tasks, and send the tasks to the task execution cluster according to the task assignment information; the task execution nodes in the task execution cluster are used to execute tasks and return the execution results to the task client nodes that submitted the tasks. The above-mentioned existing technologies require manual configuration of hardware conditions in advance and cannot achieve machine automatic allocation, resulting in uneven resource distribution and low resource utilization, which affects the efficient operation and stability of the cluster. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which can dynamically allocate computing tasks according to the resource usage of different servers and the differences in server performance in cluster task scheduling, thereby achieving efficient completion of computing tasks and improving CPU and resource utilization efficiency.

[0006] The present invention provides the following technical solutions:

[0007] A Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm is applied to the Hadoop resource allocation center. The cluster task scheduling method includes:

[0008] Idle computing nodes apply to the management node for task execution requests;

[0009] The management node receives the task request and checks whether there is a history record of running the task request;

[0010] After comparing and analyzing historical records, if the resource requirements are met with the task configuration, the task is calculated on this computing node; otherwise, the computing node is re-matched with the new task request.

[0011] Preferably, when the Node Manager of an idle Data Node computing node in the Hadoop cluster applies to the Resource Manager of the management node Name Node for task execution resources, the application information includes the memory of each computing node, disk I / O read / write rate, network bandwidth, current memory size, and disk margin.

[0012] Preferably, before the management node receives a task request and queries whether there is a historical record of running the task request, the management node randomly pulls a computing task of the Name Node computing node from the task queue or CPU running queue and disk load queue that have been initialized by the system, and queries the completed task record before the task is calculated.

[0013] Preferably, if the query history record table does not contain relevant information about the running task, the Resource Manager in the management node Name Node directly calculates the task on this Data Node computing node. After the Name Node completes the calculation task, it stores the entire calculation task in the history record table according to the job duration, CPU utilization and disk status.

[0014] Preferably, if the query history record table has relevant information about the running task, the currently assigned task is compared with the historical record information. After analyzing the historical running records and the task configuration meets the current resource requirements, the Resource Manager in the management node Name Node directly calculates this task on this Data Node computing node.

[0015] Preferably, if the historical records are compared and it is found that the initial configuration of the current task is inconsistent, the task is taken out and saved in a new queue, and then a new task is taken out from the original queue, and the management node re-queries and compares until a reasonable task is matched.

[0016] Preferably, after matching a reasonable task, the management node Name Node completes the computing task and stores the entire computing task in a history record table based on the job duration, CPU utilization, and disk status.

[0017] Preferably, when assigning tasks, the Data Node requests computing resources from the Management Node. The request includes detailed resource information for the compute node, including CPU usage, disk and memory information, and so on. The Name Node stores the requested resources in the corresponding queue. The Name Node then removes tasks from the queue and randomly assigns them to the Data Node.

[0018] Preferably, during resource detection, after the Data Node computing node receives the task assigned by the Name Node management node, it first queries the historical operation record table to verify whether it can meet the requirements for running the relevant computing task.

[0019] Preferably, if the Data Node computing node does not meet the conditions after receiving the task assigned by the Name Node management node, feedback is triggered, that is, the Data Node computing node sends a running failure signal to the management node, requests a new computing task and carries detailed resource information of the node.

[0020] Preferably, the task is reallocated. After receiving the computing node failure information, the Name Node management node re-adds the task to the queue and marks the detailed information of the node failure to wait for the management node to reallocate the task.

[0021] In addition, this technical solution can quantify the server resource utilization. Assume that the amount of data in the computing resource is ID and the amount of data output is OD. The amount of data processed by the computer resource changes. Assuming the change rate is P, then OD = P*ID, the time for a task to execute is TCT, and the frequency of disk processing and memory is IOR. Therefore, Therefore, the feedback logic can be obtained. If the IOR frequency is greater than the set threshold, the feedback mechanism is triggered, otherwise the calculation task is executed at this node.

[0022] In addition, a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm adopts a cluster task scheduling system, which includes a processor, a memory, a communication unit, an input unit, and a programmable logic unit; the processor is a central processing unit.

[0023] Preferably, a cluster task scheduling system further includes a communication bus, the communication bus uses I 2 C communication bus; the processor, memory, communication unit, input unit, and programmable logic unit are all connected via I 2 C communication bus to complete the communication between each other.

[0024] The memory stores program instruction codes, which are computer operation instructions. When performing cluster task scheduling, the memory contains at least computer program instructions that implement the following functions: an idle computing node applies to the management node for a task execution request; the management node receives the task request and queries whether there is a historical record of running the task request; the historical records are compared and analyzed. If the resource requirements are met with the task configuration, the task is calculated in this computing node; otherwise, the computing node is re-matched with a new task request.

[0025] In addition, when performing cluster task scheduling, idle computing nodes are used to apply to the management node for task execution requests; the management node receives the task request and queries whether there is a historical record of running the task request; the historical records are compared and analyzed. If the resource requirements are met with the task configuration, the task is calculated in this computing node; otherwise, the computing node is re-matched with the new task request to ensure balanced resource allocation within the cluster, improve resource utilization, and ensure efficient and stable operation of the cluster.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) The present invention provides a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which can dynamically allocate computing tasks according to the resource usage of different servers and the differences in server performance in cluster task scheduling, thereby achieving efficient completion of computing tasks and improving CPU and resource utilization efficiency.

[0028] (2) The present invention provides a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, which ensures balanced resource allocation within the cluster, improves resource utilization, and ensures efficient and stable operation of the cluster.

[0029] (3) The present invention provides a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm. By reviewing the running conditions and running status of the tasks, the scheduling tasks are reclassified. A learning model is used to solve the deficiency of the current algorithm that requires pre-setting of the hardware, thereby solving the problem that the existing technology cannot realize automatic machine allocation, resulting in uneven resource distribution and low resource utilization, and improving the operating efficiency of the Hadoop platform.

[0030] (4) The present invention provides a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm. This method solves the problem of uneven resource allocation and inability to allocate resources on demand when allocating tasks on a traditional Hadoop platform. It also solves the problem of resource allocation not being automated due to the need for manual configuration in the optimization algorithm. This method truly enables the Hadoop cluster to automatically allocate tasks on demand based on cluster resources and the amount of computing tasks during task scheduling, fundamentally improving the rationality of task scheduling and allocation, increasing computing efficiency, optimizing CPU resource utilization, and enabling the Hadoop platform to be well adapted to a variety of distributed storage and high-concurrency applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 It is a flow chart of the method of the present invention.

[0033] Figure 2 It is a flow chart of the cluster task scheduling algorithm of the present invention.

[0034] Figure 3 The present invention provides a structural block diagram of a cluster task scheduling system. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0037] Example 1:

[0038] like Figure 1-2 As shown, a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm is applied to the Hadoop resource allocation center. The cluster task scheduling method includes:

[0039] S1, the idle computing node applies to the management node for task execution request;

[0040] S2, the management node receives the task request and checks whether there is a history record of running the task request;

[0041] S3, historical records are compared and analyzed. If the resource requirements are met with the task configuration, the task is calculated in this computing node. Otherwise, the computing node is re-matched with the new task request.

[0042] When the Node Manager of an idle Data Node computing node in the Hadoop cluster applies to the Resource Manager of the management node Name Node for task execution resources, the application information includes the memory of each computing node, disk I / O read / write rate, network bandwidth, current memory size, and disk remaining space.

[0043] When the management node receives a task request and checks whether there is a history of running the task request, it randomly pulls a computing task of the NameNode computing node from the system's initialized task queue, CPU running queue, and disk load queue, and queries the completed task record before the task is calculated.

[0044] If the query history table does not contain relevant information about the running task, the Resource Manager in the management node Name Node directly calculates the task on this Data Node computing node. After the Name Node completes the calculation task, it stores the entire calculation task in the history table based on the job duration, CPU utilization, and disk status.

[0045] If the query history table contains relevant information about the running task, the currently assigned task is compared with the historical information. After analyzing the historical running records and the task configuration meets the current resource requirements, the Resource Manager in the management node NameNode directly calculates this task on this DataNode computing node.

[0046] If the comparison with the historical records reveals a discrepancy with the current task's initial configuration, the task is removed and saved to a new queue. A new task is then retrieved from the original queue, and the management node re-queries and compares until a suitable task is found. After a suitable task is found, the management node, the Name Node, completes the computation and stores the entire task in a historical record table, based on job duration, CPU utilization, and disk usage.

[0047] Example 2:

[0048] Based on Example 1, during task assignment, the Data Node requests computing resources from the management node. The request includes detailed resource information for the compute node, including CPU usage, disk and memory details, and so on. The Name Node stores the requested resources in the corresponding queue. The Name Node then removes tasks from the queue and randomly assigns them to the Data Node. During resource detection, after receiving a task assigned by the Name Node, the Data Node first queries the historical operation record table to verify whether it can run the relevant computing task. If the task assigned by the Name Node does not meet the requirements, feedback is triggered. The Data Node sends a failure signal to the management node, requesting a new computing task and carrying detailed resource information for the node. Task reassignment: Upon receiving the failure information for the compute node, the Name Node requeues the task and marks the node's failure details, waiting for the management node to reassign the task.

[0049] Example 3:

[0050] like Figure 3 As shown, based on the first embodiment, a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm adopts a cluster task scheduling system, which includes a processor, a memory, a communication unit, an input unit, and a programmable logic unit; the processor is a central processing unit. A cluster task scheduling system also includes a communication bus, which uses I 2 C communication bus; the processor, memory, communication unit, input unit, and programmable logic unit are all connected via I 2 C communication bus to complete the communication between each other.

[0051] The memory stores program instruction codes, which are computer operation instructions. When performing cluster task scheduling, the memory contains at least computer program instructions that implement the following functions: an idle computing node applies to the management node for a task execution request; the management node receives the task request and queries whether there is a historical record of running the task request; the historical records are compared and analyzed. If the resource requirements are met with the task configuration, the task is calculated in this computing node; otherwise, the computing node is re-matched with a new task request.

[0052] Example 4

[0053] On the basis of Example 1, in addition, through this technical solution, the server resource utilization can be quantified. Assume that the amount of data in the computing resource is ID and the amount of data output is OD. The amount of data processed by the computer resource changes. Assuming the change rate is P, then OD=P*ID, the time for a task execution is TCT, and the frequency of disk processing and memory is IOR. Therefore, Therefore, the feedback logic can be obtained. If the IOR frequency is greater than the set threshold, the feedback mechanism is triggered, otherwise the calculation task is executed at this node.

[0054] When performing cluster task scheduling, idle computing nodes are used to apply to the management node for task execution requests; the management node receives the task request and queries whether there is a historical record of running the task request; the historical records are compared and analyzed. If the task configuration meets the resource requirements, the task is calculated in this computing node; otherwise, the computing node is re-matched with the new task request to ensure balanced resource allocation within the cluster, improve resource utilization, and ensure efficient and stable operation of the cluster.

[0055] The device obtained by the above technical solution is a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm. It can dynamically allocate computing tasks according to the resource usage and performance differences of different servers during cluster task scheduling, achieving efficient completion of computing tasks and improving CPU and resource utilization efficiency. It ensures balanced resource allocation within the cluster, improves resource utilization, and ensures efficient and stable operation of the cluster. By reviewing the operation status and running conditions of tasks, it reclassifies scheduled tasks and uses a learning model to address the deficiency of the existing algorithm requiring pre-configured hardware. This solves the problem of uneven resource distribution and low resource utilization caused by the inability to achieve automated machine allocation in the existing technology, thereby improving the operating efficiency of the Hadoop platform. It solves the problem of uneven resource allocation and lack of on-demand allocation in the traditional Hadoop platform when allocating tasks, and also addresses the drawback of the optimization algorithm that requires manual configuration and cannot be automated to achieve resource allocation. It truly realizes fully automatic on-demand allocation of Hadoop cluster tasks according to cluster resources and computing task load during task scheduling, fundamentally improving the rationality of task scheduling and allocation, improving computing efficiency, optimizing CPU resource utilization, and making the Hadoop platform suitable for a variety of distributed storage and high-concurrency applications.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, applied to a Hadoop resource allocation center, characterized in that: Cluster task scheduling methods include: Idle computing nodes apply to the management node for task execution requests; The management node receives the task request and checks whether there is a history record of running the task request; Compare and analyze historical records. If the resource requirements are met with the task configuration, the task is calculated on this computing node. Otherwise, the computing node is re-matched with the new task request. When the Node Manager of an idle Data Node compute node in the Hadoop cluster requests resources for executing tasks from the Resource Manager of the Name Node, the request information includes the memory, disk I / O read / write rate, network bandwidth, current memory size, and disk remaining space of each compute node. During task allocation, the Data Node requests computing resources from the Management Node. The request contains detailed resource information for the compute node, including CPU usage, disk, and memory. The Name Node stores the requested resources in the corresponding queue. The Name Node then takes tasks from the queue and randomly assigns them to the Data Node. During resource detection, after the Data Node compute node receives the task assigned by the Name Node management node, it first queries the historical operation record table to verify whether it can meet the conditions for running the related computing tasks; after the Data Node compute node receives the task assigned by the Name Node management node, if the conditions are not met, feedback is triggered, that is, the Data Node compute node sends a running failure signal to the management node, requests a new computing task and carries detailed resource information of the node; the task is reallocated, and after receiving the computing node failure information, the Name Node management node re-queues the task and marks the detailed information of the node running failure, waiting for the management node to reallocate the task.

2. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm according to claim 1, characterized in that: When a management node receives a task request and checks whether there is a history of running the task request, it randomly pulls a computing task of the NameNode management node from the system's initialized task queue, CPU run queue, and disk load queue, and queries the completed task record before the task is calculated.

3. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm according to claim 2, characterized in that: The query history table does not contain relevant information about the task. The Resource Manager in the management node Name Node directly calculates the task on this Data Node computing node. After the Name Node completes the calculation task, it stores the entire calculation task in the history table based on the job duration, CPU utilization, and disk status.

4. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm according to claim 1, characterized in that: If the query history table contains relevant information about running the task, the currently assigned task is compared with the historical information. After analyzing the historical running records and confirming that the task configuration meets the current resource requirements, the Resource Manager in the management node Name Node directly calculates this task on this Data Node computing node.

5. A Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm according to claim 4, characterized in that: If the comparison with the historical records shows that the initial configuration does not match the current task, the task will be taken out and saved in a new queue, and then a new task will be taken out from the original queue. The management node will then re-query and compare until a reasonable task is matched.

6. The Hadoop cluster task scheduling method based on the positive and negative feedback load scheduling algorithm according to claim 5, characterized in that: After matching a reasonable task, the management node Name Node completes the computing task and stores the entire computing task in the history table based on the job duration, CPU utilization, and disk status.

Images