A method and apparatus for automatic discovery and avoidance of slow nodes in Hadoop

Abstract

This invention discloses a method and apparatus for automatic discovery and avoidance of slow nodes in Hadoop. The method includes: discovering slow nodes; packaging the slow node detection function into a big data application and publishing and submitting it together; in terms of computation, avoiding scheduling application executors to slow nodes; avoiding pulling data from slow nodes; stopping tasks currently executing on the slow node and rescheduling them to normal nodes. In terms of computation, this method and apparatus ensure that newly added application executors will not be scheduled to slow nodes; in terms of storage, it ensures that applications will not pull data from slow nodes, and that new data ingestion requests will not be stored on slow nodes; for executors currently computing on slow nodes, the process is stopped, and the application executor is rescheduled to a normal node to start.

Description

Technical Field

[0001] This invention relates to the field of data sources, and in particular to a method and apparatus for automatically discovering and avoiding slow nodes in Hadoop. Background Technology

[0002] Hadoop, a distributed system infrastructure developed by the Apache Software Foundation, is a general-purpose storage and analysis platform for big data. A Hadoop big data cluster consists of multiple servers; in large-scale clusters, the number of server nodes can reach tens of thousands. Slow nodes in a Hadoop cluster primarily exhibit slow network transmission and slow disk read / write speeds. When data is stored on a slow node or data processing tasks are scheduled to a slow node, the latency of the data processing task will be extremely high, with performance degrading by at least tenfold. Furthermore, because tasks are divided into numerous subtasks and run distributed across multiple nodes, the overall efficiency of the task depends on the result of the last subtask. Because tasks are constantly running, CPU and memory resources cannot be released, affecting the normal operation of other tasks. In severe cases, it can even lead to task execution failure.

[0003] In large-scale clusters, slow nodes are difficult to locate. Since the tasks assigned to the slow nodes are running normally, there is no abnormal log information in the application logs. The slow node machine load is normal, and conventional monitoring methods (such as monitoring memory utilization, CPU utilization, etc.) are also difficult to detect and alert.

[0004] When tasks run very slowly, it's necessary to manually analyze a large number of task logs to identify which nodes are slowing down. Then, further analysis of these nodes is required, including manual testing of network bandwidth and disk performance. This includes testing disk I / O performance, verifying data transfer rates over the network, and checking network card bandwidth to pinpoint the slow node. This entire process is very time-consuming and requires significant manual effort. Furthermore, even after identifying the slow node, it's impossible to prevent new tasks from being scheduled to it again unless the related services (such as NodeManager) are manually taken offline. Each server node has one NodeManager, responsible for node resource management and usage. For example, it starts an executor for a computation task, allocates memory and CPU to it, and reclaims the executor after the task completes.

[0005] The conventional approach to slow node discovery involves deploying a slow node detection program on each node to periodically test and collect metrics such as server disk and network speeds, then reporting the data to the main service program. However, this method is complex because it requires installing monitoring software on each machine. If any machine lacks this program, anomaly detection information will be lost. Furthermore, regular monitoring necessitates ensuring the stability of the monitoring program, and appropriate handling measures are required in case of anomalies. Upgrading the monitoring program involves updating it on each machine, resulting in high maintenance costs.

[0006] In this article, machine, node, server, and host are all the same concept.

[0007] like Figure 1 As shown, HDFS is a distributed file system primarily used for data storage, and it has two types of roles: the master node NameNode and the slave nodes DataNode.

[0008] A cluster has at least one NameNode. If the NameNode is configured as a primary and backup, then a cluster has two NameNode nodes.

[0009] A cluster has multiple DataNodes, which are primarily used for storing data.

[0010] If a dataset is too large, it can be distributed across multiple DataNodes, with each DataNode storing a portion of the data.

[0011] The functions of NameNode include: managing the HDFS namespace, managing data block mapping information, managing replication policies, and handling client read and write requests.

[0012] The functions of a DataNode include: Slave nodes, which have multiple execution functions for reading and writing data blocks.

[0013] like Figure 2 As shown, YARN is short for Yet Another Resource Negotiator, a cluster resource management system responsible for the unified management and scheduling of the cluster, and has scalability and fault tolerance.

[0014] Yarn in a big data cluster includes two roles: ResourceManager and NodeManager. ResourceManager is the master node, and NodeManager is the slave node.

[0015] There is only one ResourceManager in the entire cluster, which is responsible for the unified management and scheduling of cluster resources.

[0016] Main functions: handling client requests, starting / monitoring the ApplicationMaster, monitoring the NodeManager, resource allocation and scheduling.

[0017] There are many NodeManagers in the entire cluster. Each node has one NodeManager, which is responsible for node resource management and usage.

[0018] Main functions: resource management and task management for a single node, processing commands from ResourceManager, and processing commands from ApplicationMaster.

[0019] like Figure 3 As shown, the submission process for Yarn tasks is as follows:

[0020] Step 1: The client communicates with the ResourceManager to submit a request for a big data application.

[0021] Step 2: The ResourceManager receives the task submission request from the client, selects a suitable NodeManager node in the cluster, and prepares to start the ApplicationMaster.

[0022] Step 3: In the selected NodeManager node, start the ApplicationMaster.

[0023] Step 4: The ApplicationMaster and ResourceManager establish bidirectional communication. The ApplicationMaster informs the ResourceManager of the memory size, the number of CPU cores, and the number of executor processes to run the task.

[0024] Step 5: The ResourceManager receives the request from the ApplicationMaster, allocates resources to the ApplicationMaster, and the ApplicationMaster communicates with the allocated NodeManager node to request the start of the executor.

[0025] Step 6: The NodeManager node receives the resource request from the ApplicationMaster and starts one or more executor processes on this node to run tasks.

[0026] Explanation of related concepts:

[0027] Each application (task) has one ApplicationMaster, which is responsible for application management and task scheduling;

[0028] Detailed features: Requests resources for the application and further allocates them to internal tasks, with task monitoring and fault tolerance.

[0029] Executor is the execution environment for a task, used to execute a specific task. Summary of the Invention

[0030] To address the problems existing in the prior art, this invention provides a method and apparatus for automatic discovery and avoidance of slow nodes in Hadoop. In terms of computation, it ensures that newly added application executors will not be scheduled to slow nodes; in terms of storage, it ensures that applications will not pull data from slow nodes, and that newly added data entry requests will not be stored on slow nodes; for executors currently computing on slow nodes, the process is stopped, and the application executor is rescheduled to a normal node to start.

[0031] To achieve the above objectives, the present invention adopts the following technical solution:

[0032] When a big data application is executed, it is scheduled to run on multiple hosts. On each scheduled host, one or more executors are started to process the data in a distributed manner.

[0033] In one embodiment of the present invention, a method for automatic discovery and avoidance of slow Hadoop nodes is proposed, the method comprising:

[0034] S01. Discover slow nodes. Package the slow node detection and discovery function into the big data application and publish and submit it together.

[0035] S02. In terms of computation, avoid scheduling the application executor to a slow node;

[0036] S03, Avoid pulling data from slow nodes;

[0037] S04. Stop the tasks currently running on the slow node and reschedule them to a normal node.

[0038] Further, S01 includes:

[0039] S011. The user submits the application through the custom YARN API, preparing to start the application's main service, ApplicationMaster.

[0040] S012. Start ApplicationMaster and request resources;

[0041] In step S012, the hosts for the NodeManager are allocated, along with the number of application executors to be started on these allocated hosts, and the amount of memory and CPU resources to be allocated to each application executor. One or more executors can be started on one host. At this point, only the allocation operation is performed, and the application executors have not yet been actually started.

[0042] S013. Start the application executor. The ApplicationMaster receives the allocated resources and communicates directly with each allocated NodeManager to start the application executor.

[0043] S014. Nodes of actuators awaiting slow node monitoring have been detected;

[0044] S015. Start the slow node monitoring executor. On the host of the slow node monitoring executor, ApplicationMaster starts the slow node monitoring executor. On each assigned server, it retrieves the corresponding jar package from HDFS and starts the slow node monitoring executor.

[0045] The slow node monitoring executor performs slow node discovery functions, primarily in two aspects: disk and network bandwidth. Disk monitoring mainly involves monitoring the read / write operation time of each disk configured in the storage directory, including the number of monitoring sessions and the time of each read / write operation. Peak and off-peak read / write time thresholds are defined for normal traffic. If multiple consecutive read / write operations exceed the thresholds, a storage I / O performance issue is identified for the corresponding disk. Network bandwidth monitoring primarily involves monitoring whether the network interface card (NIC) speed is within the normal operating range. This includes checking the speed of the bound NIC using the `ethtool` command and testing the file transfer rate between the machine hosting the task executor and the machine receiving the data.

[0046] The machine where the task executor resides: The server to which the executor is assigned;

[0047] The machine from which data is pulled: The server assigned to the executor may need to pull data from other machines for computation. That is, the server from which the data is pulled is the machine where the data is stored.

[0048] When a slow node is detected, it is reported to the ApplicationMaster. For slow nodes, three measures are taken to ensure normal data processing without impacting performance.

[0049] (1) In terms of computation, ensure that newly added application executors will not be scheduled to slow nodes.

[0050] (2) In terms of storage, ensure that the application does not pull data from slow nodes, and that new data entry requests are not stored on slow nodes.

[0051] (3) For executors that are computing on slow nodes, stop the process and reschedule to start the application on normal nodes.

[0052] Further, S011 includes:

[0053] S0111. Use the YarnClient API to submit large data for processing;

[0054] S0112. When the ResourceManager receives the request submitted by the application, it selects a suitable NodeManager node in the big data cluster and prepares to start the application's main service, ApplicationMaster.

[0055] Furthermore, the applications include: Spark applications and MapReduce applications.

[0056] Further, S012 includes:

[0057] S0121. In the selected NodeManager node, start the ApplicationMaster;

[0058] S0122. When submitting a task through the YarnClient API, communicate with the started ApplicationMaster to inform it that it needs to request resources for slow node monitoring.

[0059] S0123. The ApplicationMaster and ResourceManager establish bidirectional communication. The ApplicationMaster informs the ResourceManager that it needs to request two types of resources: resources for running tasks and resources needed for slow node detection.

[0060] Resources refer to memory size and the number of CPUs.

[0061] S0124. Based on the resource request submitted by the ApplicationMaster, the ResourceManager allocates one or more NodeManager hosts for starting the executor process on these hosts.

[0062] Furthermore, the executor in S0124 includes: an application executor and a slow node detection executor.

[0063] Further, S014 includes:

[0064] S0141. ApplicationMaster breaks down the application into multiple subtasks. The subtasks run on the application's executor. Each application executor's subtask pulls file data from various nodes and performs calculations.

[0065] File data refers to files that store data, and the desired results are obtained through computation in the application code using frameworks like Spark or MapReduce. Computation can be simply understood as processing data, including string manipulation, data joins, data type conversions, and so on. The specific logic of how the computation is performed is implemented by the application code.

[0066] If there is a problem with the disk or network I / O on a node, it will lead to a serious performance bottleneck.

[0067] S0142. Each application executor sends the node information for pulling data to the ApplicationMaster;

[0068] The node information of the data refers to the host information, that is, the server information.

[0069] S0143. ApplicationMaster gathers the node information of the data pulled by the application and the node information of each application executor. ApplicationMaster then deduplicates these nodes.

[0070] S014 as a whole discovers which nodes the application is running on and which nodes the data it reads is stored on, and starts slow node monitoring executors on these associated nodes.

[0071] Furthermore, the node deduplication in S0143 includes: removing duplicate parts from the executors of multiple applications running on each node, or the node information pulling data, and retaining one executor for slow node monitoring for each node.

[0072] Further, S02 includes:

[0073] S021. Divide the resource pool for big data cluster computing according to the host;

[0074] S022, Slow Node Resource Pool Switching

[0075] Further, S021 includes:

[0076] S0211. When a slow node is detected, the slow node host is adjusted to the faulty resource pool through the host switching API of the resource pool, thereby ensuring that the executor of subsequent big data applications will not be scheduled to the slow node host.

[0077] S0212. Set the resource pool of the big data cluster to the Capacity Scheduler type;

[0078] S0213. Divide the cluster resources into multiple resource pools. Each resource pool allocates cluster resources according to a certain ratio. Each resource pool corresponds to one or more hosts, but one host can only belong to one resource pool.

[0079] S0214. When submitting a big data application, the name of the resource pool must be specified;

[0080] S0215. Configure a dedicated fault resource pool. All task submissions should not specify this resource pool.

[0081] Data is stored on slow nodes and services are provided through DataNodes. If the DataNode service is stopped, data will still be pulled from that node, which will cause a large number of application tasks to retry, thus slowing down the efficiency of application execution.

[0082] Further, S03 includes:

[0083] S031. For abnormal nodes, the slow node monitoring and processing program calls the NameNode's decommission command to remove the DataNode from the storage node, resolve the DataNode role of the slow node, and add the node to the NameNode's blacklist.

[0084] This ensures that no data read or write operation will select this node.

[0085] S032. Periodically check nodes in the contract termination state. If the network or disk problems of the slow node return to normal, the slow node monitoring program calls the NameNode's commission command to reauthorize the slow node as a normal node and perform normal data reading and writing.

[0086] Further, S04 includes:

[0087] When ApplicationMaster detects a slow node, it notifies the application executor where the slow node resides, calls the process destruction operation, and terminates the executor process.

[0088] ApplicationMaster requests another compute node from ResourceManager to start the application's executor and perform normal data processing.

[0089] In one embodiment of the present invention, an automatic discovery and avoidance device for Hadoop slow nodes is also proposed, the device comprising:

[0090] The module for discovering slow nodes should be packaged into the big data application and published together.

[0091] The slow node avoidance module, in terms of computation, avoids the application executor being scheduled to a slow node;

[0092] Avoid the data retrieval module and avoid retrieving data from slow nodes;

[0093] The scheduling module stops the tasks currently executing on the slow node and reschedules them to normal nodes.

[0094] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned Hadoop slow node automatic discovery and avoidance method.

[0095] In one embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that executes a Hadoop slow node auto-discovery and avoidance method.

[0096] Beneficial effects:

[0097] The present invention proposes a method and apparatus for automatic discovery and avoidance of slow nodes in Hadoop. The slow node monitoring program is released along with the big data application and performs monitoring while the big data application is running, which reduces the maintenance cost of the slow node monitoring program. Slow nodes can be dealt with in real time to ensure the normal execution of running applications. It can also ensure that newly added applications will not be scheduled to slow nodes or pull data from slow nodes. Attached Figure Description

[0098] Figure 1 This is a schematic diagram of the existing HDFS architecture;

[0099] Figure 2 This is a schematic diagram of the existing Yarn architecture;

[0100] Figure 3 This is a flowchart of the submission process for existing Yarn tasks;

[0101] Figure 4 This is a schematic diagram of the Hadoop slow node automatic discovery and avoidance method according to an embodiment of the present invention;

[0102] Figure 5 A schematic diagram of the scheduling structure of a big data application during execution in this invention;

[0103] Figure 6 This is a schematic diagram of the Hadoop slow node automatic discovery and avoidance device according to an embodiment of the present invention;

[0104] Figure 7 This is a schematic diagram of a computer device structure according to an embodiment of the present invention. Detailed Implementation

[0105] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0106] The technical terms involved in this invention and their explanations are as follows:

[0107] HDFS: Distributed File System;

[0108] ethtool: A Linux command used to query and set network interface card (NIC) parameters;

[0109] Capacity Scheduler: Container scheduler;

[0110] DataNode: Data node;

[0111] NameNode's decommission: The command to revoke the authorization of NameNode.

[0112] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0113] According to an embodiment of the present invention, a method and apparatus for automatic discovery and avoidance of Hadoop slow nodes are proposed. In terms of computation, it ensures that newly added application executors will not be scheduled to slow nodes; in terms of storage, it ensures that applications will not pull data from slow nodes, and newly added data entry requests will not be stored on slow nodes; for executors that are currently computing on slow nodes, the process is stopped and rescheduled to start the application executor on a normal node.

[0114] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0115] like Figure 5 As shown, a big data application, when executed, will be scheduled to run on multiple hosts. On each scheduled host, one or more executors will be started to process the data in a distributed manner.

[0116] Figure 4 This is a schematic diagram of a Hadoop slow node automatic discovery and avoidance method according to an embodiment of the present invention. Figure 4 As shown, the method includes:

[0117] S01. Discover slow nodes. Package the slow node detection and discovery function into the big data application and publish and submit it together.

[0118] S02. In terms of computation, avoid scheduling the application executor to a slow node;

[0119] S03, Avoid pulling data from slow nodes;

[0120] S04. Stop the tasks currently running on the slow node and reschedule them to a normal node.

[0121] S01 includes:

[0122] S011. The user submits the application through the custom YARN API, preparing to start the application's main service, ApplicationMaster.

[0123] S012. Start ApplicationMaster and request resources;

[0124] In step S012, the hosts for the NodeManager are allocated, along with the number of application executors to be started on these allocated hosts, and the amount of memory and CPU resources to be allocated to each application executor. One or more executors can be started on one host. At this point, only the allocation operation is performed, and the application executors have not yet been actually started.

[0125] S013. Start the application executor. The ApplicationMaster receives the allocated resources and communicates directly with each allocated NodeManager to start the application executor.

[0126] S014. Nodes of actuators awaiting slow node monitoring have been detected;

[0127] S015. Start the slow node monitoring executor. On the host of the slow node monitoring executor, ApplicationMaster starts the slow node monitoring executor. On each assigned server, it retrieves the corresponding jar package from HDFS and starts the slow node monitoring executor.

[0128] The slow node monitoring executor performs slow node discovery functions, primarily in two aspects: disk and network bandwidth. Disk monitoring mainly involves monitoring the read / write operation time of each disk configured in the storage directory, including the number of monitoring sessions and the time of each read / write operation. Peak and off-peak read / write time thresholds are defined for normal traffic. If multiple consecutive read / write operations exceed the thresholds, a storage I / O performance issue is identified for the corresponding disk. Network bandwidth monitoring primarily involves monitoring whether the network interface card (NIC) speed is within the normal operating range. This includes checking the speed of the bound NIC using the `ethtool` command and testing the file transfer rate between the machine hosting the task executor and the machine receiving the data.

[0129] The machine where the task executor resides: The server to which the executor is assigned;

[0130] The machine from which data is pulled: The server assigned to the executor may need to pull data from other machines for computation. That is, the server from which the data is pulled is the machine where the data is stored.

[0131] When a slow node is detected, it is reported to the ApplicationMaster. For slow nodes, three measures are taken to ensure normal data processing without impacting performance.

[0132] (1) In terms of computation, ensure that newly added application executors will not be scheduled to slow nodes.

[0133] (2) In terms of storage, ensure that the application does not pull data from slow nodes, and that new data entry requests are not stored on slow nodes.

[0134] (3) For executors that are computing on slow nodes, stop the process and reschedule to start the application on normal nodes.

[0135] S011 includes:

[0136] S0111. Use the YarnClient API to submit large data for processing;

[0137] S0112. When the ResourceManager receives the request submitted by the application, it selects a suitable NodeManager node in the big data cluster and prepares to start the application's main service, ApplicationMaster.

[0138] The applications include: Spark applications and MapReduce applications.

[0139] S012 includes:

[0140] S0121. In the selected NodeManager node, start the ApplicationMaster;

[0141] S0122. When submitting a task through the YarnClient API, communicate with the started ApplicationMaster to inform it that it needs to request resources for slow node monitoring.

[0142] S0123. The ApplicationMaster and ResourceManager establish bidirectional communication. The ApplicationMaster informs the ResourceManager that it needs to request two types of resources: resources for running tasks and resources needed for slow node detection.

[0143] Resources refer to memory size and the number of CPUs.

[0144] Based on the resource request submitted by the ApplicationMaster, the ResourceManager allocates one or more NodeManager hosts for starting the executor process on these hosts.

[0145] The executor in S0124 includes: the application executor and the slow node detection executor.

[0146] S014 includes:

[0147] S0141. ApplicationMaster breaks down the application into multiple subtasks. The subtasks run on the application's executor. Each application executor's subtask pulls file data from various nodes and performs calculations.

[0148] File data refers to files that store data, and the desired results are obtained through computation in the application code using frameworks like Spark or MapReduce. Computation can be simply understood as processing data, including string manipulation, data joins, data type conversions, and so on. The specific logic of how the computation is performed is implemented by the application code.

[0149] If there is a problem with the disk or network I / O on a node, it will lead to a serious performance bottleneck.

[0150] S0142. Each application executor sends the node information for pulling data to the ApplicationMaster;

[0151] The node information of the data refers to the host information, that is, the server information.

[0152] S0143. ApplicationMaster gathers the node information of the data pulled by the application and the node information of each application executor. ApplicationMaster then deduplicates these nodes.

[0153] S014 as a whole discovers which nodes the application is running on and which nodes the data it reads is stored on, and starts slow node monitoring executors on these associated nodes.

[0154] The node deduplication in S0143 includes: removing duplicate parts from the executors of multiple applications running on each node, or the node information pulling data, and retaining one executor for slow node monitoring for each node.

[0155] S02 includes:

[0156] S021. Divide the resource pool for big data cluster computing according to the host;

[0157] S022, Slow Node Resource Pool Switching

[0158] S021 includes:

[0159] S0211. When a slow node is detected, the slow node host is adjusted to the faulty resource pool through the host switching API of the resource pool, thereby ensuring that the executor of subsequent big data applications will not be scheduled to the slow node host.

[0160] S0212. Set the resource pool of the big data cluster to the Capacity Scheduler type;

[0161] S0213. Divide the cluster resources into multiple resource pools. Each resource pool allocates cluster resources according to a certain ratio. Each resource pool corresponds to one or more hosts, but one host can only belong to one resource pool.

[0162] S0214. When submitting a big data application, the name of the resource pool must be specified;

[0163] S0215. Configure a dedicated fault resource pool. All task submissions should not specify this resource pool.

[0164] Data is stored on slow nodes and services are provided through DataNodes. If the DataNode service is stopped, data will still be pulled from that node, which will cause a large number of application tasks to retry, thus slowing down the efficiency of application execution.

[0165] S03 includes:

[0166] S031. For abnormal nodes, the slow node monitoring and processing program calls the NameNode's decommission command to remove the DataNode from the storage node, resolve the DataNode role of the slow node, and add the node to the NameNode's blacklist.

[0167] This ensures that no data read or write operation will select this node.

[0168] S032. Periodically check nodes in the contract termination state. If the network or disk problems of the slow node return to normal, the slow node monitoring program calls the NameNode's commission command to reauthorize the slow node as a normal node and perform normal data reading and writing.

[0169] S04 includes:

[0170] When ApplicationMaster detects a slow node, it notifies the application executor where the slow node resides, calls the process destruction operation, and terminates the executor process.

[0171] ApplicationMaster requests another compute node from ResourceManager to start the application's executor and perform normal data processing.

[0172] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0173] To provide a clearer explanation of the above-mentioned method for automatically discovering and avoiding slow Hadoop nodes, a specific embodiment will be used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0174] S201: Manually save the address and the corresponding service into the h.txt file.

[0175] The slow node monitoring program is provided as a JAR file. During deployment, the path to the slow node monitoring JAR file is specified using the `--jars` option. The specific command is as follows:

[0176] spark-submit \

[0177] --master yarn # Specifies that tasks should be submitted via yarn

[0178] --class org.tech.AppClass # Specifies the application's class

[0179] --executor-nums 10 # Specifies the number of executors for the application.

[0180] --executor-cores 4 # Specifies the number of CPU cores used for execution per application

[0181] --queue root.app # Specifies the resource pool name. Note that it cannot be a faulty resource pool.

[0182] --jars / home / app / slow-detect.jar # Path to the slow node detection program's JAR file

[0183] / home / app / myapp.jar # Path to the application's JAR file

[0184] Based on the same inventive concept, this invention also proposes an automatic discovery and avoidance device for Hadoop slow nodes. The implementation of this device can refer to the implementation of the method described above, and repeated details will not be repeated. The term "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0185] Figure 6 This is a schematic diagram of the Hadoop slow node automatic discovery and avoidance device according to an embodiment of the present invention. Figure 6 As shown, the device includes:

[0186] Slow Node Discovery Module 101: Package the slow node detection and discovery function into the big data application and publish and submit it together;

[0187] Slow node avoidance module 102: In terms of computation, it avoids the application executor being scheduled to a slow node;

[0188] Avoid data retrieval module 103; avoid retrieval of data from slow nodes;

[0189] Scheduling module 104: Stop the task being executed on the slow node and reschedule it to a normal node.

[0190] It should be noted that although several modules of the Hadoop slow node auto-discovery and avoidance device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0191] Based on the aforementioned inventive concept, such as Figure 7 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned Hadoop slow node automatic discovery and avoidance method.

[0192] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned Hadoop slow node automatic discovery and avoidance method.

[0193] The present invention proposes a method and apparatus for automatic discovery and avoidance of slow nodes in Hadoop. The slow node monitoring program is released along with the big data application and performs monitoring while the big data application is running, which reduces the maintenance cost of the slow node monitoring program. Slow nodes can be dealt with in real time to ensure the normal execution of running applications. It can also ensure that newly added applications will not be scheduled to slow nodes or pull data from slow nodes.

[0194] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0195] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for automatic discovery and avoidance of slow nodes in Hadoop, characterized in that, The method includes: S01. Discover slow nodes. Package the slow node detection and discovery function into the big data application and publish and submit it together. S01 includes: S011. The user submits the application through the custom YARN API, preparing to start the application's main service, ApplicationMaster. S012. Start ApplicationMaster and request resources; S013. Start the application executor. The ApplicationMaster receives the allocated resources and communicates directly with each allocated NodeManager to start the application executor. S014. Nodes of actuators awaiting slow node monitoring have been detected; S015. Start the slow node monitoring executor. On the host of the slow node monitoring executor, ApplicationMaster starts the slow node monitoring executor. On each assigned server, it retrieves the corresponding jar package from HDFS and starts the slow node monitoring executor. S02. In terms of computation, avoid scheduling the application executor to a slow node; S03, Avoid pulling data from slow nodes; S03 includes: S031. For abnormal nodes, the slow node monitoring and processing program calls the NameNode's decommission command to remove the DataNode from the storage node, resolve the DataNode role of the slow node, and add the node to the NameNode's blacklist. S032. Periodically check nodes in the contract termination state. If the network or disk problems of the slow node return to normal, the slow node monitoring program calls the NameNode's commission command to reauthorize the slow node as a normal node and perform normal data reading and writing. S04. Stop the task being executed on the slow node and reschedule it to a normal node.

2. The Hadoop slow node automatic discovery and avoidance method according to claim 1, characterized in that, S011 includes: S0111. Use the YarnClient API to submit large data for processing; S0112. When the ResourceManager receives the request submitted by the application, it selects a suitable NodeManager node in the big data cluster and prepares to start the application's main service, ApplicationMaster.

3. The Hadoop slow node automatic discovery and avoidance method according to claim 2, characterized in that, The applications include: Spark applications and MapReduce applications.

4. The Hadoop slow node automatic discovery and avoidance method according to claim 1, characterized in that, S012 includes: S0121. In the selected NodeManager node, start the ApplicationMaster; S0122. When submitting a task through the YarnClient API, communicate with the started ApplicationMaster to inform it that it needs to request resources for slow node monitoring. S0123. The ApplicationMaster and ResourceManager establish bidirectional communication. The ApplicationMaster informs the ResourceManager that it needs to request two types of resources: resources for running tasks and resources needed to perform slow node detection. Resources refer to memory size and the number of CPUs. S0124. Based on the resource request submitted by the ApplicationMaster, the ResourceManager allocates one or more NodeManager hosts for starting the executor process on these hosts.

5. The Hadoop slow node automatic discovery and avoidance method according to claim 4, characterized in that, The executor in S0124 includes: the application executor and the slow node detection executor.

6. The Hadoop slow node automatic discovery and avoidance method according to claim 1, characterized in that, S014 includes: S0141. ApplicationMaster breaks down the application into multiple subtasks. The subtasks run on the application's executor. Each application executor's subtask pulls file data from various nodes and performs calculations. S0142. Each application executor sends the node information for pulling data to the ApplicationMaster; S0143. ApplicationMaster gathers the node information of the data pulled by the application and the node information of each application executor. ApplicationMaster then deduplicates these nodes.

7. The Hadoop slow node automatic discovery and avoidance method according to claim 6, characterized in that, The node deduplication in S0143 includes: removing duplicate parts from the executors of multiple applications running on each node, or the node information pulling data, and retaining one executor for slow node monitoring for each node.

8. The Hadoop slow node automatic discovery and avoidance method according to claim 1, characterized in that, S02 includes: S021. Divide the resource pool for big data cluster computing according to the host; S022, Slow node resource pool switching.

9. The Hadoop slow node automatic discovery and avoidance method according to claim 8, characterized in that, S021 includes: S0211. When a slow node is detected, the slow node host is adjusted to the faulty resource pool through the host switching API of the resource pool, thereby ensuring that the executor of subsequent big data applications will not be scheduled to the slow node host. S0212. Set the resource pool of the big data cluster to the Capacity Scheduler type; S0213. Divide the cluster resources into multiple resource pools. Each resource pool allocates cluster resources according to a certain ratio. Each resource pool corresponds to one or more hosts, but one host can only belong to one resource pool. S0214. When submitting a big data application, the name of the resource pool must be specified; S0215. Configure a dedicated fault resource pool. All task submissions should not specify this resource pool.

10. The Hadoop slow node automatic discovery and avoidance method according to claim 1, characterized in that, S04 includes: When ApplicationMaster detects a slow node, it notifies the application executor where the slow node resides, calls the process destruction operation, and terminates the executor process. ApplicationMaster requests another compute node from ResourceManager to start the application's executor and perform normal data processing.

11. A device for automatic discovery and avoidance of slow nodes in Hadoop, characterized in that, The device includes: The module for discovering slow nodes should be packaged into the big data application and published together. The slow node detection module includes: Users submit their applications via a custom YARN API, preparing to launch the application's main service, ApplicationMaster. Start ApplicationMaster and request resources; The ApplicationMaster receives the allocated resources and communicates directly with each allocated NodeManager to start the application's executor. The node of the executor to be started for slow node monitoring was discovered; Start the slow node monitoring executor. On the host of the slow node monitoring executor, ApplicationMaster starts the slow node monitoring executor. On each assigned server, it retrieves the corresponding jar package from HDFS and starts the slow node monitoring executor. The slow node avoidance module, in terms of computation, avoids the application executor being scheduled to a slow node; Avoid the data retrieval module and avoid retrieving data from slow nodes; The data evasion avoidance module includes: For abnormal nodes, the slow node monitoring and handling program calls the NameNode's decommission command to remove the DataNode from the storage node, resolve the DataNode role of the slow node, and add the node to the NameNode's blacklist. The slow node monitoring program periodically checks the nodes in the contract termination status. If the network or disk problems of the slow node return to normal, the slow node monitoring program calls the NameNode's commission command to reauthorize the slow node as a normal node and enable it to perform normal data reading and writing. The scheduling module stops the tasks currently executing on the slow node and reschedules them to normal nodes.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1-10.

Images