Method and device for elastically and telescopically supporting NodeManager resource scheduling of big data cluster

Abstract

The invention discloses a method and device for elastically and telescopically supporting NodeManager resource scheduling of a big data cluster. The method comprises the steps that S01, host load evaluation and resource management are carried out, and whether the load of a host is excessive or not is judged; s02, according to a NodeManager resource optimization strategy, dynamic optimization of resources is carried out; s03, automatic management of NodeManager resources: according to the data obtained in the step S01, determining whether the resource allocation of the NodeManager needs to be adjusted or not; and S04, identifying and processing NodeManager resource saturation, and discovering a NodeManager machine with resource saturation in time by continuously monitoring a cluster state. According to the method and the device for elastically and telescopically supporting the NodeManager resource scheduling of the big data cluster, the NodeManager resource scheduling of the big data cluster can support automatic elastic telescoping through the technical means of improving the NodeManager resource utilization rate of a single physical machine, controlling the upper limit of the NodeManager resource of an over-configured CPU and a memory, discovering and responding to a saturated machine, expelling running tasks and the like.

Description

Technical Field

[0001] The present invention relates to the field of data resource scheduling, and in particular to a method and device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster. Background Art

[0002] With the rapid development of business, the scale of Hadoop big data clusters is getting larger and larger, and the demand for NodeManager resources such as CPU and memory is also increasing. This will bring challenges to the scalability, NodeManager resource utilization, cost-effectiveness and ability to cope with sudden demand of big data clusters, and will face the following problems:

[0003] (1) Poor scalability: When faced with a rapid increase in memory and CPUNodeManager resource requirements, the traditional approach usually requires the purchase of a large amount of new physical hardware to meet the demand. However, this approach will lead to a decrease in the scalability and responsiveness of the system. Purchasing and deploying new hardware takes time and may face the problem of uneven NodeManager resource utilization.

[0004] (2) Low NodeManager resource utilization: The NodeManager resource usage of many applications often fluctuates greatly. During peak hours, the demand for NodeManager resources may be high, but during off-peak hours or idle time, these NodeManager resources may not be fully utilized. This unstable NodeManager resource utilization leads to a waste of NodeManager resources and increases costs.

[0005] (3) High NodeManager resource cost: In order to cope with peak loads, sufficient physical NodeManager resources must be configured, including infrastructure such as hardware devices, power supply and cooling equipment. The investment cost of increasing these NodeManager resources may be high, especially when burst loads need to be met.

[0006] (4) NodeManager resource fragmentation: Due to dynamic adjustments to services and changes in tasks, a large number of NodeManager resource fragments that cannot be efficiently utilized may appear. These fragmented NodeManager resources cannot be used in combination, resulting in reduced NodeManager resource utilization. Fragmented NodeManager resources may be scattered across the cluster, resulting in a waste of NodeManager resources.

[0007] (5) Unable to meet sudden loads: During business peaks or in special scenarios, there may be sudden peaks in NodeManager resource demand. Without a flexible NodeManager resource scheduling and management mechanism, the cluster will not be able to meet these sudden load requirements in a timely manner, resulting in performance degradation or task delays. Summary of the invention

[0008] In order to solve the problems existing in the prior art, the present invention provides a method and device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster. By improving the NodeManager resource utilization of a single physical machine, controlling the upper limit of over-configured CPU and memory NodeManager resources, discovering and responding to saturated machines, and expelling running tasks, the NodeManager resource scheduling of a big data cluster can support automatic elastic scaling.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] In one embodiment of the present invention, a method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster is proposed, the method comprising:

[0011] S01. Host load assessment and resource management. On a single host, the host's CPU and memory usage indicators, and the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold are used to evaluate whether the host's load is excessive.

[0012] Furthermore, the calculation method of the usage rate of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold is as follows: before starting the NodeManager, the threshold of the available memory size and the number of CPU cores is configured. As computing resources, the maximum memory and the number of CPU cores used by the computing process will not exceed the configured threshold; Container is the basic process unit for the NodeManager to start the computing task, and is used for the computing processing of the task.

[0013] Further, the S01 includes:

[0014] S011. Obtain the CPU and memory usage of the current machine and the usage of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold;

[0015] S012. Determine whether the CPU and memory usage of the current machine are lower than a preset first threshold (Threshold 1). If lower than the first threshold, it means that the remaining NodeManager resources of the machine are large, and it is determined that the NodeManager resource configuration needs to be improved.

[0016] Furthermore, the first threshold in S012 is: a threshold of available memory size and number of CPU cores.

[0017] S013. If the CPU and memory usage of the current machine is higher than the preset first threshold (Threshold 1), the machine is saturated. Observe the ratio of NodeManager to the total usage of the machine.

[0018] S014. If the ratio of NodeManager to the total machine usage is higher than the preset second threshold (Threshold2), it means that in the machine where NodeManager resources are saturated, the NodeManager resource usage of the NodeManager task has exceeded expectations. It will be judged that the NodeManager resource configuration needs to be reduced to suppress the submission of subsequent tasks, and the specific NodeManager resource that exceeds the threshold is recorded as Memory or CPU, which plays a role in Container eviction;

[0019] Furthermore, the second threshold in S014 is: the threshold of NodeManager resource saturation, which is used to determine whether the resource usage of NodeManager exceeds the expected limit. When the resource usage of NodeManager reaches or exceeds this threshold, the system will consider that the NodeManager resources are saturated and action may be required. The second threshold Threshold2 is usually set manually.

[0020] S015. If the ratio of NodeManager to the total machine usage is lower than the preset second threshold (Threshold2), the NodeManager resource occupancy of the NodeManager task is still within the expected range, and it is determined that the current NodeManager resource configuration needs to be maintained, and subsequent changes in the machine NodeManager resources need to be continuously observed.

[0021] S02, NodeManager resource optimization strategy, in order to achieve efficient use of NodeManager resources, dynamic optimization of resources;

[0022] Further, the S02 includes:

[0023] S021. Obtain resource quantity: obtain the current resource quantity of each NodeManager in the Yarn cluster;

[0024] S022. Calculate the resource change amount. Calculate the change rate of NodeManager resources and multiply it by the current resource amount to predict the expected resource change amount. This change amount can be positive or negative, indicating an increase or decrease in resources. Add the change amount to the current resource amount to get the expected adjusted resource amount.

[0025] Furthermore, the change rate of the NodeManager resources in S022 is: is the ratio of the resource usage of the NodeManager in the Yarn cluster to change over time. This indicator measures the efficiency of resource usage and predicts the changing trend of resource demand.

[0026] Furthermore, the step of calculating the change rate of the NodeManager resources includes:

[0027] S0221. Collect data. Collect resource usage data of NodeManager at different time points. This data may include CPU usage, memory usage, disk usage, etc.

[0028] S0222. Determine the time range and select an appropriate time range. The time range depends on the time scale of the user's analysis;

[0029] S0223. Calculate resource usage changes: for each time point, calculate resource usage changes;

[0030] S0224. Calculate the change rate, the calculation formula is: change rate = (resource usage (current) - resource usage (previous)) / resource usage (previous)) × 100%;

[0031] S0225, average rate of change, to analyze the changes over a period of time, calculate the average of all the rates of change over that period of time to get a more stable indicator;

[0032] S0226. Consider resource usage patterns. When calculating the rate of change, factor in resource usage patterns and periodicity. For example, some applications may have peak resource usage at certain times.

[0033] S0227. Analyze the results and analyze the calculated change rate to determine whether the resource allocation of the NodeManager needs to be adjusted.

[0034] S023. Resource quantity inspection.

[0035] Further, the S023 includes:

[0036] S0231. Check the range of the change to ensure that the change does not exceed the preset upper and lower limits. The upper limit is based on the total resources of the physical machine and the set ratio (for example, the CPU upper limit is 1.2 times the total CPU resources of the physical machine), and the lower limit is based on the configuration of the NodeManager.

[0037] S0232, invalid fluctuation test, if the resource change amount continues to fluctuate within a small range, this may be regarded as invalid fluctuation, and the system will reject the resource adjustment request to avoid frequent resource changes;

[0038] S0233. Time interval check: when the interval between two resource changes exceeds the set threshold (IntervalThreshold), resource adjustment is allowed to reduce frequent resource fluctuations.

[0039] In this way, it can be ensured that the dynamic adjustment of NodeManager resources meets the current needs of the cluster without causing excessive fluctuations or frequent adjustments of resources.

[0040] S03: Automatic management of NodeManager resources. Based on the data obtained in S01, determine whether the resource allocation of NodeManager needs to be adjusted. If it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased. If the NodeManager resources are idle, the resources are recycled for use by other tasks.

[0041] Furthermore, the resource allocation in S03 includes:

[0042] S0311. Node selection: traverse all nodes in the cluster and select nodes suitable for allocating resources based on resource quantity and task priority. This can be achieved through simple loop traversal or more complex scheduling algorithms.

[0043] S0312, resource check: after selecting a node, check whether the available CPU and memory resources meet the task requirements. If the resources are insufficient, reselect the node or adjust the resource allocation;

[0044] S0313. Resource allocation update: After determining resource allocation, update the resource allocation information of the node and associate the task with the corresponding CPU and memory resources to facilitate resource management and monitoring.

[0045] Furthermore, the recycling of resources in S03 includes:

[0046] S0321. Task monitoring: Regularly check or monitor the running status of tasks through event-driven mechanisms to identify idle or low-priority tasks;

[0047] S0322, idle state judgment, check whether the task has been completed or is in a suspended state. Tasks in the above states no longer need to occupy NodeManager resources;

[0048] S0323, resource amount comparison, for idle tasks, compare the CPU and memory resources occupied by them with the preset recycling threshold, if the resource usage is lower than the threshold, recycle the resources;

[0049] S0324, resource release, determine the task of recycling resources, and release the CPU and memory resources occupied by them;

[0050] S0325. Update resource allocation information. After resource recovery is completed, update the resource allocation information of the node, mark the recovered resources as available, and reallocate them.

[0051] This automated resource management strategy can effectively improve the resource utilization of the Yarn cluster, ensure that resources are allocated on demand, and recycle idle resources in a timely manner, thereby optimizing the performance and efficiency of the entire cluster.

[0052] S04. Identification and processing of NodeManager resource saturation. By continuously monitoring the cluster status, NodeManager machines with resource saturation can be discovered in a timely manner. Resource saturation will affect the stable operation of the cluster, so it needs to be processed in a timely manner.

[0053] Further, the S04 includes:

[0054] S041, resource saturation identification, the proportion of resources occupied by NodeManager exceeds the second preset threshold (Threshold2), the resource usage of NodeManager tasks has exceeded expectations, and measures are needed to reduce resource configuration;

[0055] S042. Analyze the reasons for resource configuration adjustment, analyzing the reasons for the need to reduce resource configuration. If the reason is an increase in memory usage, further determine whether the current memory usage reaches a preset threshold of memory.

[0056] S043, Container eviction policy, when the memory usage exceeds the threshold, the Container eviction mechanism will be triggered;

[0057] Furthermore, the rules followed by the S043 expulsion process include:

[0058] S0431. Get the actual CPU or memory usage of all Containers and sort them from high to low.

[0059] S0432. Based on the sorting results, Containers with the highest usage are evicted one by one until resource usage drops below the safety threshold.

[0060] S0433. During the eviction process, containers in high priority or specific queues will be skipped to avoid affecting critical tasks.

[0061] S044. Differentiate the reasons for eviction. Containers are evicted because of node resource pressure rather than problems with the container itself. Yarn's NMWebService (NodeManager Web Service) adds a dedicated API and identifier.

[0062] S045, Task failure handling, Spark and Hive frameworks have been improved to identify container failures caused by eviction, skip the failure accumulation logic, and allow tasks to be retried. This avoids job failures caused by too many consecutive failures.

[0063] In one embodiment of the present invention, a device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster is also proposed, the device comprising:

[0064] Host load assessment and resource management module: On a single host, the host's CPU and memory usage indicators, and the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold are used to evaluate whether the host's load exceeds the limit.

[0065] NodeManager resource optimization strategy module, which dynamically optimizes resources to achieve efficient utilization of NodeManager resources;

[0066] The automatic management module of NodeManager resources determines whether the resource allocation of NodeManager needs to be adjusted based on the data obtained in S01; if it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased; if the NodeManager resources are idle, the resources are recycled for use by other tasks;

[0067] The NodeManager resource saturation identification and processing module continuously monitors the cluster status and promptly detects NodeManager machines with resource saturation. Resource saturation affects the stable operation of the cluster and therefore needs to be handled in a timely manner.

[0068] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned elastic scaling method for supporting NodeManager resource scheduling of a big data cluster is implemented.

[0069] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program for executing a method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster.

[0070] Beneficial effects:

[0071] The present invention provides a method and device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster, which can optimize NodeManager resource utilization, reduce costs, and improve the scalability and responsiveness of the cluster. Through intelligent over-allocation judgment, accurate NodeManager resource calculation, and automated NodeManager resource allocation, it is possible to dynamically optimize NodeManager resource allocation, improve system performance and efficiency, and respond to machine saturation in a timely manner to ensure stable execution of tasks. It avoids performance degradation and task delays caused by NodeManager resource shortages, and improves the reliability and stability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a flow chart of a method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to the present invention;

[0073] Figure 2 It is a schematic diagram of the calculation flow of S01 of the present invention;

[0074] Figure 3 It is a schematic diagram of the structure of the device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to the present invention;

[0075] Figure 4 It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0078] According to an embodiment of the present invention, a method and device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster are proposed. By improving the NodeManager resource utilization of a single physical machine, controlling the upper limit of over-configured CPU and memory NodeManager resources, discovering and responding to saturated machines, and expelling running tasks, the NodeManager resource scheduling of a big data cluster can support automatic elastic scaling.

[0079] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0080] In the context of Hadoop YARN (Yet Another Resource Negotiator), NodeManager (NM) and Container are two key concepts:

[0081] NodeManager(NM):

[0082] NodeManager is a daemon process in the Hadoop cluster that runs on each node in the cluster.

[0083] It is responsible for managing the node's resources, including CPU, memory, and disk.

[0084] NodeManager monitors the health of nodes and reports their resource usage to ResourceManager (RM), another daemon in the YARN cluster that is responsible for resource management and task scheduling for the entire cluster.

[0085] NodeManager is also responsible for executing tasks assigned by ResourceManager, which include starting and managing Containers.

[0086] Container:

[0087] Container is an abstract concept in YARN, representing a group of resources, such as CPU, memory, or disk.

[0088] Container is the execution environment for running tasks, which encapsulates all the resources required for the task.

[0089] In YARN, a task can be a MapReduce job, a Spark job, or other types of distributed computing tasks.

[0090] The NodeManager starts one or more Containers for each task, which run under the supervision of the NodeManager until the task is completed.

[0091] The size of the container can be dynamically allocated according to the needs of the task to ensure efficient use of resources.

[0092] In short, NodeManager is the component responsible for resource management and task execution in the Hadoop cluster, while Container is a collection of resources assigned to tasks by NodeManager to perform specific computing work. This design enables YARN to flexibly manage cluster resources and support various types of distributed computing tasks.

[0093] The present invention relates to a method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster, the method comprising:

[0094] S01. Host load assessment and resource management. On a single host, the host's CPU and memory usage indicators, and the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold are used to evaluate whether the host's load is excessive.

[0095] The calculation method of the usage rate of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold is as follows: before starting the NodeManager, the threshold of the available memory size and the number of CPU cores is configured. As computing resources, the maximum memory and the number of CPU cores used by the computing process will not exceed the configured threshold; Container is the basic process unit for the NodeManager to start the computing task, and is used for the computing processing of the task.

[0096] like Figure 2 As shown, the S01 includes:

[0097] S011. Obtain the CPU and memory usage of the current machine and the usage of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold;

[0098] S012. Determine whether the CPU and memory usage of the current machine are lower than a preset first threshold (Threshold 1). If lower than the first threshold, it means that the remaining NodeManager resources of the machine are large, and it is determined that the NodeManager resource configuration needs to be improved.

[0099] The first threshold in S012 is: the threshold of the available memory size and the number of CPU cores.

[0100] S013. If the CPU and memory usage of the current machine is higher than the preset first threshold (Threshold 1), the machine is saturated. Observe the ratio of NodeManager to the total usage of the machine.

[0101] S014. If the ratio of NodeManager to the total machine usage is higher than the preset second threshold (Threshold2), it means that in the machine where NodeManager resources are saturated, the NodeManager resource usage of the NodeManager task has exceeded expectations. It will be judged that the NodeManager resource configuration needs to be reduced to suppress the submission of subsequent tasks, and the specific NodeManager resource that exceeds the threshold is recorded as Memory or CPU, which plays a role in Container eviction;

[0102] The second threshold in S014 is: the threshold of NodeManager resource saturation, which is used to determine whether the resource usage of NodeManager exceeds the expected limit. When the resource usage of NodeManager reaches or exceeds this threshold, the system will consider that the NodeManager resources are saturated and may need to take action. The second threshold Threshold2 is usually set manually.

[0103] S015. If the ratio of NodeManager to the total machine usage is lower than the preset second threshold (Threshold2), the NodeManager resource occupancy of the NodeManager task is still within the expected range, and it is determined that the current NodeManager resource configuration needs to be maintained, and subsequent changes in the machine NodeManager resources need to be continuously observed.

[0104] S02, NodeManager resource optimization strategy, in order to achieve efficient use of NodeManager resources, dynamic optimization of resources;

[0105] The S02 includes:

[0106] S021. Obtain resource quantity: obtain the current resource quantity of each NodeManager in the Yarn cluster;

[0107] S022. Calculate the resource change amount. Calculate the change rate of NodeManager resources and multiply it by the current resource amount to predict the expected resource change amount. This change amount can be positive or negative, indicating an increase or decrease in resources. Add the change amount to the current resource amount to get the expected adjusted resource amount.

[0108] The change rate of the NodeManager resources in S022 is: the ratio of the resource usage of the NodeManager in the Yarn cluster to change over time. This indicator measures the efficiency of resource usage and predicts the change trend of resource demand.

[0109] The steps for calculating the change rate of the NodeManager resources include:

[0110] S0221. Collect data. Collect resource usage data of NodeManager at different time points. This data may include CPU usage, memory usage, disk usage, etc.

[0111] S0222. Determine the time range and select an appropriate time range. The time range depends on the time scale of the user's analysis;

[0112] S0223. Calculate resource usage changes: for each time point, calculate resource usage changes;

[0113] S0224. Calculate the change rate, the calculation formula is: change rate = (resource usage (current) - resource usage (previous)) / resource usage (previous)) × 100%;

[0114] S0225, average rate of change, to analyze the changes over a period of time, calculate the average of all the rates of change over that period of time to get a more stable indicator;

[0115] S0226. Consider resource usage patterns and factor in resource usage patterns and periodicity when calculating the rate of change;

[0116] S0227. Analyze the results and analyze the calculated change rate to determine whether the resource allocation of the NodeManager needs to be adjusted.

[0117] S023. Resource quantity inspection.

[0118] The S023 includes:

[0119] S0231. Check the range of the change to ensure that the change does not exceed the preset upper and lower limits. The upper limit is based on the total resources of the physical machine and the set ratio, and the lower limit is based on the configuration of the NodeManager.

[0120] S0232, invalid fluctuation test, if the resource change amount continues to fluctuate within a small range, this may be regarded as invalid fluctuation, and the system will reject the resource adjustment request to avoid frequent resource changes;

[0121] S0233. Time interval check: when the interval between two resource changes exceeds the set threshold (IntervalThreshold), resource adjustment is allowed to reduce frequent resource fluctuations.

[0122] In this way, it can be ensured that the dynamic adjustment of NodeManager resources meets the current needs of the cluster without causing excessive fluctuations or frequent adjustments of resources.

[0123] S03: Automatic management of NodeManager resources. Based on the data obtained in S01, determine whether the resource allocation of NodeManager needs to be adjusted. If it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased. If the NodeManager resources are idle, the resources are recycled for use by other tasks.

[0124] The resource allocation in S03 includes:

[0125] S0311. Node selection: traverse all nodes in the cluster and select nodes suitable for allocating resources based on resource quantity and task priority. This can be achieved through simple loop traversal or more complex scheduling algorithms.

[0126] S0312, resource check: after selecting a node, check whether the available CPU and memory resources meet the task requirements. If the resources are insufficient, reselect the node or adjust the resource allocation;

[0127] S0313. Resource allocation update: After determining resource allocation, update the resource allocation information of the node and associate the task with the corresponding CPU and memory resources to facilitate resource management and monitoring.

[0128] The recycling of resources in S03 includes:

[0129] S0321. Task monitoring: Regularly check or monitor the running status of tasks through event-driven mechanisms to identify idle or low-priority tasks;

[0130] S0322, idle state judgment, check whether the task has been completed or is in a suspended state. Tasks in the above states no longer need to occupy NodeManager resources;

[0131] S0323, resource amount comparison, for idle tasks, compare the CPU and memory resources occupied by them with the preset recycling threshold, if the resource usage is lower than the threshold, recycle the resources;

[0132] S0324, resource release, determine the task of recycling resources, and release the CPU and memory resources occupied by them;

[0133] S0325. Update resource allocation information. After resource recovery is completed, update the resource allocation information of the node, mark the recovered resources as available, and reallocate them.

[0134] This automated resource management strategy can effectively improve the resource utilization of the Yarn cluster, ensure that resources are allocated on demand, and recycle idle resources in a timely manner, thereby optimizing the performance and efficiency of the entire cluster.

[0135] S04. Identification and processing of NodeManager resource saturation. By continuously monitoring the cluster status, NodeManager machines with resource saturation can be discovered in a timely manner. Resource saturation will affect the stable operation of the cluster, so it needs to be processed in a timely manner.

[0136] The S04 includes:

[0137] S041, resource saturation identification, the proportion of resources occupied by NodeManager exceeds the second preset threshold (Threshold2), the resource usage of NodeManager tasks has exceeded expectations, and measures are needed to reduce resource configuration;

[0138] S042. Analyze the reasons for resource configuration adjustment, analyzing the reasons for the need to reduce resource configuration. If the reason is an increase in memory usage, further determine whether the current memory usage reaches a preset threshold of memory.

[0139] S043, Container eviction policy, when the memory usage exceeds the threshold, the Container eviction mechanism will be triggered;

[0140] The rules followed in the S043 expulsion process include:

[0141] S0431. Get the actual CPU or memory usage of all Containers and sort them from high to low.

[0142] S0432. Based on the sorting results, Containers with the highest usage are evicted one by one until resource usage drops below the safety threshold.

[0143] S0433. During the eviction process, containers in high priority or specific queues will be skipped to avoid affecting critical tasks.

[0144] S044. Differentiate the reasons for eviction. Containers are evicted because of node resource pressure rather than problems with the container itself. Yarn's NMWebService (NodeManager Web Service) adds a dedicated API and identifier.

[0145] S045, Task failure handling, Spark and Hive frameworks have been improved to identify container failures caused by eviction, skip the failure accumulation logic, and allow tasks to be retried. This avoids job failures caused by too many consecutive failures.

[0146] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0147] In order to explain the above-mentioned method of elastically scaling to support NodeManager resource scheduling of a big data cluster more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0148] like Figure 1 As shown, the present invention relates to a method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster, the method comprising:

[0149] S01. Host load assessment and resource management. On a single host, the host's CPU and memory usage indicators, and the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold are used to evaluate whether the host's load is excessive.

[0150] The calculation method of the usage rate of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold is as follows: before starting the NodeManager, the threshold of the available memory size and the number of CPU cores is configured. As computing resources, the maximum memory and the number of CPU cores used by the computing process will not exceed the configured threshold; Container is the basic process unit for the NodeManager to start the computing task, and is used for the computing processing of the task.

[0151] For example, the computing resource thresholds of NodeManager are: 20G memory and 10 CPU cores. The memory occupied by Container is 10G and the number of CPUs is 6. The ratio of memory to CPU is: 10G / 20G = 0.5 and 6 / 10 = 0.6.

[0152] like Figure 2 As shown, the S01 includes:

[0153] S011. Obtain the CPU and memory usage of the current machine and the usage of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold;

[0154] S012. Determine whether the CPU and memory usage of the current machine are lower than a preset first threshold (Threshold 1). If lower than the first threshold, it means that the remaining NodeManager resources of the machine are large, and it is determined that the NodeManager resource configuration needs to be improved.

[0155] The first threshold in S012 is: the threshold of the available memory size and the number of CPU cores.

[0156] S013. If the CPU and memory usage of the current machine is higher than the preset first threshold (Threshold 1), the machine is saturated. Observe the ratio of NodeManager to the total usage of the machine.

[0157] S014. If the ratio of NodeManager to the total machine usage is higher than the preset second threshold (Threshold2), it means that in the machine where NodeManager resources are saturated, the NodeManager resource usage of the NodeManager task has exceeded expectations. It will be judged that the NodeManager resource configuration needs to be reduced to suppress the submission of subsequent tasks, and the specific NodeManager resource that exceeds the threshold is recorded as Memory or CPU, which plays a role in Container eviction;

[0158] The second threshold in S014 is: the threshold of NodeManager resource saturation, which is used to determine whether the resource usage of NodeManager exceeds the expected limit. When the resource usage of NodeManager reaches or exceeds this threshold, the system will consider that the NodeManager resources are saturated and may need to take action. The second threshold Threshold2 is usually set manually.

[0159] S015. If the ratio of NodeManager to the total machine usage is lower than the preset second threshold (Threshold2), the NodeManager resource occupancy of the NodeManager task is still within the expected range, and it is determined that the current NodeManager resource configuration needs to be maintained, and subsequent changes in the machine NodeManager resources need to be continuously observed.

[0160] S02, NodeManager resource optimization strategy, in order to achieve efficient use of NodeManager resources, dynamic optimization of resources;

[0161] The S02 includes:

[0162] S021. Obtain resource quantity: obtain the current resource quantity of each NodeManager in the Yarn cluster;

[0163] S022. Calculate the resource change amount. Calculate the change rate of NodeManager resources and multiply it by the current resource amount to predict the expected resource change amount. This change amount can be positive or negative, indicating an increase or decrease in resources. Add the change amount to the current resource amount to get the expected adjusted resource amount.

[0164] The change rate of the NodeManager resources in S022 is: the ratio of the resource usage of the NodeManager in the Yarn cluster to change over time. This indicator measures the efficiency of resource usage and predicts the change trend of resource demand.

[0165] The steps for calculating the change rate of the NodeManager resources include:

[0166] S0221. Collect data. Collect resource usage data of NodeManager at different time points. This data may include CPU usage, memory usage, disk usage, etc.

[0167] S0222. Determine the time range and select an appropriate time range. The time range depends on the time scale of the user's analysis;

[0168] S0223. Calculate the change in resource usage: For each time point, calculate the change in resource usage. For example, if you are interested in CPU usage, you need to calculate the CPU usage at each time point and find the difference between them.

[0169] S0224. Calculate the change rate, the calculation formula is: change rate = (resource usage (current) - resource usage (previous)) / resource usage (previous)) × 100%;

[0170] S0225, average rate of change, to analyze the changes over a period of time, calculate the average of all the rates of change over that period of time to get a more stable indicator;

[0171] S0226. Consider resource usage patterns. When calculating the rate of change, factor in resource usage patterns and periodicity. For example, some applications may have peak resource usage at certain times.

[0172] S0227. Analyze the results and analyze the calculated change rate to determine whether the resource allocation of the NodeManager needs to be adjusted.

[0173] For example, if the NodeManager used 50% of the CPU resources an hour ago and currently uses 60% of the CPU resources, the CPU resource change rate can be calculated as follows: CPU change rate = ((60% - 50%) / 50%) × 100% = 20%

[0174] This means that the CPU resource usage of NodeManager increased by 20% during this hour. Based on this rate of change, the system administrator can decide whether to increase or decrease the resource allocation of NodeManager to optimize resource usage efficiency.

[0175] S023. Resource quantity inspection.

[0176] The S023 includes:

[0177] S0231. Check the range of the change to ensure that the change does not exceed the preset upper and lower limits. The upper limit is based on the total resources of the physical machine and the set ratio (for example, the CPU upper limit is 1.2 times the total CPU resources of the physical machine), and the lower limit is based on the configuration of the NodeManager.

[0178] S0232, invalid fluctuation test, if the resource change amount continues to fluctuate within a small range, this may be regarded as invalid fluctuation, and the system will reject the resource adjustment request to avoid frequent resource changes;

[0179] S0233. Time interval check: when the interval between two resource changes exceeds the set threshold (IntervalThreshold), resource adjustment is allowed to reduce frequent resource fluctuations.

[0180] For example:

[0181] Assume that we set the upper limit ratio of CPU resources to 1.2, the upper limit ratio of memory resources to 0.8, and the CPU and memory resources configured for NodeManager to be 10 and 20 respectively. If the available CPU resources are 100 and the memory resources are 80, we can calculate the upper and lower limits of the resources as follows:

[0182] CPU UpperLimit = CPU ratio * available CPU resources = 100 * 1.2 = 120

[0183] CPU LowerLimit = CPU resources configured by NodeManager = 10

[0184] Memory Upper Limit = memory ratio * available memory resources = 80 * 0.8 = 64

[0185] Memory LowerLimit = NodeManager configured memory resource = 20

[0186] Therefore, the change in CPU resources should be between 10 and 120, and the change in memory resources should be between 20 and 64.

[0187] In this way, it can be ensured that the dynamic adjustment of NodeManager resources meets the current needs of the cluster without causing excessive fluctuations or frequent adjustments of resources.

[0188] S03: Automatic management of NodeManager resources. Based on the data obtained in S01, determine whether the resource allocation of NodeManager needs to be adjusted. If it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased. If the NodeManager resources are idle, the resources are recycled for use by other tasks.

[0189] The resource allocation in S03 includes:

[0190] S0311. Node selection: traverse all nodes in the cluster and select nodes suitable for allocating resources based on resource quantity and task priority. This can be achieved through simple loop traversal or more complex scheduling algorithms.

[0191] S0312, resource check: after selecting a node, check whether the available CPU and memory resources meet the task requirements. If the resources are insufficient, reselect the node or adjust the resource allocation;

[0192] S0313. Resource allocation update: After determining resource allocation, update the resource allocation information of the node and associate the task with the corresponding CPU and memory resources to facilitate resource management and monitoring.

[0193] The recycling of resources in S03 includes:

[0194] S0321. Task monitoring: Regularly check or monitor the running status of tasks through event-driven mechanisms to identify idle or low-priority tasks;

[0195] S0322, idle state judgment, check whether the task has been completed or is in a suspended state. Tasks in the above states no longer need to occupy NodeManager resources;

[0196] S0323, resource amount comparison, for idle tasks, compare the CPU and memory resources occupied by them with the preset recycling threshold, if the resource usage is lower than the threshold, recycle the resources;

[0197] S0324, resource release, determine the task of recycling resources, and release the CPU and memory resources occupied by them;

[0198] S0325. Update resource allocation information. After resource recovery is completed, update the resource allocation information of the node, mark the recovered resources as available, and reallocate them.

[0199] This automated resource management strategy can effectively improve the resource utilization of the Yarn cluster, ensure that resources are allocated on demand, and recycle idle resources in a timely manner, thereby optimizing the performance and efficiency of the entire cluster.

[0200] S04. Identification and processing of NodeManager resource saturation. By continuously monitoring the cluster status, NodeManager machines with resource saturation can be discovered in a timely manner. Resource saturation will affect the stable operation of the cluster, so it needs to be processed in a timely manner.

[0201] The S04 includes:

[0202] S041, resource saturation identification, the proportion of resources occupied by NodeManager exceeds the second preset threshold (Threshold2), the resource usage of NodeManager tasks has exceeded expectations, and measures are needed to reduce resource configuration;

[0203] S042. Analyze the reasons for resource configuration adjustment, analyzing the reasons for the need to reduce resource configuration. If the reason is an increase in memory usage, further determine whether the current memory usage reaches a preset threshold of memory.

[0204] S043, Container eviction policy, when the memory usage exceeds the threshold, the Container eviction mechanism will be triggered;

[0205] The rules followed in the S043 expulsion process include:

[0206] S0431. Get the actual CPU or memory usage of all Containers and sort them from high to low.

[0207] S0432. Based on the sorting results, Containers with the highest usage are evicted one by one until resource usage drops below the safety threshold.

[0208] S0433. During the eviction process, containers in high priority or specific queues will be skipped to avoid affecting critical tasks.

[0209] S044. Differentiate the reasons for eviction. Containers are evicted because of node resource pressure rather than problems with the container itself. Yarn's NMWebService (NodeManager Web Service) adds a dedicated API and identifier.

[0210] S045, Task failure handling, Spark and Hive frameworks have been improved to identify container failures caused by eviction, skip the failure accumulation logic, and allow tasks to be retried. This avoids job failures caused by too many consecutive failures.

[0211] Based on the same inventive concept, the present invention also proposes a device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0212] Figure 3 This is a schematic diagram of the structure of the device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to the present invention. Figure 3 As shown, the device comprises:

[0213] The host load evaluation and resource management module 110 evaluates whether the host load is excessive by using the host's CPU and memory usage indicators and the sum of the NodeManager-started Container memory and CPU usage to the NodeManager computing resource threshold on a single host.

[0214] The NodeManager resource optimization strategy module 120 performs dynamic optimization of resources in order to achieve efficient utilization of NodeManager resources;

[0215] The automatic management module 130 of NodeManager resources determines whether the resource allocation of NodeManager needs to be adjusted according to the data obtained in S01; if it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased; if the NodeManager resources are idle, the resources are recycled for use by other tasks;

[0216] The NodeManager resource saturation identification and processing module 140 continuously monitors the cluster status and promptly finds the NodeManager machines with resource saturation. Resource saturation will affect the stable operation of the cluster, so it needs to be processed in time.

[0217] It should be noted that although several modules of the apparatus for elastically scaling and supporting NodeManager resource scheduling of a big data cluster are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0218] Based on the above invention concept, Figure 4 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned elastic scaling method for supporting NodeManager resource scheduling of a big data cluster is implemented.

[0219] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster.

[0220] The present invention provides a method and device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster, which can optimize NodeManager resource utilization, reduce costs, and improve the scalability and responsiveness of the cluster. Through intelligent over-allocation judgment, accurate NodeManager resource calculation, and automated NodeManager resource allocation, it is possible to dynamically optimize NodeManager resource allocation, improve system performance and efficiency, and respond to machine saturation in a timely manner to ensure stable execution of tasks. It avoids performance degradation and task delays caused by NodeManager resource shortages, and improves the reliability and stability of the overall system.

[0221] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0222] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0223] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0224] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0225] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0226] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0227] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0228] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0229] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0230] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster, characterized in that: The method includes: S01. Host load assessment and resource management. On a single host, the host's CPU and memory usage indicators, and the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold are used to evaluate whether the host's load is excessive. S02, NodeManager resource optimization strategy, in order to achieve efficient use of NodeManager resources, dynamic optimization of resources; S03: Automatic management of NodeManager resources. Based on the data obtained in S01, determine whether the resource allocation of NodeManager needs to be adjusted. If it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased. If the NodeManager resources are idle, the resources are recycled for use by other tasks. S04. Identification and processing of NodeManager resource saturation. By continuously monitoring the cluster status, NodeManager machines with resource saturation can be discovered in a timely manner. Resource saturation will affect the stable operation of the cluster, so it needs to be processed in a timely manner.

2. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 1, characterized in that: The calculation method of the usage rate of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold is as follows: before starting the NodeManager, the threshold of the available memory size and the number of CPU cores is configured. As computing resources, the maximum memory and the number of CPU cores used by the computing process will not exceed the configured threshold; Container is the basic process unit for the NodeManager to start the computing task, and is used for the computing processing of the task.

3. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 1, characterized in that: The S01 includes: S011. Obtain the CPU and memory usage of the current machine and the usage of the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold; S012. Determine whether the CPU and memory usage of the current machine are lower than a preset first threshold. If lower than the first threshold, it indicates that the remaining NodeManager resources of the machine are large, and it is determined that the NodeManager resource configuration needs to be increased. S013. If the CPU and memory usage of the current machine is higher than the preset first threshold, the machine is saturated, and the ratio of NodeManager to the total usage of the machine is observed; S014. If the ratio of NodeManager to the total machine usage is higher than the preset second threshold, it means that in the machine where NodeManager resources are saturated, the NodeManager resource usage of the NodeManager task has exceeded expectations. It will be judged that the NodeManager resource configuration needs to be reduced to suppress the submission of subsequent tasks, and the specific NodeManager resource that exceeds the threshold is recorded as Memory or CPU, which plays a role in Container eviction; S015. If the ratio of NodeManager to the total machine usage is lower than the preset second threshold, the NodeManager resource occupancy of the NodeManager task is still within the expected range, and it is determined that the current NodeManager resource configuration needs to be maintained, and subsequent changes in the machine NodeManager resources need to be continuously observed.

4. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 3 is characterized in that: The first threshold in S012 is: the threshold of the available memory size and the number of CPU cores.

5. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 3 is characterized in that: The second threshold in S014 is: the NodeManager resource saturation threshold, which is used to determine whether the NodeManager resource usage exceeds the expected limit. When the NodeManager resource usage reaches or exceeds this threshold, the system will consider that the NodeManager resources are saturated and may need to take action. The second threshold is set manually.

6. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 1, characterized in that: The S02 includes: S021. Obtain resource quantity: obtain the current resource quantity of each NodeManager in the Yarn cluster; S022. Calculate the resource change amount: calculate the change rate of the NodeManager resources and multiply it by the current resource amount to predict the expected resource change amount this time. Add the change amount to the current resource amount to obtain the expected adjusted resource amount. S023. Resource quantity inspection.

7. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 6 is characterized in that: The change rate of the NodeManager resources in S022 is: the ratio of the resource usage of the NodeManager in the Yarn cluster to change over time. This indicator measures the efficiency of resource usage and predicts the change trend of resource demand.

8. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 6 is characterized in that: The steps for calculating the change rate of the NodeManager resources include: S0221. Collect data: collect resource usage data of NodeManager at different time points; S0222. Determine the time range and select an appropriate time range. The time range depends on the time scale of the user's analysis; S0223. Calculate resource usage changes: for each time point, calculate resource usage changes; S0224. Calculate the change rate, the calculation formula is: change rate = (resource usage (current) - resource usage (previous)) / resource usage (previous)) × 100%; S0225, average rate of change, to analyze the changes over a period of time, calculate the average of all the rates of change over that period of time to get a more stable indicator; S0226. Consider resource usage patterns and factor in resource usage patterns and periodicity when calculating the rate of change; S0227. Analyze the results and analyze the calculated change rate to determine whether the resource allocation of the NodeManager needs to be adjusted.

9. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 6, characterized in that: The S023 includes: S0231. Check the range of the change to ensure that the change does not exceed the preset upper and lower limits. The upper limit is based on the total resources of the physical machine and the set ratio, and the lower limit is based on the configuration of the NodeManager. S0232, invalid fluctuation test, if the resource change amount continues to fluctuate within a small range, this may be regarded as invalid fluctuation, and the system will reject the resource adjustment request to avoid frequent resource changes; S0233. Time interval check: when the interval between two resource changes exceeds the set threshold, resource adjustment is allowed to reduce frequent resource fluctuations.

10. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 1, characterized in that: The resource allocation in S03 includes: S0311, node selection, traverse all nodes in the cluster and select nodes suitable for resource allocation based on resource quantity and task priority; S0312, resource check: after selecting a node, check whether the available CPU and memory resources meet the task requirements. If the resources are insufficient, reselect the node or adjust the resource allocation; S0313. Resource allocation update: After determining resource allocation, update the resource allocation information of the node and associate the task with the corresponding CPU and memory resources to facilitate resource management and monitoring.

11. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 1, characterized in that: The recycling of resources in S03 includes: S0321. Task monitoring: Regularly check or monitor the running status of tasks through event-driven mechanisms to identify idle or low-priority tasks; S0322, idle state judgment, check whether the task has been completed or is in a suspended state. Tasks in the above states no longer need to occupy NodeManager resources; S0323, resource amount comparison, for idle tasks, compare the CPU and memory resources occupied by them with the preset recycling threshold, if the resource usage is lower than the threshold, recycle the resources; S0324, resource release, determine the task of recycling resources, and release the CPU and memory resources occupied by them; S0325. Update resource allocation information. After resource recovery is completed, update the resource allocation information of the node, mark the recovered resources as available, and reallocate them.

12. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 1, characterized in that: The S04 includes: S041, resource saturation identification: the proportion of resources occupied by NodeManager exceeds the preset threshold, and the resource usage of NodeManager tasks has exceeded expectations. Measures are needed to reduce resource configuration; S042. Analyze the reasons for resource configuration adjustment, analyzing the reasons for the need to reduce resource configuration. If the reason is an increase in memory usage, further determine whether the current memory usage reaches a preset threshold of memory. S043, Container eviction policy, when the memory usage exceeds the threshold, the Container eviction mechanism will be triggered; 13. The method for elastically scaling and supporting NodeManager resource scheduling of a big data cluster according to claim 12, characterized in that: The rules followed in the S043 expulsion process include: S0431. Get the actual CPU or memory usage of all Containers and sort them from high to low. S0432. Based on the sorting results, Containers with the highest usage are evicted one by one until resource usage drops below the safety threshold. S0433. During the eviction process, containers in high priority or specific queues will be skipped; S044. Differentiate the reasons for eviction. Containers are evicted because of node resource pressure rather than problems with the container itself. Yarn's NMWebService adds a dedicated API and identifier. S045. Task failure processing: The Spark and Hive frameworks are improved to identify container failures caused by eviction, skip the failure count accumulation logic, and allow task retries.

14. A device for elastically scaling and supporting NodeManager resource scheduling of a big data cluster, characterized in that: The device includes: Host load assessment and resource management module: On a single host, the host's CPU and memory usage indicators, and the sum of the Container memory and CPU started by the NodeManager to the NodeManager computing resource threshold are used to evaluate whether the host's load exceeds the limit. NodeManager resource optimization strategy module, which dynamically optimizes resources to achieve efficient utilization of NodeManager resources; The automatic management module of NodeManager resources determines whether the resource allocation of NodeManager needs to be adjusted based on the data obtained in S01; if it is detected that the node resources are over-allocated, that is, the resource demand exceeds the current allocation, the resource allocation should be increased; if the NodeManager resources are idle, the resources are recycled for use by other tasks; The NodeManager resource saturation identification and processing module continuously monitors the cluster status and promptly detects NodeManager machines with resource saturation. Resource saturation affects the stable operation of the cluster and therefore needs to be handled in a timely manner.

15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 13.

Images