Method and apparatus for controlling distributed cluster load

Abstract

The application provides a distributed cluster load control method and device, which is applied to a distributed cluster system. The distributed cluster system comprises a system resource manager and a plurality of computer nodes connected with the system resource manager. The distributed cluster load control method comprises the following steps: obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system, wherein the target node is one of the plurality of computer nodes; adjusting the kernel default parameters of the target node to preset kernel parameters, so as to reduce the operating system memory occupation of the target node; and performing load control on each computer node in the distributed cluster system after the adjustment. The application can effectively improve the resource utilization of the distributed cluster system.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of distributed cluster, and particularly relates to a distributed cluster load control method and device. BACKGROUND

[0002] With the development of big data technology, the number of jobs and various jobs on the big data computing cluster are increasing, and the big data computing cluster often faces the situation that the user resources are insufficient and the job running cannot be submitted, and the actual resource utilization rate of each computer node is not high. And the big data distributed system has special macro parameters, such as cluster dimension queue and scheduling strategy, and the existing technology directly realizes the macro function by adjusting the macro parameter configuration, such as modifying the queue and the scheduling strategy. Generally, the kernel default parameters of the computer node will not be modified. Because the adjustment of the micro operating system kernel default parameters is difficult and the number is relatively large, the operation of one parameter may involve thousands of machines, so the technical personnel in the field generally do not consider micro adjustment, and the macro cluster queue, scheduling strategy and other parameters are easy to adjust and the number is relatively small. Therefore, in the existing distributed system, in order to maintain the stability of the whole cluster system, the kernel default parameters of the computer node are not modified, and the system self-contained kernel default parameters are used. However, the macro adjustment effect and the micro level default parameters are not good, which leads to low resource utilization rate of the distributed cluster system.

[0003] That is, the resource utilization rate of the distributed cluster system in the prior art is low. SUMMARY

[0004] The present application provides a distributed cluster load control method and device, which aims to solve the problem of low resource utilization rate of the distributed cluster system in the prior art.

[0005] In a first aspect, the present application provides a distributed cluster load control method applied to a distributed cluster system, wherein the distributed cluster system comprises a system resource manager and a plurality of computer nodes connected with the system resource manager, and the distributed cluster load control method comprises the following steps:

[0006] obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system, wherein the target node is one of the plurality of computer nodes, and the kernel default parameters comprise a memory recycling trend control parameter, a memory recycling dirty data threshold and a dirty data expiration time;

[0007] adjusting the kernel default parameters of the target node to preset kernel parameters to reduce the operating system occupied memory of the target node, wherein the kernel default parameters comprise a memory recycling trend control parameter, a memory recycling dirty data threshold and a dirty data expiration time.

[0008] controlling load of each computer node in the distributed cluster system after the adjustment of the parameters.

[0009] Optionally, the controlling load of each computer node in the distributed cluster system after the adjustment of the parameters comprises:

[0010] obtaining task load of each computer node in the distributed cluster system after the adjustment of the parameters;

[0011] determining whether a first computer node exists in each computer node, wherein the task load of the first computer node satisfies a high load condition;

[0012] if the first computer node exists in each computer node, obtaining a second computer node whose task load satisfies a low load condition;

[0013] migrating a task running on the first computer node to the second computer node.

[0014] Optionally, the high load condition comprises that CPU usage is higher than a first preset CPU usage and memory usage is higher than a first preset memory usage; and the low load condition comprises that CPU usage is lower than a second preset CPU usage and memory usage is lower than a second preset memory usage.

[0015] Optionally, the memory recycling trend control parameter represents a kernel's tendency to recycle memory used for file directory and inode cache,

[0016] the adjusting the kernel default parameter of the target node to a preset kernel parameter to reduce operating system memory occupancy of the target node comprises:

[0017] increasing a default value of the memory recycling trend control parameter to a first preset value.

[0018] Optionally, the adjusting the kernel default parameter of the target node to a preset kernel parameter to reduce operating system memory occupancy of the target node comprises:

[0019] decreasing a default value of a dirty data threshold of memory recycling of the target node to a second preset value, wherein a process running on the target node writes dirty data back to a disk on the target node when a proportion between dirty data occupancy space and system memory of the target node reaches the second preset value.

[0020] Optionally, the adjusting the kernel default parameter of the target node to a preset kernel parameter to reduce operating system memory occupancy of the target node comprises:

[0021] The default value of the dirty data expiration time is reduced to a third preset value, wherein the target node is confirmed as dirty data and written back to the disk on the target node when the data residence time exceeds the third preset value.

[0022] Optionally, the target node comprises a node resource manager, which is configured to communicate with the system resource manager and manage the computer node on which the node resource manager is located. The load control on each computer node in the distributed cluster system after the parameter adjustment comprises:

[0023] The node resource scheduling parameter of the node resource manager is adjusted to improve the configuration capacity of the target node.

[0024] Optionally, the node resource scheduling parameter comprises a schedulable virtual CPU quantity. The adjustment of the node resource scheduling parameter of the node resource manager to improve the configuration capacity of the target node comprises:

[0025] A default value of the schedulable virtual CPU quantity of the target node is obtained.

[0026] The default value of the schedulable virtual CPU quantity is increased to a fourth preset value.

[0027] Optionally, the node resource scheduling parameter comprises a schedulable memory capacity. The adjustment of the node resource scheduling parameter of the node resource manager to improve the configuration capacity of the target node comprises:

[0028] A default value of the schedulable memory capacity of the target node is obtained.

[0029] The default value of the schedulable memory capacity is increased to a fifth preset value.

[0030] In a second aspect, the present application provides a distributed cluster load control device, which is applied to a distributed cluster system. The distributed cluster system comprises a system resource manager and a plurality of computer nodes connected to the system resource manager. The distributed cluster load control device comprises:

[0031] An obtaining unit is configured to obtain kernel default parameters of a target node in the distributed cluster system when the distributed cluster system is started. The target node is one of the plurality of computer nodes. The kernel default parameters comprise a memory recovery trend control parameter, a memory recovery dirty data threshold, and a dirty data expiration time.

[0032] A parameter adjustment unit is configured to adjust the kernel default parameters of the target node to preset kernel parameters to reduce the operating system occupied memory of the target node.

[0033] A load control unit is configured to control the load of each computer node in the distributed cluster system after the parameter adjustment.

[0034] Optionally, the load control unit is configured to:

[0035] acquire the task load of each computer node in the distributed cluster system after the parameter adjustment.

[0036] determine whether a first computer node exists in each computer node, wherein the task load of the first computer node satisfies a high-load condition.

[0037] if the first computer node exists in each computer node, acquire a second computer node whose task load satisfies a low-load condition.

[0038] migrate a task running on the first computer node to the second computer node.

[0039] Optionally, the high-load condition includes that the CPU usage is higher than a first preset CPU usage and the memory usage is higher than a first preset memory usage; and the low-load condition includes that the CPU usage is lower than a second preset CPU usage and the memory usage is lower than a second preset memory usage.

[0040] Optionally, the memory reclamation tendency control parameter indicates the kernel's tendency to reclaim the memory used for file directory and inode cache,

[0041] the parameter adjustment unit is configured to:

[0042] increase the default value of the memory reclamation tendency control parameter to a first preset value.

[0043] Optionally, the parameter adjustment unit is configured to:

[0044] decrease the default value of the memory reclamation dirty data threshold of the target node to a second preset value, wherein a process running on the target node writes dirty data back to the disk of the target node when the proportion between the dirty data occupied space and the system memory of the target node reaches the second preset value.

[0045] Optionally, the parameter adjustment unit is configured to:

[0046] decrease the default value of the dirty data expiration time to a third preset value, wherein the target node is confirmed as dirty data and writes back to the disk of the target node when the residence time of the data exceeds the third preset value.

[0047] Optionally, the target node comprises a node resource manager, the node resource manager is configured to communicate with the system resource manager and manage the computer node where the node resource manager is located, and the parameter adjustment unit is configured to:

[0048] adjust a node resource scheduling parameter of the node resource manager to improve the configuration capacity of the target node.

[0049] Optionally, the node resource scheduling parameter comprises a number of schedulable virtual CPUs, and the parameter adjustment unit is configured to:

[0050] obtain a default value of the number of schedulable virtual CPUs of the target node;

[0051] increase the default value of the number of schedulable virtual CPUs to a fourth preset value.

[0052] Optionally, the node resource scheduling parameter comprises a schedulable memory capacity, and the parameter adjustment unit is configured to:

[0053] obtain a default value of the schedulable memory capacity of the target node;

[0054] increase the default value of the schedulable memory capacity to a fifth preset value.

[0055] In a third aspect, the present application provides a computer device, the computer device comprising:

[0056] one or more processors;

[0057] a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the control method of the distributed cluster load according to any one of the first aspect.

[0058] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium stores a plurality of instructions, the instructions are adapted to be loaded by a processor to execute the steps in the control method of the distributed cluster load according to any one of the first aspect.

[0059] The application provides a distributed cluster load control method and device, which is applied to a distributed cluster system, the distributed cluster system comprising a system resource manager and a plurality of computer nodes connected with the system resource manager, the distributed cluster load control method comprising: obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system, wherein the target node is one of the plurality of computer nodes; adjusting the kernel default parameters of the target node to preset kernel parameters to reduce operating system memory occupation of the target node; and performing load control on each computer node in the distributed cluster system after the adjustment of the parameters. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0061] Figure 1 is a scene schematic diagram of the distributed cluster system provided by the embodiments of the present application;

[0062] Figure 2 is an embodiment flow diagram of a distributed cluster load control method provided in an embodiment of the present application;

[0063] Figure 3 is another embodiment flow diagram of a distributed cluster load control method provided in an embodiment of the present application;

[0064] Figure 4 is an embodiment structure diagram of a distributed cluster load control device provided in an embodiment of the present application;

[0065] Figure 5 is an embodiment structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0067] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0068] In this application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the application. Details are set forth in the following description for purpose of explanation. It should be appreciated that one of ordinary skill in the art would realize that the application can be practiced without the use of these specific details. In other instances, well-known structures and processes are not elaborated as details of such structures and processes can interfere with an understanding of this description. The present application is not limited to the embodiments shown, but is intended to include all alternatives consistent with the principles of the present application and the appended claims.

[0069] The embodiments of the present application provide a distributed cluster load control method and device, which are described in detail as follows.

[0070] Please refer to Figure 1 , Figure 1 The following is a scene schematic diagram of the distributed cluster system provided by the embodiments of the present application. The distributed cluster system 10 can include a computer device 111, and the distributed cluster load control device is integrated in the computer device 111.

[0071] In the embodiments of the present application, the distributed cluster system 10 includes a system resource manager ResourceManager and a plurality of computer nodes 14 connected with the system resource manager ResourceManager. The computer nodes 14 include a node resource manager NodeManager and a plurality of containers. The system resource manager ResourceManager is connected with a plurality of clients 12 and the computer device 111. The computer device 111 and the clients 12 can access each computer node 14 through the system resource manager ResourceManager.

[0072] Specifically, the system resource manager (ResourceManager) is responsible for the resource management and allocation of the whole cluster, and is a global resource management system. The node resource manager (NodeManager) reports the resource usage to the system resource manager (ResourceManager) in a heartbeat manner. The resource usage is mainly the CPU usage and the memory usage. The system resource manager (ResourceManager) only accepts the resource report information from the node resource manager (NodeManager), and the specific resource processing is left to the node resource manager (NodeManager) to handle. The node resource manager (NodeManager) is the resource and task manager on each computer node 14, and is the agent for managing the machine, responsible for the running of the computer node 14 program, and the management and monitoring of the node resources. The distributed cluster system 10 runs a node resource manager (NodeManager) on each computer node 14. The node resource manager (NodeManager) reports the resource usage of the node and the running state of the container process to the system resource manager (ResourceManager) at regular intervals.

[0073] When a user submits an application to the distributed cluster system 10 through the client 12, the application includes an ApplicationMaster program, a command to start the ApplicationMaster, a user program, etc. The system resource manager ResourceManager allocates a first container process to the application and communicates with the corresponding node resource manager NodeManager to require it to start the ApplicationMaster of the application in the container. The ApplicationMaster first registers with the system resource manager ResourceManager, so that the user can directly check the running state of the application through the system resource manager ResourceManager, and then it will apply for resources for each task and monitor the running state until the end of the running. The ApplicationMaster applies for and collects resources from the system resource manager ResourceManager in a polling manner through an RPC (Remote Procedure Call) protocol. Once the ApplicationMaster applies for resources, it communicates with the corresponding node resource manager NodeManager to require it to start the task. The node resource manager NodeManager sets up a running environment for the task, writes a task starting command to a script, and starts the task by running the script. Each task reports its state and progress to the ApplicationMaster through a certain RPC protocol, so that the ApplicationMaster can grasp the running state of each task at any time, so that the ApplicationMaster can restart the task when the task fails. During the running of the application, the user can query the current running state of the application through the RPC to the ApplicationMaster at any time. After the application is completed, the ApplicationMaster unregisters with the system resource manager ResourceManager and closes itself.

[0074] In the embodiments of the present application, the computer device 111, the computer node 14 and the client 12 can be independent servers, or a server network or a server cluster composed of servers. For example, the computer device 111 described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0075] In the embodiments of the present application, the computer device 111 described above can be a general-purpose computer device or a special-purpose computer device. In specific implementation, the computer device 111 can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, an embedded device, etc. The embodiments are not limited to the type of the computer device 111.

[0076] Those skilled in the art can understand that, Figure 1 The application environment shown in the above is only one application scenario of the scheme of the present application, and does not constitute a limitation on the application scenarios of the scheme of the present application. Other application environments can include more or fewer computer devices than those shown in the above, for example, Figure 1 The above only shows one computer device, and it can be understood that the distributed cluster system 10 can also include one or more other computer devices that can process data, which are not limited in detail here. Figure 1 The above only shows one computer device, and it can be understood that the distributed cluster system 10 can also include one or more other computer devices that can process data, which are not limited in detail here.

[0077] In addition, as shown in the above, Figure 1 The distributed cluster system 10 can also include a memory 112 for storing data.

[0078] It should be noted that, Figure 1 The scenario diagram of the distributed cluster system 10 shown in the above is only one example. The distributed cluster system 10 and the scenario described in the embodiments are used to more clearly illustrate the technical scheme of the embodiments of the present application, and do not constitute a limitation on the technical scheme provided by the embodiments of the present application. Those skilled in the art can know that, as the distributed cluster system 10 evolves and new business scenarios appear, the technical scheme provided by the embodiments of the present application is also applicable to similar technical problems.

[0079] First, the embodiments of the present application provide a distributed cluster load control method. The distributed cluster load control method includes: being applied to a distributed cluster system, the distributed cluster system including a system resource manager and a plurality of computer nodes connected to the system resource manager. The distributed cluster load control method includes: obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system, wherein the target node is one of the plurality of computer nodes, and the kernel default parameters include a memory recycling trend control parameter, a memory recycling dirty data threshold, and a dirty data expiration time; adjusting the kernel default parameters of the target node to preset kernel parameters to reduce the memory occupied by the operating system of the target node; and performing load control on each computer node in the distributed cluster system after the adjustment.

[0080] As shown in the above, Figure 2 Figure 2 ​is a flowchart of an embodiment of a method for controlling a distributed cluster load in the embodiments of the present application. The method for controlling a distributed cluster load includes the following steps S201-S203:

[0081] S201, obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system.

[0082] The target node is one of the plurality of computer nodes 14. It should be noted that the present application only needs to adjust the parameters of one computer node 14. The parameters of all computer nodes 14 in the distributed cluster system 10 can be adjusted according to the same method, thereby improving the resource utilization of the distributed cluster system 10.

[0083] The kernel is the core of an operating system. It is the first layer of software expansion based on hardware, which provides the most basic functions of the operating system and is the basis of the operating system. It is responsible for managing the processes, memory, device drivers, files, and network systems of the system, and determines the performance and stability of the system.

[0084] The kernel default parameters are default parameters commonly used by the nodes in the current distributed cluster system. Since there are many types of kernel default parameters and it is complex to adjust the kernel default parameters, arbitrary changes will affect the performance of the machine. In the distributed cluster system, there are many computer nodes 14. Generally, technicians optimize the distributed cluster system from a macroscopic system level, such as modifying the queue and scheduling strategy, and generally do not modify the kernel default parameters of the computer nodes. These microscopic computer node parameters are only used to adjust the memory and cache usage of the operating system, and cannot directly improve the capacity of the macroscopic distributed system to achieve overselling. Therefore, in the existing distributed system, in order to maintain the stability of the entire cluster system, the kernel default parameters of the computer nodes are generally not modified, and the system comes with the kernel default parameters. However, these kernel default parameters limit the capacity of the computer nodes 14 and the capacity of the distributed cluster system 10, resulting in low resource utilization of the computer nodes 14 in the distributed cluster system 10.

[0085] In the embodiments of the present application, the kernel default parameters can include a default value of a memory recycling trend control parameter, wherein the memory recycling trend control parameter represents the tendency of the kernel to recycle the directory and inode cache. The memory recycling trend control parameter is generally represented by vm.vfs_cache_pressure, and the general default value of the memory recycling trend control parameter is 100.

[0086] The kernel default parameters can include a default value of a dirty data threshold for memory reclamation. The dirty data threshold for memory reclamation is an absolute maximum amount of system memory that can be filled with dirty data. When the amount of data in the system reaches the dirty data threshold for memory reclamation, all dirty data must be committed to disk, and all new I / O blocks are blocked until the dirty data is written to disk. The dirty data threshold for memory reclamation is generally represented by vm.dirty_ratio. The generally used default value of the dirty data threshold for memory reclamation is 40, which means that when the capacity of dirty data reaches 40% of the system memory, the dirty data is committed to disk.

[0087] The kernel default parameters can include a default value of a dirty data expiration time for the target node. The dirty data expiration time indicates that if the data stays in the memory for more than the value, the computer node 14 will determine the data as dirty data and write it back to the disk on the target node. The dirty data expiration time is generally represented by vm.dirty_expire_centisecs. The generally used default value of the dirty data expiration time is 3000 ms.

[0088] S202, adjusting the kernel default parameters of the target node to the preset kernel parameters to reduce the size of the occupied memory space of the operating system of the target node.

[0089] In one specific embodiment, adjusting the kernel default parameters of the target node to the preset kernel parameters to reduce the size of the occupied memory space of the operating system of the target node can include increasing the default value of the memory reclamation trend control parameter to a first preset value, and the first preset value is greater than the default value of the memory reclamation trend control parameter. Optionally, the value range of the first preset value is 300 to 500. Increasing the memory reclamation trend control parameter vfs_cache_pressure from the default value 100 to 300 to 500 will cause the kernel to be more inclined to reclaim the directory and inode cache, so that the operating system of the computer node 14 can reclaim the cache memory more frequently and limit the size of the page cache, thereby reducing the size of the occupied memory space of the operating system of the target node, releasing more memory for application programs, and improving the resource utilization of the computer node 14 in the distributed cluster system 10.

[0090] In one specific embodiment, adjusting the kernel default parameters of the target node to the preset kernel parameters to reduce the size of the memory space occupied by the operating system of the target node can include: reducing the default value of the memory dirty data threshold of the target node to a second preset value, and the second preset value is less than the default value of the memory dirty data threshold. Wherein, when the proportion of the dirty data occupied space to the system memory of the target node reaches the second preset value, the process running on the target node writes the dirty data back to the disk on the target node. Optionally, the first preset value ranges from 0 to 10. Reducing the memory dirty data threshold vm.dirty_ratio from the default value 40 to 0-10 will cause the process to write out dirty data faster, which will limit the size of the page cache, release more memory for application use, and improve the resource utilization of the computer node 14 in the distributed cluster system 10.

[0091] In one specific embodiment, adjusting the kernel default parameters of the target node to the preset kernel parameters to reduce the size of the memory space occupied by the operating system of the target node can include: reducing the default value of the memory dirty data threshold of the target node to a second preset value, and the second preset value is less than the default value of the memory dirty data threshold. Wherein, when the proportion of the dirty data occupied space to the system memory of the target node reaches the second preset value, the process running on the target node writes the dirty data back to the disk on the target node. Optionally, the first preset value ranges from 0 to 10. Reducing the memory dirty data threshold vm.dirty_ratio from the default value 40 to 0-10 will cause the process to write out dirty data faster, which will limit the size of the page cache, release more memory for application use, and improve the resource utilization of the computer node 14 in the distributed cluster system 10.

[0092] In one specific embodiment, the kernel default parameters further include a swap partition weight parameter, which represents the relative weight of swapping out runtime memory, generally controlled by vm.swappiness. The size of the relative weight parameter of the swapped out runtime memory has a great relationship with how to use the swap partition. The default value of the swap partition weight parameter vm.swappiness is generally 60, indicating that when the memory usage exceeds 100%-60%=40%, the swap partition is used.

[0093] The adjusting the kernel default parameters of the target node to preset kernel parameters to reduce the memory space occupied by the operating system of the target node can include: reducing the default value of the swap partition weight parameter to a sixth preset value. The sixth preset value is 0-10. Reducing the default value of the swap partition weight parameter vm.swappiness from 60 to 0-10 can make the memory usage be swapped only when the memory usage is high, make the data more willing to write data to the disk, limit the page cache size, thereby providing more memory space for the application program, and improve the resource utilization of the computer node 14 in the distributed cluster system 10.

[0094] In a specific embodiment, the kernel default parameters further include a swap area parameter, and the swap area parameter represents a swap area to physical memory size ratio, and the general default value is 1. When the RAM usage exceeds the limit, in order to make the system more stable, a part of the space on the hard disk is divided as a memory buffer, that is, a swap area. When the memory usage exceeds the limit, the kernel stores the idle data in the memory in the swap area.

[0095] The adjusting the kernel default parameters of the target node to preset kernel parameters to reduce the memory space occupied by the operating system of the target node can include: increasing the default value of the swap area parameter to a seventh preset value. The seventh preset value is 0-10. Increasing the default value of the swap area parameter from 1 to 1.5-3 can make the data more willing to write to the disk, limit the page cache size, thereby providing more memory space for the application program, and improve the resource utilization of the computer node 14 in the distributed cluster system 10.

[0096] S203, performing load control on each computer node in the distributed cluster system after the parameter adjustment.

[0097] In the embodiments of the application, the load control is performed on each computer node 14 in the distributed cluster system 10 after the parameter adjustment, and the load control includes:

[0098] (1) obtaining the task load of each computer node 14 in the distributed cluster system 10 after the parameter adjustment.

[0099] In the embodiments of the application, the task load can be CPU usage and / or memory usage. For example, the task load of the computer node 14 is that the CPU usage is 90%, and the memory usage is 50%.

[0100] (2) determining whether a first computer node exists in each computer node 14, wherein the task load of the first computer node satisfies a high-load condition.

[0101] In the embodiments of the present application, the high load condition includes: the CPU usage is higher than a first preset CPU usage, and the memory usage is higher than a first preset memory usage. For example, the first preset CPU usage is 90%, and the first preset memory usage is 90%. The first preset CPU usage and the first preset memory usage can be set according to specific conditions.

[0102] (3) If the first computing node exists in each computer node 14, a second computing node with a task load lower than a second preset load value is obtained.

[0103] In the embodiments of the present application, the low load condition includes: the CPU usage is lower than a second preset CPU usage, and the memory usage is lower than a second preset memory usage. For example, the second preset CPU usage is 50%, and the second preset memory usage is 50%. The second preset CPU usage and the second preset memory usage can be set according to specific conditions. The second preset CPU usage is less than the first preset CPU usage, and the second preset memory usage is less than the first preset memory usage.

[0104] (4) The task running on the first computing node is migrated to the second computing node.

[0105] In a specific embodiment, the application identifier (Application ID) of the application program running on the first computing node is detected, and a corresponding application list is recorded. The execution of each task in the application list is suspended, the tasks in the application list are distributed to each second computing node, and the execution of each task in the application list is resumed. When the number of second computing nodes is multiple, the tasks in the application list can be evenly distributed to each second computing node. Each second computing node can be assigned a corresponding weight coefficient based on the task load of each second computing node, and each second computing node can be assigned a corresponding task according to the weight coefficient of each second computing node.

[0106] In the prior art, the effect of adjusting the cluster macro layer to improve the available resources of the application program is not obvious. In the present application, the kernel default parameters of the target node in the distributed cluster system are obtained when the distributed cluster system is started, and the kernel default parameters of the target node are adjusted to reduce the memory occupied by the operating system of the target node, thereby improving the available resources of the application program. Without hardware and software architecture modification, the available resources of the application program can be improved by adjusting the kernel default parameters of the computer node micro layer, and the load control of the distributed cluster system after parameter adjustment is performed, thereby effectively improving the resource utilization of the distributed cluster system.

[0107] Further, refer to Figure 3, each computer node 14 includes a node resource manager NodeManager, which is used to communicate with the system resource manager ResourceManager and manage the computer node 14 where it is located. The distributed cluster load control method includes S301-S304:

[0108] S301, obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system.

[0109] In the embodiment of the present application, the specific steps of S301 can refer to S201 in the previous embodiment, which will not be repeated here.

[0110] S302, adjusting the kernel default parameters of the target node to preset kernel parameters to reduce the size of the operating system occupied memory space of the target node.

[0111] In the embodiment of the present application, the specific steps of S302 can refer to S202 in the previous embodiment, which will not be repeated here.

[0112] S303, adjusting the node resource scheduling parameters of the node resource manager to improve the configuration capacity of each target node.

[0113] In the embodiment of the present application, the node scheduling resource parameters of the node resource manager can include a number of schedulable virtual CPUs and a schedulable memory capacity. The number of schedulable virtual CPUs is generally represented by yarn.nodemanager.resource.cpu-vcores, which represents the number of virtual CPUs that the cluster can use on the computer node 14, and the default is 80. Generally, the system recommends that the default value be configured to be the same as the number of physical cores. The schedulable memory capacity is generally represented by yarn.nodemanager.resource.memory-mb, which represents the memory that the cluster can use on the computer node 14, and the default is 8G.

[0114] In one specific embodiment, adjusting the node resource scheduling parameters of the node resource manager to improve the configuration capacity of each target node can include: obtaining a default value of the schedulable memory capacity of the target node; and increasing the default value of the schedulable virtual CPU number to a fourth preset value, the fourth preset value being greater than the default value of the schedulable virtual CPU number. Optionally, the fourth preset value is in the range of 2-4 times the number of physical cores of the computer node 14. For example, if the number of physical cores of the computer node 14 is 80, the fourth preset value is 160-320. That is, the schedulable virtual CPU number of the node resource manager NodeManager is 160-320. The process scheduling capability, hyper-threading capability, etc. can be fully utilized to effectively improve the CPU utilization, improve the performance of the computer, and improve the system reliability.

[0115] In one specific embodiment, adjusting the node resource scheduling parameters of the node resource manager to improve the configuration capacity of each target node can include: obtaining a default value of the schedulable memory capacity of the target node; and increasing the default value of the schedulable virtual CPU number to a fourth preset value, the fourth preset value being greater than the default value of the schedulable virtual CPU number. Optionally, the fourth preset value is in the range of 2-4 times the number of physical cores of the computer node 14. For example, if the number of physical cores of the computer node 14 is 80, the fourth preset value is 160-320. That is, the schedulable virtual CPU number of the node resource manager NodeManager is 160-320. The process scheduling capability, hyper-threading capability, etc. can be fully utilized to effectively improve the CPU utilization, improve the performance of the computer, and improve the system reliability.

[0116] Further, the CGroup is started, and the node resource manager NodeManager limits the container in the target node through the CGroup to allocate resources in the target node, thereby limiting the abnormal surge of CPU usage. The CGroup, control group, provides a mechanism for controlling the use of resources by a specific group of processes. The CGroup binds a process set to one or more subsystems. Specifically, when a multi-threaded loop bug of an application program causes the CPU usage to surge, the corresponding container is closed through the CGroup. For example, the CPU suddenly increases to 15000%, which is usually caused by a software bug. The node resource manager kills the container in time through the CGroup and requires the developer to check the code and debug offline.

[0117] S304, performing load control on each computer node in the distributed cluster system after the parameter adjustment.

[0118] In the embodiment, the specific steps of S304 can refer to S203 in the previous embodiment, which will not be described here.

[0119] In a preferred embodiment, the default value of the memory reclamation trend control parameter is increased to a first preset value, i.e., the memory reclamation trend control parameter vfs_cache_pressure is increased from the default value 100 to 300 to 500; the default value of the memory reclamation dirty data threshold of the target node is reduced to a second preset value, i.e., the memory reclamation dirty data threshold vm.dirty_ratio is reduced from the default value 40 to 0 to 10; the default value of the dirty data expiration time is reduced to a third preset value, i.e., the memory reclamation dirty data threshold vm.dirty_expire_centisecs is reduced from the default value 3000 ms to 0 ms to 10 ms; the default value of the swap partition weight parameter is reduced to a sixth preset value, i.e., the default value of the swap partition weight parameter vm.swappiness is reduced from 60 to 0 to 10; the default value of the swap area parameter is increased to a seventh preset value, i.e., the default value of the swap area parameter is increased from 1 to 1.5 to 3; the default value of the number of schedulable virtual CPUs is increased to a fourth preset value, i.e., the number of schedulable virtual CPUs of the node resource manager NodeManager is increased from the default value 80 to 160 to 320; the default value of the schedulable memory capacity is increased to a fifth preset value, i.e., the default value of the schedulable memory capacity is increased to 2 to 4 times the physical memory of the computer node 14. After a large number of researches by the inventors, the use of the set of parameters together can significantly improve the resource utilization of the distributed cluster system 10, and the effect is qualitatively improved compared to the adjustment of a single parameter.

[0120] On the one hand, the embodiment reduces the operating system memory occupied by each computer node in the distributed cluster system by adjusting the kernel parameters, and on the other hand, the embodiment adjusts the parameters of the node resource manager NodeManager on the computer node to improve the configuration capacity of the computer node, without the need for hardware and software architecture changes to effectively improve the resource utilization of the distributed cluster system.

[0121] In order to better implement the distributed cluster load control method in the embodiment, on the basis of the distributed cluster load control method, the embodiment also provides a distributed cluster load control device, as shown in Figure 4 The distributed cluster system includes a system resource manager and a plurality of computer nodes connected to the system resource manager, and the distributed cluster load control device 300 includes:

[0122] The acquisition unit 301 is configured to acquire a kernel default parameter of a target node in the distributed cluster system when starting the distributed cluster system, wherein the target node is one of the plurality of computer nodes.

[0123] The parameter adjustment unit 302 is configured to adjust the kernel default parameter of the target node to a preset kernel parameter to reduce the memory occupied by the operating system of the target node, wherein the kernel default parameter includes a memory recycling trend control parameter, a memory recycling dirty data threshold, and a dirty data expiration time.

[0124] The load control unit 303 is configured to perform load control on each computer node in the distributed cluster system after the parameter adjustment.

[0125] Optionally, the load control unit 303 is configured to:

[0126] acquire a task load of each computer node in the distributed cluster system after the parameter adjustment.

[0127] determine whether a first computer node exists in the computer nodes, wherein the task load of the first computer node satisfies a high-load condition.

[0128] If the first computer node exists in the computer nodes, acquire a second computer node whose task load satisfies a low-load condition.

[0129] migrate a task running on the first computer node to the second computer node.

[0130] Optionally, the high-load condition includes that a CPU usage rate is higher than a first preset CPU usage rate and a memory usage rate is higher than a first preset memory usage rate, and the low-load condition includes that the CPU usage rate is lower than a second preset CPU usage rate and the memory usage rate is lower than a second preset memory usage rate.

[0131] Optionally, the memory recycling trend control parameter represents a kernel tendency to recycle a file directory and an inode cache,

[0132] The parameter adjustment unit 302 is configured to:

[0133] increase a default value of the memory recycling trend control parameter to a first preset value.

[0134] Optionally, the parameter adjustment unit 302 is configured to:

[0135] decrease a default value of the memory recycling dirty data threshold of the target node to a second preset value, wherein a process running on the target node writes dirty data back to a disk on the target node when a proportion between a dirty data occupied space and a system memory of the target node reaches the second preset value.

[0136] Optionally, the parameter adjustment unit 302 is configured to:

[0137] decrease the default value of the dirty data expiration time to a third preset value, wherein the target node is confirmed as dirty data and written back to the disk on the target node when the data residence time exceeds the third preset value.

[0138] Optionally, the target node comprises a node resource manager configured to communicate with the system resource manager and manage the computer node where the target node is located, and the parameter adjustment unit 302 is configured to:

[0139] adjust the node resource scheduling parameter of the node resource manager to improve the configuration capacity of the target node.

[0140] Optionally, the node resource scheduling parameter comprises a number of schedulable virtual CPUs, and the parameter adjustment unit 302 is configured to:

[0141] obtain a default value of the number of schedulable virtual CPUs of the target node;

[0142] increase the default value of the number of schedulable virtual CPUs to a fourth preset value.

[0143] Optionally, the node resource scheduling parameter comprises a schedulable memory capacity, and the parameter adjustment unit 302 is configured to:

[0144] obtain a default value of the schedulable memory capacity of the target node;

[0145] increase the default value of the schedulable memory capacity to a fifth preset value.

[0146] The embodiments of the present application further provide a computer device integrating any of the distributed cluster load control apparatuses provided by the embodiments of the present application, and the computer device comprises:

[0147] one or more processors;

[0148] a memory; and

[0149] one or more application programs, wherein the one or more application programs are stored in the memory and configured to execute the steps in the distributed cluster load control method in any of the embodiments of the distributed cluster load control method.

[0150] As shown in Figure 5 , a structural schematic diagram of a computer device related to the embodiments of the present application is shown, and specifically:

[0151] The computer device can include a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a power supply 403, an input unit 404, and the like. Those skilled in the art can understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and can include more or fewer components than shown in the figure, or combine certain components, or different component arrangements. Among them:

[0152] The processor 401 is the control center of the computer device, connects various parts of the entire computer device through various interfaces and lines, executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, thereby overall monitoring the computer device. Optionally, the processor 401 can include one or more processing cores; the processor 401 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like, and preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0153] The memory 402 can be used to store software programs and modules, and the processor 401 executes various function applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access for the processor 401 to the memory 402.

[0154] The computer device further includes a power supply 403 for supplying power to the various components. Preferably, the power supply 403 is logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, power consumption management, and the like. The power supply 403 can further include one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

[0155] The computer device can further include an input unit 404 for receiving input digital or character information and generating keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0156] Although not shown, the computer device can further include a display unit and the like, which will not be described here. In the present embodiment, the processor 401 in the computer device loads one or more executable files corresponding to processes of one or more application programs into the memory 402 and runs the application programs stored in the memory 402 according to the following instructions, thereby implementing various functions, such as:

[0157] Obtaining kernel default parameters of a target node in a distributed cluster system when starting the distributed cluster system, wherein the target node is one of a plurality of computer nodes; adjusting the kernel default parameters of the target node to preset kernel parameters to reduce memory occupied by an operating system of the target node; and performing load control on each computer node in the distributed cluster system after the adjustment.

[0158] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0159] To this end, the embodiments of the present application provide a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like. The computer readable storage medium has a computer program stored thereon, which is loaded by a processor to execute the steps in any of the distributed cluster load control methods provided by the embodiments of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0160] The kernel default parameters of a target node in the distributed cluster system are acquired when the distributed cluster system is started, wherein the target node is one of the plurality of computer nodes; the kernel default parameters of the target node are adjusted to preset kernel parameters to reduce the memory occupied by the operating system of the target node; and the load of each computer node in the distributed cluster system after the adjustment is controlled.

[0161] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be repeated here.

[0162] In the implementation, the above units or structures can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above units or structures can be referred to the method embodiments above, which will not be repeated here.

[0163] The specific implementation of the above operations can be referred to the above embodiments, which will not be repeated here.

[0164] The distributed cluster load control method and device provided by the embodiments of the present application are described in detail above, and the principle and implementation mode of the present application are described by applying specific examples in this paper. The above embodiment is only used to help understand the method and its core idea of the present application. Meanwhile, according to the idea of the present application, the specific implementation mode and application range will be changed by the skilled in the art, and the content of the specification should not be understood as the limitation of the present application.

Claims

1. A method for controlling distributed cluster load, characterized by, The application is applied to a distributed cluster system including a system resource manager and a plurality of computer nodes connected with the system resource manager, and a control method of the distributed cluster load includes: obtaining kernel default parameters of a target node in the distributed cluster system when starting the distributed cluster system, wherein the target node is one of the plurality of computer nodes, and the kernel default parameters include a memory recycling trend control parameter, a memory recycling dirty data threshold and a dirty data expiration time; adjusting the kernel default parameters of the target node to preset kernel parameters to reduce the operating system memory occupancy of the target node; wherein the default value of the memory recycling trend control parameter is increased to a first preset value, the first preset value is greater than the default value of the memory recycling trend control parameter; the default value of the memory recycling dirty data threshold of the target node is reduced to a second preset value, the second preset value is less than the default value of the memory recycling dirty data threshold; and the default value of the dirty data expiration time is reduced to a third preset value, the third preset value is less than the default value of the dirty data expiration time; controlling the load of each computer node in the distributed cluster system after the adjustment.

2. The control method of distributed cluster load according to claim 1, wherein, The load control of each computer node in the distributed cluster system after the adjustment includes: obtaining the task load of each computer node in the distributed cluster system after the adjustment; determining whether there is a first computer node in each computer node, wherein the task load of the first computer node meets a high load condition; if there is the first computer node in each computer node, obtaining a second computer node whose task load meets a low load condition; migrating the task running on the first computer node to the second computer node.

3. The control method of distributed cluster load according to claim 2, wherein, The high load condition includes that the CPU usage is higher than a first preset CPU usage, and the memory usage is higher than a first preset memory usage; and the low load condition includes that the CPU usage is lower than a second preset CPU usage, and the memory usage is lower than a second preset memory usage.

4. The control method of distributed cluster load according to claim 1, wherein, The memory recycling trend control parameter represents the kernel recycling tendency for file directory and inode cache. The adjustment of the kernel default parameters of the target node to the preset kernel parameters to reduce the operating system memory occupancy of the target node includes:

5. The control method of distributed cluster load according to claim 1, wherein, when the proportion of the dirty data occupancy space and the system memory of the target node reaches the second preset value, the process running on the target node writes the dirty data back to the disk on the target node. The adjustment of the kernel default parameters of the target node to the preset kernel parameters to reduce the operating system memory occupancy of the target node includes:

6. The control method of distributed cluster load according to claim 1, wherein, when the residence time of the data exceeds the third preset value, the target node is confirmed as dirty data and writes back to the disk on the target node. ​ 7. The control method of distributed cluster load according to claim 1, wherein, The target node includes a node resource manager, which communicates with the system resource manager and manages the computer node it resides in. Prior to performing load control on each computer node in the tuned distributed cluster system, the process includes: Adjust the node resource scheduling parameters of the node resource manager to increase the configuration capacity of the target node.

8. The control method of distributed cluster load according to claim 7, wherein, The node resource scheduling parameters include the number of schedulable virtual CPUs. Adjusting the node resource scheduling parameters of the node resource manager to increase the configuration capacity of the target node includes: Obtain the default value for the number of schedulable virtual CPUs of the target node; Increase the default value of the number of schedulable virtual CPUs to the fourth preset value.

9. The control method of distributed cluster load according to claim 7, wherein, The node resource scheduling parameters include schedulable memory capacity. Adjusting the node resource scheduling parameters of the node resource manager to increase the configuration capacity of the target node includes: Obtain the default value of the schedulable memory capacity of the target node; Increase the default value of the schedulable memory capacity to the fifth preset value.

10. A control device for distributed cluster load, characterized by, Applied to a distributed cluster system, the distributed cluster system including a system resource manager and multiple computer nodes connected to the system resource manager, the load control device of the distributed cluster includes: The acquisition unit is used to acquire the kernel default parameters of the target node in the distributed cluster system when the distributed cluster system is started, wherein the target node is one of the plurality of computer nodes, and the kernel default parameters include memory reclamation trend control parameters, memory reclamation dirty data threshold, and dirty data expiration time; A parameter adjustment unit is used to adjust the default kernel parameters of the target node to preset kernel parameters to reduce the memory occupied by the operating system of the target node; wherein, the default value of the memory reclamation trend control parameter is increased to a first preset value, the first preset value being greater than the default value of the memory reclamation trend control parameter; the default value of the dirty data threshold for memory reclamation of the target node is decreased to a second preset value, the second preset value being less than the default value of the dirty data threshold for memory reclamation; and the default value of the dirty data expiration time is decreased to a third preset value, the third preset value being less than the default value of the dirty data expiration time. The load control unit is used to control the load of each computer node in the distributed cluster system after parameter tuning.

11. A computer readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the steps in the distributed cluster load control method according to any one of claims 1 to 9.

Images