Linux Large-Scale Cluster Application Management System

Abstract

The present invention proposes a Linux large-scale cluster application management system, which realizes unified centralized monitoring and management of different cluster applications based on an agentless hierarchical application management model. It divides the applications in the Linux cluster and distributed environment into primary and standby applications, parallel applications, resource pool applications and mutually exclusive applications according to the operation mode, and divides the management objects into three types according to the functions: application, application group and application system. In this model, a multi-tree pruning algorithm is used to solve the dependency relationship between multiple applications; a method for comprehensively evaluating the application status according to the three dimensions of operation mode, process keyword and process number is proposed; an application balancing scheduling algorithm is proposed for resource pool mode applications; an application cloning technology is proposed to clone the application on one host to another host; and cache and parallelization technology are used to improve the execution efficiency of the model. Based on this model, a Linux large-scale cluster application management system is implemented, which improves the automation level of IT operation and maintenance.

Description

Technical Field

[0001] The present invention relates to the technical field of cluster management, and in particular to a Linux large-scale cluster application management system. Background Art

[0002] A cluster refers to a group of independent computers that are interconnected through a high-speed network. They form a group and provide services in the mode of a single system. Clusters are divided into load balancing clusters, high-availability clusters, high-performance computing clusters, grid computing clusters, etc. according to their functions. A distributed system is a system in which hardware or software components are distributed on different network computers and communicate and coordinate with each other only through message passing. There are distributed storage, distributed computing, distributed databases, etc. The use of cluster and distributed technology can improve system performance and availability, and the performance can be elastically expanded, but the difficulty of system operation and maintenance and the cost of IT operation and maintenance will increase dramatically. The business that originally ran on a small computer requires dozens of servers in a cluster environment. In a distributed environment, a system needs to be split into multiple business modules and run on different servers. This change has greatly increased the complexity and integration of the IT platform. Traditional operation and maintenance tools and technologies cannot meet the requirements of high-quality operation of such business systems.

[0003] In order to solve the problem of application management in a multi-host environment, some automated operation and maintenance software has emerged, such as Ansible, Puppet, Saltstack, and PSSH. They can implement batch system configuration, batch program deployment, batch command execution and other functions for multiple hosts, but they cannot yet serve as a general application management platform.

[0004] In 2016, Gartner proposed the concept of intelligent operation and maintenance, namely AIOps (Artificial Intelligence for IT Operations), which aims to combine artificial intelligence with operation and maintenance and improve operation and maintenance efficiency through machine learning methods. IBM Cloud Pak for Watson AIOps is an AI-driven, hybrid cloud-oriented integrated tool launched by IBM, which has the ability to evaluate, diagnose and resolve events in mission-critical workloads. AIOps is the next-generation solution for IT operations, but it has high technical barriers, few mature products, and is only used in large Internet companies.

[0005] Subsequently, the DevOps (Development + Operations) concept of integrated operation and maintenance changed the traditional software engineering development model and integrated software operation and maintenance into the entire software development cycle. Under the DevOps process, operation and maintenance personnel are involved in the development process during project development, and developers will also participate in system operation and maintenance in the early stages of operation and maintenance. The software developed in this way will be more reliable and easy to maintain. However, DevOps is still in development, and there are very few systems developed using the DevOps concept. Although domestic DevOps has shown a steady development trend, there is still a clear gap compared with the international level. Therefore, DevOps will not be able to change the current status of IT operation and maintenance in the short term.

[0006] At present, cluster application management systems are relatively mature in the field of high-performance computing clusters (HPC), such as IBM LoadLeveler, openPBS / Torque, slurm, LSF, CONDOR, SGE, etc. They provide functions such as resource classification, job monitoring, and job scheduling for multiple hosts in a parallel environment. However, they are mainly aimed at parallel computing task environments, with fine granularity of resource management and complex configuration. They are not suitable for general clusters and distributed environments with a large number of task types and coarse scheduling granularity.

[0007] Veritas InfoScale Availability is a widely used cluster environment application management software. It has the function of managing multiple cluster resources and provides management functions such as guarding, starting, stopping, and monitoring for cluster applications. It can ensure that key applications continue to operate normally when encountering unexpected failures. However, the software has the following problems: (1) The application group classification logic is unreasonable. It can only be divided into groups according to hosts, not functions. All applications in the group must use the same host list, resulting in a large number of application groups; (2) The monitoring function is simple and not intelligent enough. The monitoring results are only for each host, and there is no global evaluation result; (3) The software only supports two operating modes: active-standby and parallel. It cannot manage and monitor mutually exclusive applications and resource pool applications; (4) The software uses a flat mode. All hosts must be in a cluster. It does not support hierarchical management. As the number of hosts and applications increases, the management difficulty increases accordingly; (5) The software only provides simple monitoring, starting, and stopping operations, and has no other operation and maintenance tools; (6) The software uses an agent mode. Each host needs to install agent software; (8) The software is commercial software.

[0008] The present invention is based on the Gansu meteorological big data cloud platform, which has 4 AIX servers and 143 Linux servers. The applications running on the servers are divided into 12 independent systems and 85 non-repetitive applications according to their functions, and almost all systems are managed manually. Due to the lack of a centralized and unified application management system, in the process of application management, IT operation and maintenance personnel need to log in to different hosts many times, and also need to remember the start and stop procedures, storage paths, host and other information of each application. It is easy to forget the application startup program, input errors, miss the application, start the application that should not be started, etc. The efficiency of system management is very low and prone to errors. At the same time, the cost of system operation and maintenance is high. Summary of the invention

[0009] In view of the above problems, the present invention proposes a universal application management model and implements an application management system, which can manage various applications in cluster and distributed environments, has a grading function, and makes up for the shortcomings of other management systems. At the same time, the system is used to manage all applications of the Gansu Meteorological Big Data Cloud Platform, and the management effect is very efficient.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A Linux large-scale cluster application management system, which is implemented based on a general application management model, characterized in that the system adopts a hierarchical management mode, and is composed of a first-level application management system and a second-level super management system, wherein the first-level application management system is arranged on each subsystem to implement the management function of the general application management model; the second-level super management system is used to manage the first-level application management system deployed on each subsystem;

[0012] Super management system, used to manage multiple application management systems;

[0013] The application management system is used to manage all applications within the application system.

[0014] Furthermore, the application management system includes a configuration management module, an application control module, an application monitoring module, a system monitoring module, an operation and maintenance tool module, a log module, and a display module;

[0015] Configuration management module, used to configure parameters in the system;

[0016] Application monitoring module, used to monitor the application status in the system;

[0017] Application control module, used to start, stop, restart and other operations on each application;

[0018] System monitoring module, used to monitor the usage of host CPU, memory, disk space, and disk IONODE space;

[0019] The operation and maintenance tool module is used to provide multiple maintenance tools to realize remote process viewing, remote command execution, application backup and restore, application cloning, crontab backup and restore, establish ssh mutual trust between multiple hosts, and configuration conversion;

[0020] A log module is used to record the operations performed by the application management system to form an operation log;

[0021] The display module is used to display the operation results.

[0022] Furthermore, the general application management model includes application management objects, dependencies between application management objects, and management behaviors of application management objects;

[0023] Application management objects, including applications, application groups, and application systems;

[0024] Dependency, which is used to describe the startup order between different applications;

[0025] Management behavior is used to describe various operation behaviors for different management objects.

[0026] Furthermore, the application runs on one or more hosts, has the same startup command, resides in memory after startup, and provides continuous service;

[0027] The application group is composed of multiple applications that implement a function according to certain rules;

[0028] The application system is composed of multiple application groups combined according to certain rules.

[0029] Furthermore, the applications are divided into primary and standby applications, parallel applications, resource pool applications and mutually exclusive applications according to the operation mode;

[0030] Active / standby applications can only run on one of the hosts in the host list;

[0031] Parallel application, running on all hosts in the host list;

[0032] Resource pool applications run on any number of hosts in the resource pool;

[0033] Mutually exclusive applications only support application groups. Only one of two different application groups can be started at the same time. If one application group is started, the other application group must be stopped.

[0034] Furthermore, the startup method of the application includes:

[0035] Multi-host parallel startup: multiple hosts are started in parallel from the host list. Candidate startup hosts are obtained from the host list in sequence. This is suitable for parallel applications and resource pool applications.

[0036] Balanced startup mode: Start multiple hosts in parallel. Candidate startup hosts are automatically selected from the host list after being sorted by CPU load. This mode is suitable for resource pool mode applications.

[0037] Single-host startup runs on multiple hosts. The startup command is executed on only one host. The candidate startup host selects the first available host from the host list. It is suitable for parallel applications or resource pool applications with built-in single-point startup functions.

[0038] Furthermore, the dependency relationship is divided into intra-application group dependency relationship and inter-application group dependency relationship;

[0039] The dependency within an application group is used to describe the start and stop order of multiple applications within an application group. It is determined by the order of the applications in the group and is an implicit dependency that does not need to be declared.

[0040] The dependencies between application groups include implicit dependencies between groups and explicit dependencies between groups. The implicit dependencies between groups are used to describe the startup order of application groups in an application system. This relationship does not require configuration and is implicit in the order of application groups in the application system. The explicit dependencies between groups are used to describe the startup order between any application groups and are configurable in the application group.

[0041] Further, the configurable explicit inter-group dependencies include: a start-first group, a restart-later group, and a mutually exclusive group;

[0042] Start first group, used to set the upstream application group that must be started first when this application group starts. One application group supports configuring multiple upstream application groups;

[0043] Post-restart group: used to set the downstream application groups that need to be restarted when the current application group is restarted. Multiple groups can be supported.

[0044] Mutually exclusive group, used to set another application group that must be stopped when this application group is started.

[0045] Furthermore, the management behavior includes system monitoring, application monitoring, application control and operation and maintenance tools;

[0046] System monitoring, used to monitor the host CPU, memory, disk space, and disk IONODE space usage, and can set monitoring alarm thresholds and exclusion items;

[0047] Application monitoring is used to monitor whether the host, application, and application group are running normally, evaluate the application status from the perspective of running mode, process, and number of processes, and make qualitative judgments on each host, application, and application group;

[0048] Application control is used to implement application operations in a cluster or distributed environment. The operations include starting, stopping, forced stopping, restarting, active / standby switching, load balancing, and monitoring. The control objects are application systems, application groups, and applications.

[0049] Operation and maintenance tools are used to implement the operation and maintenance of applications in cluster or distributed environments, including remote process viewing, remote command execution, application backup and restore, application cloning, crontab backup and restore, ssh mutual trust tools, and VCS configuration conversion.

[0050] Furthermore, the application monitoring uses application status intelligent evaluation technology to evaluate the host and application running status, which includes process monitoring, host status evaluation and application status evaluation;

[0051] Process monitoring is used to monitor the number of processes in the application by setting monitoring items, and evaluate the process status as 0-normal, 1-partially abnormal, 2-not started, 3-ssh abnormal according to the number of processes corresponding to each monitoring item in the host. The evaluation formula is:

[0052]

[0053] Where i is the monitoring item number, ni is the number of processes of the ith monitoring item, Pi_min is the lower limit of the number of processes, Pi_max is the upper limit of the number of processes, and Pi represents the status of the ith monitoring item;

[0054] The host status assessment adopts a two-time assessment method. The first is pre-assessment and the second is assessment correction. During the pre-assessment, the pre-assessment results of a single host status are divided into 0-normal, 1-partially abnormal, 2-not started, and 3-ssh abnormal according to the status of the application process on the host. The pre-assessment formula is:

[0055]

[0056] Where N represents all monitoring items, i is the monitoring item number, Pi is the status of the i-th monitoring item, and Hj is the first evaluation result of the j-th host;

[0057] Assessment correction: the pre-assessment results are corrected in combination with the application mode. The corrected assessment results are divided into normal, partial abnormal, not started abnormal, ssh abnormal, violation of operation mode, and idle;

[0058] Application status assessment evaluates active-standby mode applications, parallel mode applications, and resource pool mode applications based on the host status assessment results. The assessment results are divided into three states: normal, abnormal, and partially abnormal.

[0059] The beneficial effects of the present invention are:

[0060] First, the application management system in the present invention has low requirements for the operating environment. In terms of hardware, only one computer is needed, and in terms of software, only Linux bash is needed. No other software needs to be installed, and no software needs to be installed on the managed machine, and no additional configuration is required. However, the "Tianjing" management system currently used by the Gansu Provincial Meteorological Big Data Cloud Platform requires 15 servers, and software such as Hadoop distributed file system, ElasticSearch and MySQL database, kafka message middleware, Flume log system, tomcat, nginx, client agent, etc. are installed, which is very large and complex to configure.

[0061] Second, it has good versatility. It can manage Linux cluster applications with different operation modes and startup methods, as well as non-cluster applications.

[0062] Third, the functions are complete. The management system includes three functions: monitoring, control, and operation and maintenance tools, while software such as Prometheus and Veritas InfoScale Availability only focus on one aspect.

[0063] Fourth, it is easy to use. One command can manage multiple cluster applications, and there is no need to remember the running host, running command, running parameters, running user and other information of each cluster.

[0064] Fifth, accurate monitoring. Active monitoring technology can obtain the application status on the monitored host in real time, while other monitoring systems with agents have a certain delay; fault information includes four parts: host, application, process, and fault cause, which can quickly locate the fault.

[0065] Sixth, high monitoring efficiency. Using parallel technology, the status of more than 360 applications and host status on more than 140 servers can be monitored within 5 seconds. Adding hosts and applications has no effect on monitoring efficiency.

[0066] Seventh, intelligent alarm. When there is no abnormality in the application system, it is in a silent state at night, only recording, and not sending any information. During the day, it sends a "safe" message every hour. When there is an abnormality, it sends an alarm message every ten minutes, and sends a "safe" message in the first ten minutes after the fault is restored.

[0067] Eighth, low management overhead. No agent software needs to be run on the managed machine, and it does not continuously occupy the host's memory, CPU and other resources. The monitoring command pushed by the management host to the managed computer is a parallel command after the monitoring commands of all applications on the computer are merged. Only one command needs to be executed for one monitoring, which has little interference with the managed computer. Existing management software needs to run agent software, which occupies CPU and memory resources; Veritas InfoScale Availability monitors each application independently, and each monitoring command is switched to a general user through su. When there are many applications on a host, it occupies more CPU and responds very slowly. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a schematic diagram of the application management system architecture;

[0069] Figure 2a is the initial startup tree, Figure 2b is the pruned startup tree;

[0070] Figure 3 Configure the diagram for the spark group;

[0071] Figure 4 This is a screenshot of the monitoring information when the spark group is normal;

[0072] Figure 5 This is a diagram of WeChat warning information. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0074] The present invention proposes a universal application management model, which includes management objects, dependency relationships, management behaviors and hierarchical management.

[0075] 1. Management Objects in the Application Management Model

[0076] In a Linux stand-alone environment, the object of application management is the process. However, in a cluster or distributed environment, in addition to the process, it is also necessary to consider the running host, running mode, and dependencies between processes. In order to facilitate management, the management objects in the cluster and distributed environment are abstracted and encapsulated into three types: application, application group, and application system. Only these three objects are operated during use, and other details are shielded.

[0077] 1. Application

[0078] Programs running on one or more hosts with the same startup command are called applications. They can be one program or multiple programs placed in a startup script. After these programs are started, they stay in memory and provide continuous services instead of exiting after execution. Applications are the smallest management objects and do not need to be further subdivided. The information encapsulated in applications mainly includes application name, running host, running mode, startup program, stop program, process monitoring items, running user, etc.

[0079] 2. Application Group

[0080] Multiple applications that implement a function are combined according to certain rules and managed as a management object. This combination is called an application group. For the convenience of management, application groups are divided according to functions, and multiple applications that implement a certain business function are divided into the same group.

[0081] The order in which applications appear in an application group determines the order in which commands are executed when starting or stopping the application group. This implicit order relationship is called intra-group dependency.

[0082] 3. Application system

[0083] For large systems, many application groups will be generated after being divided according to functions. Multiple application groups are combined according to certain rules and managed in the form of a management object. This combination is called an application system.

[0084] In a cluster or distributed environment, applications run on multiple computers, and the multiple hosts running a certain application are called a host list. According to the application running mode in Linux cluster and distributed environment, the application running mode is divided into four types: active-standby mode, parallel mode, resource pool mode, and mutually exclusive mode. And according to the running mode, applications are divided into active-standby applications, parallel applications, resource pool applications, and mutually exclusive applications.

[0085] (1) Active / standby application: refers to an application that can only run on one of the hosts in the host list, and the other hosts are used as cold standby. For example, MySQL runs in active / standby mode. Active / standby applications can set floating IP addresses, which are only bound to the server where the application is activated.

[0086] (2) Parallel applications: An application runs on all hosts in the host list, such as Cassandra and Ceph service processes.

[0087] (3) Resource pool application: The host list is a resource pool. A task runs on any number of hosts in the pool. The number of hosts running the application is less than the number of hosts in the resource pool.

[0088] (4) Mutually exclusive applications: Only one of two different application groups can be started at the same time. If one application group is started, the other must be stopped. Mutually exclusive applications only support application groups. If two applications in an application group need to run in a mutually exclusive manner, they are split into two application groups.

[0089] According to the application startup method in a cluster or distributed environment, the application startup methods are divided into the following three types.

[0090] (1) Parallel startup of applications on multiple hosts: The application is started in parallel from multiple hosts in the host list. Candidate startup hosts are obtained from the host list in sequence. This is suitable for parallel applications and resource pool applications.

[0091] (2) Application startup in a balanced manner: The application is started in parallel from multiple hosts. Candidate startup hosts are automatically selected from the host list after being sorted by CPU load. This is suitable for resource pool mode applications.

[0092] (3) Single-host startup application: The application runs on multiple hosts, but the startup command only needs to be executed on one host. In this case, the candidate startup host selects the first available host from the host list. This is suitable for parallel applications or resource pool applications with built-in single-point startup function.

[0093] 2. Dependencies in the Application Management Model

[0094] The dependencies in the model are used to describe the startup sequence between different applications. For example, in the Linux service management system systemd, six dependencies are defined, including bindsTo, requires, requisite, wants, part of, conflicts, and two sequence relationships, including after and before. In Veritas InfoScaleAvailability, a dependency called requires is defined, which is a strong dependency. When the upstream application fails to start, the downstream application will not start. All the dependencies defined in the present invention are weak dependencies, that is, if the upstream application fails to start, the subsequent startup operations will continue to be executed. The dependencies are divided into:

[0095] 1. Dependencies within an application group

[0096] The startup order relationship of different applications within an application group is called the intra-group dependency. The intra-group dependency clarifies the startup and stop order of multiple applications in the group. This dependency is implicit and does not need to be declared. It is determined by the order of applications in the group. When starting, the applications are started in order from front to back, and when stopping, they are stopped in order from back to front.

[0097] 2. Dependencies between application groups

[0098] Complex businesses usually have multiple application groups, which may be interrelated. When starting or restarting, they must be executed in a certain order to ensure service availability. This relationship is called inter-group dependency. Inter-group dependency includes implicit inter-group dependency and explicit inter-group dependency.

[0099] In an application system, the order in which application groups appear is called implicit inter-group dependency. This dependency does not need to be configured and is implicit in the order of application groups. When starting, stopping, and restarting an application system, only implicit dependencies are used. The dependency configured in the application group is called explicit inter-group dependency. This relationship is used when starting or restarting a single or multiple application groups. Each group can be configured with three explicit inter-group dependencies:

[0100] (1) Startup group first: The upstream application group that must be started before the current application group is started. One application group can be configured with multiple upstream application groups.

[0101] (2) Post-restart group: When the current application group is restarted, the downstream application groups that need to be restarted can support multiple ones.

[0102] (3) Mutually exclusive group: When this application group is started, another application group must be stopped. This configuration must be in pairs. For example, if A and B are mutually exclusive groups, then B must be configured in A and A must be configured in B.

[0103] 3. Management Behavior of Application Management Model

[0104] In a cluster or distributed environment, the management objects include applications, application groups, and application systems. The management behaviors for these objects can be summarized as application monitoring, system monitoring, application control, and operation and maintenance tools.

[0105] 1. Application monitoring

[0106] Application monitoring is the basis of application management, and mainly evaluates whether the host, application, and application group are running normally. In a cluster or distributed environment, the application status is evaluated from three aspects: operation mode, process, and number of processes. In addition to providing monitoring functions, application monitoring must give a qualitative evaluation value for each host, application, and application group to determine whether it is normal.

[0107] 2. System monitoring

[0108] Provides host CPU, memory, disk space, disk IONODE space and other usage monitoring functions, and can set monitoring alarm thresholds. In disk space monitoring, you can set exclusion items so that the shared disk in the cluster is only monitored on one host, and other hosts are not monitored, reducing the amount of monitoring information output.

[0109] 3. Application Control

[0110] Application control implements the operation functions of applications in a cluster or distributed environment, mainly including start, stop, forced stop, restart, active / standby switching, guarding, monitoring, load balancing and other functions.

[0111] (1) Startup: Startup includes two modes: regular startup and smart startup. Regular startup does not detect explicit inter-group dependencies and directly starts the application group. Smart startup generates a startup tree for the application groups involved according to the explicit inter-group dependencies and starts them in order from the top of the tree to the root. The startup command has the function of preventing repeated startup. For applications that have already been started, the startup command is automatically ignored.

[0112] (2) Stop: You can stop a running application, application group, or application system.

[0113] (3) Forced stop: First execute the stop operation, then retrieve the process according to the process monitoring item as the root user, and execute the kill -9 operation. This is mainly used to solve the problem that some built-in programs cannot normally close the application.

[0114] (4) Restart: There are three types of restart: regular restart, smart restart, and safe restart. Regular restart does not detect explicit inter-group dependencies and directly executes the restart operation. Regular restart uses batch parallel restart. Smart restart automatically restarts dependent application groups according to explicit inter-group dependencies. Safe restart means restarting parallel or resource pool applications one by one. Only after the application on the previous server is completely restarted can the application be started on the next host. During the restart, the service provided by the application remains uninterrupted.

[0115] (5) Active / standby switching: For active / standby mode applications, the currently active host is switched to the specified host or the next available host.

[0116] (6) Guardian: It runs as a daemon process of an application, providing guard functions for an application or an application group. When an application fails, it automatically restarts to achieve high availability of the application.

[0117] (7) Monitoring: Run in a scheduled mode, periodically monitor the running status of the application, generate monitoring logs, and restart the application for applications that are running abnormally and are set to restart automatically.

[0118] (8) Load balancing: In resource pool mode, running applications are rebalanced according to the CPU load of the hosts in the resource pool, and heavily loaded applications are migrated to hosts with lighter CPU loads.

[0119] 4. Operation and maintenance tools

[0120] Operation and maintenance tools are frequently used in daily operation and maintenance in cluster or distributed environments. They include the following functions:

[0121] (1) Remote process viewing: View detailed process information on all hosts according to monitoring items.

[0122] (2) Remote command execution: Execute the input command on the host where an application is located.

[0123] (3) Application backup and restore: Back up and restore application software on the host.

[0124] (4) Application cloning: Copy an application from one host to another host, keeping the permissions and operating environment unchanged.

[0125] (5)crontab backup and restore: backup and restore the scheduled tasks on the host.

[0126] (6) SSH mutual trust tool: Establish SSH mutual trust between multiple hosts, including creating a new SSH mutual trust cluster, adding hosts to a cluster, merging two SSH mutual trust clusters, testing, and repairing SSH login errors.

[0127] (7) Configuration conversion: converting the application configuration file of Veritas InfoScale into the application group defined in the present invention.

[0128] 4. Hierarchical management of application management models

[0129] In a large-scale cluster or distributed environment, there are many hosts and applications. Different application systems may be maintained by different personnel. If a management system is used, the configuration information is too centralized and multiple people can easily cause confusion. At the same time, a management system requires the establishment of SSH mutual trust between hosts. If the number of servers is large, this task is heavy and there are certain security risks.

[0130] Hierarchical management is to divide a large system into two levels according to its functions. The first level is the basic application system, which is responsible for implementing all management functions mentioned in the model. The second level is the advanced management system, which is responsible for managing the primary application management systems deployed on each subsystem and providing a single point management function for the entire system. After adopting hierarchical management, it is possible to achieve single-point centralized management of all applications, local and professional management of each subsystem's configuration information, and isolate faults between different systems. A subsystem configuration error will not affect other systems.

[0131] The above is the management model proposed by the present invention. Based on the management model, the present invention uses bash language to implement an application management system. The design principles of the management system include:

[0132] First, no dependency: the system is developed in bash language, does not rely on any third-party modules or middleware, and can run directly on Linux machines.

[0133] Second, agentless: The system adopts agentless mode. The software only needs to be deployed on one machine, and no agent needs to be installed on the managed machine.

[0134] Third, the combination of parallel and serial: the monitoring module adopts parallel mode to improve the execution efficiency of monitoring commands, and the control module adopts serial or parallel mode according to the application startup mode to ensure that the task is started correctly.

[0135] Fourth, low overhead: monitoring behavior should minimize interference with the monitored machines and reduce the additional overhead brought to the managed machines.

[0136] This application management system adopts a hierarchical management model, which consists of a super management system and an application management system. The super management system manages multiple application management systems, and the application management system manages all applications in an application system. The application management system consists of modules such as configuration management, application control, application monitoring, system monitoring, operation and maintenance tools, logs, and displays. The system architecture is shown in the attached figure. Figure 1 The main technologies and algorithms used in the application control, application monitoring, system monitoring, and operation and maintenance tool modules of the system include: application status intelligent evaluation technology, parallel technology, cache acceleration technology, application cloning technology, multi-tree pruning and layering algorithm, and resource pool load balancing scheduling algorithm.

[0137] 1. Application status intelligent assessment technology

[0138] Accurately monitoring the application status is the basis of application management. General monitoring is based on processes, such as Veritas InfoScale. When a process exists on a host, the status is online, and if it does not exist, it is offline. However, due to different application operation modes, an offline host is not necessarily abnormal, such as a standby machine of an application in the master-slave mode or an unscheduled host in the resource pool mode. At the same time, an online host is not necessarily normal, such as more than two hosts in the master-slave mode are online. Therefore, the present invention proposes a method for evaluating the operation status of hosts and applications in combination with operation modes, processes, and the number of processes, and provides a qualitative evaluation result of whether each host and each application is normal.

[0139] (1) Process monitoring

[0140] The lowest level of monitoring is process monitoring. Process monitoring parameters are defined by monitoring items. The monitoring items of an application consist of two items: process keywords and process quantity. A process keyword is a string that can uniquely identify a process. It is extracted from the output of the ps or dockerps command. For processes that cannot be retrieved with a single keyword, a composite keyword can be used. Composite keywords are divided into AND keywords and non-keywords according to the logical operation relationship. AND keywords "key1-key2" means that key1 and key2 are included, and non-keywords "key1-key2" means that key1 is included but key2 is not included. The number of processes indicates the number of processes in normal operation. The expression methods are "=n", ">=n", and range. "=n" means that when the number of processes is n, it is normal, and other numbers are treated as abnormal; ">=n" means that when the number of processes is at least n, it is normal, and when it is less than n, it is treated as abnormal; the range can indicate the minimum and maximum number of processes in normal operation. If an application has multiple processes, multiple process monitoring items can be configured.

[0141] According to the number of processes corresponding to each monitoring item in the host, the process status is evaluated as 0 (normal), 1 (partially abnormal), 2 (not started), and 3 (ssh abnormal). The formula is as follows:

[0142]

[0143] Where i is the monitoring item number, ni is the number of processes of the ith monitoring item, Pi_min is the lower limit of the number of processes, Pi_max is the upper limit of the number of processes, and Pi represents the status of the ith monitoring item.

[0144] (2) Host status assessment

[0145] The host status is evaluated twice. The first is a pre-evaluation. According to the status of the application process on the host, the single host status is pre-evaluated into four types: 0 (normal), 1 (partially abnormal), 2 (not started), and 3 (ssh abnormal). The formula is as follows:

[0146]

[0147] Where N represents all monitoring items, i is the monitoring item number, Pi is the status of the i-th monitoring item, and Hj is the first evaluation result of the j-th host;

[0148] The second time is to revise the pre-assessment results based on the pre-assessment and in combination with the application operation mode. After the revision, the assessment status includes six states: normal, partial abnormal, not started abnormal, ssh abnormal, violation of operation mode, and idle. The second assessment rules are shown in Table 1.

[0149] Table 1 Host status second evaluation rules

[0150] Table 1 The second evaluation rule of host state

[0151]

[0152] (3) Application status assessment

[0153] Based on the host status evaluation results, the application is evaluated, and the evaluation results are normal, abnormal, and partially abnormal. The evaluation rules are as follows:

[0154] Active / standby mode application: if only one host is in normal state and the others are not started, the application is normal; if all hosts are not started, the application is abnormal; if more than two hosts are normal or one is abnormal or partially abnormal, the application is partially abnormal.

[0155] Parallel mode application: If all hosts are normal, the application status is normal; if all hosts are abnormal, the application status is abnormal; if there is a host that is not running or has an abnormal or partially abnormal host, the application status is partially abnormal.

[0156] Resource pool mode application: If the number of normal hosts meets the minimum and maximum number of running hosts, and there are no abnormal or some abnormal hosts, the application status is normal; if all hosts are not started, the application status is abnormal; if there are hosts with abnormal, partially abnormal or violating the running mode, the application status is partially abnormal.

[0157] The above-mentioned process monitoring, host status assessment, and application status assessment are progressive. The status of the latter is obtained based on the previous ones. The output monitoring results include these three statuses, which can accurately locate faults.

[0158] 2. Parallel design

[0159] Parallelism between application systems: When executing monitoring commands in the hierarchical management mode, the super manager uses background multi-process mode to call the application management system on each management host, and monitoring on different subsystems is executed in parallel.

[0160] Inter-group parallelism: When the input application group contains multiple groups, parallelization technology is adopted to start a process for each application group and execute monitoring commands in parallel.

[0161] Application parallelism: When an application includes multiple hosts, monitoring commands are submitted to multiple hosts in parallel.

[0162] Monitoring command parallelism: The monitoring of multiple applications on a server is run in parallel, and multiple monitoring commands are executed simultaneously.

[0163] 3. Cache acceleration technology

[0164] This management system adopts agentless mode, using ssh to actively connect to the remote host for corresponding operations. When there are many applications on a host, multiple ssh logins to the remote host are required, which has a great impact on the host. The use of cache technology can reduce the number of ssh logins, reduce the impact on the host, and improve monitoring performance.

[0165] When an application group contains multiple applications, before the command executes the monitoring command, the monitoring commands of different applications in the application group are first merged by host. When the monitoring command of the first application is executed, all information is returned and cached in the memory. When the monitoring commands of other applications in the group are executed, the results of the first execution cache are directly used, and there is no need to repeatedly execute the ssh command. This technology can greatly improve the execution speed of the monitoring command.

[0166] 4. Multi-tree pruning and layering algorithm

[0167] The function of the multi-branch tree pruning and layering algorithm is to convert the dependency relationship into a non-repeating layered multi-branch tree with the current application group as the root. This algorithm first uses the pre-order traversal method to search each node, and automatically prunes and moves duplicate nodes. At the same time, the tree is dynamically layered while traversing. A single traversal can generate a layered non-repeating multi-branch tree. Finally, the tree is converted into a sequential structure based on the tree layer information.

[0168] For example, Figure 2a It is the initial startup tree generated according to the dependency relationship of application group A. According to the dependency agreement, the child node must be started before the parent node. There is no strict startup order between brothers. By default, they are started from left to right. Figure 2a In the example, application group E appears twice in different layers, and application group F appears twice in the same layer. Figure 2b 6-layer multi-fork number without repetition.

[0169] The rule for pruning duplicate nodes is: for duplicate nodes at different levels, prune the subtrees at the lower level and connect them to the higher level. Figure 2a The subtree with E as the root under node B is moved to node G. When nodes at the same level are repeated, the right node is retained and the left node is deleted. If the left node has child nodes, its child nodes are merged into the child nodes of the right node. In this example, Figure 2a Node F is directly deleted. Figure 2b The startup order of the generated application A is: (L)->(J, K)->(E)->(H, G, F)->(B, C, D)->(A).

[0170] 5. Application cloning

[0171] The application cloning of this system can copy the application software, user information, firewall, system services, kernel parameters, etc. on one host to another host with the same operating system. It specifically includes two steps: (1) Source host information extraction: Extract the application software information, user information, user group, sub-group information, firewall, system services, limit parameters, kernel parameters, NAS mount information in fstab, etc. on the source host. (2) Application reconstruction: Reconstruct the application on the target host based on the information extracted from the source host. After reconstruction, the application software and operating environment on the target host are the same as those on the target host.

[0172] 6. Resource pool load balancing scheduling algorithm

[0173] Resource pool load balancing scheduling is used to schedule an application according to the host CPU load, and to load or migrate the application to different hosts reasonably to prevent individual hosts from being overloaded. The migration of the present invention refers to stopping the application on one host and starting it on another host. Its scheduling strategy is as follows: The CPU load priority principle is used for the initial scheduling. When the resource pool application is started for the first time, the application is automatically loaded on the host with the lightest CPU load. In the later load balancing scheduling, the double majority principle is used. In order to avoid frequent scheduling or scheduling between hosts with little difference in CPU load, this system only migrates the running hosts whose CPU load ranking (from small to large) exceeds more than half of the resource pool hosts when performing load balancing. For example, if there are 10 hosts in the resource pool, if the CPU load ranking of the host running the application is 4, it will not be migrated; at most only half of the running hosts will be migrated each time.

[0174] Veritas InfoScale Availability is a widely used cluster management system that provides a high availability solution for applications and implements operations such as starting, stopping, monitoring, and fault recovery for applications in the cluster. The application management system (referred to as app_manage) of the present invention can implement the application management function of InfoScale, and can also implement the functions that do not exist in it, making up for its shortcomings. The comparison of the functions of the two is shown in Table 2, where "yes" in the table means support or availability, and "no" means no support or absence.

[0175] Table 2 Functional comparison between application management system and InfoScale

[0176] Table2 Function comparison between application management system andInfoScale

[0177]

[0178] Example

[0179] In order to further illustrate the functions of the model and application system proposed in the present invention, the big data cloud platform of Gansu Meteorological Bureau is taken as the background, and the application management system proposed in the present invention is used to manage the big data cloud platform of Gansu Meteorological Bureau.

[0180] 1. Operating environment

[0181] The big data cloud platform of Gansu Meteorological Bureau has 147 Unix and Linux hosts, which are divided into 12 independent application systems according to their functions. The 12 application systems establish root user SSH mutual trust, select 2 hosts as management hosts for each system, deploy the application management system, and form a super management cluster with the management hosts of each system. Select any one of them as the super management host to deploy the application super management system. There are 85 non-repetitive application groups, 367 applications, and 517 process monitoring items in the 12 application systems. The system division is shown in Table 3. Table 4 is a summary of several typical application group configurations, including multiple scenarios such as master-slave applications, parallel applications, resource pool applications, and mutually exclusive applications supported by this system. The " / " in the table indicates that the application does not have this configuration.

[0182] Table 3 Application system of Gansu meteorological big data cloud platform

[0183] Table 3 Gansu province meteorological big data Cloud Platform application system

[0184]

[0185] Table 4 Configuration information of typical application groups of meteorological big data cloud platform

[0186] Table 4 Configuration information of typical application group of meteorological big data cloud platform

[0187]

[0188] Take the spark application group as an example, the configuration is as follows Figure 3 As shown in the figure, it includes two applications, master and slave. The master application runs in active-standby mode and can run on any host in the host list and provide virtual IP address services. The slave application runs in resource pool mode and runs on any two hosts except the master. Figure 4 It is the monitoring information output by the monitoring command when spark is running normally.

[0189] 2. Use of application management system

[0190] (1) Start and stop the application: use app_manage.sh- <start|stop|restart|safe_restart|switch

[0191] |rebalance><application group|application>[host] command, you can manually start, stop, restart, safely restart, switch between active and standby, and load balance application groups and applications.

[0192] (2) Manual monitoring function usage: Execute supper_manage.sh all–stateany on the super management host to complete the application and system status check on 147 machines within 8 seconds. Adding more hosts will not increase the execution time.

[0193] (3) Use of automatic monitoring mode: Add a monitoring and alarm script alarm.sh to the crontab on the super management host. The script executes the supper_manage.sh all-monitor supper / monitor command every ten minutes, sends abnormal information to the monitoring personnel through enterprise WeChat, and automatically restarts abnormal applications with the AutoRestart attribute. The WeChat alarm policy is: if an abnormal application is detected, an alarm message will be output, and a normal message will be sent once in the first 10 minutes of fault recovery; if there is no fault, a message about all normal applications will be sent once an hour during the day, and it will be silent at night without sending any information. The WeChat alarm information can be found at Figure 5 .

[0194] (3) Daemon mode: app_manage.sh –daemon <[group.application]>, starts the daemon function for the specified application and automatically restarts the application when an abnormality occurs.

[0195] (4) Application cloning: Use the app_manage.sh -clone [group.application] <source host> <target host> command to clone the application on the source host to the target host to package and release the application.

[0196] (5) Remote command execution: app_manage.sh-exec [group.application] “ls –l / home / a.txt”, executes the ls command on the host in the host list of all hosts of the specified application.

[0197] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A Linux large-scale cluster application management system, which is implemented based on a general application management model and is characterized by: The system adopts a hierarchical management mode, which consists of a first-level application management system and a second-level super management system. The first-level application management system is set on each subsystem to realize the management function of the general application management model; the second-level super management system is used to manage the first-level application management system deployed on each subsystem; Super management system, used to manage multiple application management systems; Application management system, used to manage all applications within the application system; The general application management model includes application management objects, dependencies between application management objects, and management behaviors of application management objects; Application management objects, including applications, application groups, and application systems; Dependency, which is used to describe the startup order between different applications; Management behavior, which is used to describe various operation behaviors for different management objects; The applications are divided into primary and standby applications, parallel applications, resource pool applications and mutually exclusive applications according to the operation mode; Active / standby applications can only run on one of the hosts in the host list; Parallel application, running on all hosts in the host list; Resource pool applications run on any number of hosts in the resource pool; Mutually exclusive applications only support application groups. Only one of two different application groups can be started at the same time. If one application group is started, the other application group must be stopped. The startup method of the application includes: Multi-host parallel startup: multiple hosts are started in parallel from the host list. Candidate startup hosts are obtained from the host list in sequence. This is suitable for parallel applications and resource pool applications. Balanced startup mode: Start multiple hosts in parallel. Candidate startup hosts are automatically selected from the host list after being sorted by CPU load. This mode is suitable for resource pool mode applications. Single host startup: runs on multiple hosts. The startup command is executed on only one host. The candidate startup host selects the first available host from the host list. It is suitable for parallel applications or resource pool applications with built-in single-point startup function. The dependency relationship is divided into intra-application group dependency relationship and inter-application group dependency relationship; The dependency within an application group is used to describe the start and stop order of multiple applications within an application group. It is determined by the order of the applications in the group and is an implicit dependency that does not need to be declared. Inter-application group dependencies include implicit inter-group dependencies and explicit inter-group dependencies; the implicit inter-group dependencies are used to describe the startup order of application groups in an application system, do not need to be configured, and are implicit in the order of application groups in the application system; explicit inter-group dependencies are used to describe the startup order between any application groups and are configurable in the application group; The management behaviors include system monitoring, application monitoring, application control and operation and maintenance tools; System monitoring, used to monitor the host CPU, memory, disk space, and disk IONODE space usage, and can set monitoring alarm thresholds and exclusion items; Application monitoring is used to monitor whether the host, application, and application group are running normally, evaluate the application status from the perspective of running mode, process, and number of processes, and make qualitative judgments on each host, application, and application group; Application control is used to implement application operations in a cluster or distributed environment. The operations include starting, stopping, forced stopping, restarting, active / standby switching, load balancing, and monitoring. The control objects are application systems, application groups, and applications. Operation and maintenance tools, used to implement the operation and maintenance of applications in cluster or distributed environments, including remote process viewing, remote command execution, application backup and restore, application cloning, crontab backup and restore, ssh mutual trust tools, and VCS configuration conversion; The application monitoring uses application status intelligent evaluation technology to evaluate the host and application running status, which includes process monitoring, host status evaluation and application status evaluation; Process monitoring is used to monitor the number of processes in the application by setting monitoring items, and evaluate the process status as 0-normal, 1-partially abnormal, 2-not started, 3-ssh abnormal according to the number of processes corresponding to each monitoring item in the host. The evaluation formula is: Where i is the monitoring item number, ni is the number of processes of the ith monitoring item, Pi_min is the lower limit of the number of processes, Pi_max is the upper limit of the number of processes, and Pi represents the status of the ith monitoring item; The host status assessment adopts a two-step assessment method. The first step is pre-assessment and the second step is assessment correction. During the pre-assessment, the pre-assessment result of a single host status is divided into 0-normal, 1-partially abnormal, 2-not started, and 3-ssh abnormal according to the status of the application process on the host. The pre-assessment formula is: Where N represents all monitoring items, i is the monitoring item number, Pi is the status of the i-th monitoring item, and Hj is the first evaluation result of the j-th host; Assessment correction: the pre-assessment results are corrected in combination with the application mode. The corrected assessment results are divided into normal, partially abnormal, not started abnormal, ssh abnormal, violation of operation mode, and idle; Application status assessment evaluates active-standby mode applications, parallel mode applications, and resource pool mode applications based on the host status assessment results. The assessment results are divided into three states: normal, abnormal, and partially abnormal.

2. The Linux large-scale cluster application management system according to claim 1, characterized in that: The application management system includes a configuration management module, an application control module, an application monitoring module, a system monitoring module, an operation and maintenance tool module, a log module, and a display module; Configuration management module, used to configure parameters in the system; Application monitoring module, used to monitor the application status in the system; Application control module, used to start, stop and restart each application; System monitoring module, used to monitor the usage of host CPU, memory, disk space, and disk IONODE space; The operation and maintenance tool module is used to provide multiple maintenance tools to realize remote process viewing, remote command execution, application backup and restore, application cloning, crontab backup and restore, establish ssh mutual trust between multiple hosts, and configuration conversion; A log module, used to record the operations performed by the application management system to form an operation log; The display module is used to display the operation results.

3. The Linux large-scale cluster application management system according to claim 1, characterized in that: The application runs on one or more hosts, has the same startup command, resides in memory after startup, and provides continuous services; The application group is composed of multiple applications that implement a function according to certain rules; The application system is composed of multiple application groups combined according to certain rules.

4. The Linux large-scale cluster application management system according to claim 1, characterized in that: Configurable explicit inter-group dependencies include: start-first group, restart-last group, and mutually exclusive group; Start first group, used to set the upstream application group that must be started first when this application group starts. One application group supports configuring multiple upstream application groups; Post-restart group: used to set the downstream application groups that need to be restarted when the current application group is restarted. Multiple groups can be supported. A mutually exclusive group is used to set another application group that must be stopped when this application group is started.

Images