Meteorological infrastructure resource pool large-scale heterogeneous host centralized management method and system

Abstract

The invention discloses a centralized management method and system for large-scale heterogeneous hosts of a meteorological infrastructure resource pool, and the method comprises the following steps: (1), obtaining the information of hosts needing batch operation through a configuration file in a command line mode based on a Linux system; (2) decoding the obtained configuration file, loading a single or batch command list needing to be executed according to decoding information, and logging in a target host for execution; step (3), detecting the host with abnormal login and giving an error prompt, and then skipping to continue to execute the next host; and (4) reading standard output and standard error output of the target host terminal to obtain an execution result of each command, classifying and redisplaying the execution result on a remote centralized management console, and recording a log file at the same time. According to the invention, centralized and batch management of large-scale heterogeneous meteorological infrastructure resources can be realized.

Description

Technical Field

[0001] This invention relates to the field of meteorological infrastructure resource management technology. Specifically, it relates to a method and system for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools. Background Technology

[0002] The meteorological infrastructure resource pool supports numerous core meteorological operations and scientific research applications, including the meteorological big data cloud platform, meteorological communication system, meteorological forecasting system, meteorological integrated information monitoring system, and meteorological government affairs system. Currently, the resource pool contains over 3,000 physical servers from more than ten brands and dozens of models, using operating systems such as Red Hat, SUSE, AIX, and multiple Linux distributions including Kylin, UnionTech, and FusionOS. Because each host operates independently, a non-clustered architecture cannot achieve cluster management. Facing a large-scale, distributed, heterogeneous physical host network and the daily influx of business changes and technical support needs, efficient management has become a challenge. Meteorological infrastructure resource management has traditionally relied on system administrators manually logging in and managing each server individually. With the widespread application of distributed and big data technologies such as the meteorological big data cloud platform, the number of servers has surged, and traditional manual management can no longer meet the requirements of large-scale host management.

[0003] Currently, this field offers open-source automated operation and maintenance tools, including Ansible. While powerful, these tools are complex to use, requiring the installation of various modules and the learning of specialized syntax, resulting in high learning and usage costs. Commercial batch job management systems also exist, offering intuitive and user-friendly graphical interfaces. However, their browser-based operation hinders convenient secondary retrieval and processing of Linux command results and lacks flexibility. Therefore, it is necessary to independently develop a lightweight and user-friendly centralized management tool tailored to the characteristics of meteorological operations and the daily management needs of the meteorological basic resource pool. Summary of the Invention

[0004] Therefore, the technical problem to be solved by this invention is to provide a centralized management method and system for large-scale heterogeneous hosts in a meteorological infrastructure resource pool, so as to realize centralized and batch management of large-scale heterogeneous meteorological infrastructure resources. In this invention, heterogeneous refers to the ability to support heterogeneous hosts containing multiple server models and operating system types; centralized batch management means that instead of logging into each host individually, a series of commands are uniformly initiated on the management terminal for a batch of hosts.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A centralized management method for large-scale heterogeneous mainframes in meteorological infrastructure resource pools includes the following steps:

[0007] Step (1): On the Linux system, use a program to obtain the host information that needs to be batch operated through the configuration file via the command line.

[0008] Step (2): Decode the obtained configuration file, load the list of single or batch commands to be executed according to the decoded information, and log in to the target host to execute them;

[0009] Step (3): Detect the host with login failure and give an error message, then skip to the next host;

[0010] Step (4): Read the standard output and standard error output of the target host terminal to obtain the execution results of each command, and display them in the remote centralized management console in categories, while recording them in the log file.

[0011] In the above-mentioned centralized management method for large-scale heterogeneous hosts of meteorological infrastructure resource pool, step (1) includes the host IP address, login account, user login password, SSH port, and hardware device model / operating system type in the configuration file.

[0012] The above-mentioned centralized management method for large-scale heterogeneous hosts in the meteorological infrastructure resource pool uses a single IP address or a range of IP addresses as the host IP address, and the configuration file is in text or Excel format. The configuration file uses base64 encoding to encrypt and store the information.

[0013] In the above-mentioned centralized management method for large-scale heterogeneous hosts of meteorological infrastructure resource pool, in step (2), the format of the loaded command is consistent with the built-in command of Linux system; when the command is executed: for command execution operations, the SSH protocol is used to log in to the target host and execute one or more commands in sequence; for file transfer operations, the SFTP protocol is used to log in to the target host and execute file upload or download operations.

[0014] Furthermore, this invention supports command orchestration and execution of group commands. A host management task often requires many steps to complete. For example, creating a new file system involves creating a hard disk partition -> formatting the partition -> creating a mount point -> mounting the file system -> setting directory ownership and group -> adding auto-start configuration, etc. In addition to supporting the execution of single commands on a batch of hosts, this invention also supports the execution of group commands. Group commands include three scenarios:

[0015] (1) There is no dependency between commands. In this case, the command execution module can be called multiple times.

[0016] (2) There is a dependency relationship between commands, that is, the prerequisite for executing the next command is that the previous command is executed successfully. In this case, it is necessary to first determine the execution status (success or failure) of the previous command, and then continue to execute the next command after confirming that it has been executed successfully.

[0017] (3) Information sharing is required between commands. That is, the execution parameters of the next command need to be determined based on the execution result of the previous command and other auxiliary information. In this case, the previous command needs to be executed successfully and the execution result needs to be obtained. If necessary, auxiliary information in the task file also needs to be read to determine the execution parameters of the next command.

[0018] In the above-mentioned centralized management method for large-scale heterogeneous hosts of meteorological infrastructure resource pool, in step (2), for heterogeneous hardware models / operating system types, the specific command format executed by each host is selected by matching the "device model / operating system type" field in the configuration file.

[0019] In the above-mentioned centralized management method for large-scale heterogeneous hosts of meteorological infrastructure resource pool, in step (2), when logging into the target host, the authentication method of username and password or key is used for login; abnormal login situations include authentication error and login timeout.

[0020] In the above-mentioned centralized management method for large-scale heterogeneous hosts of meteorological infrastructure resource pool, in step (4), the log file is a text or excel format file.

[0021] The aforementioned centralized management method for large-scale heterogeneous hosts in the meteorological infrastructure resource pool also includes the following steps: performing advanced retrieval or further processing on log files to obtain more accurate information.

[0022] A centralized management system for large-scale heterogeneous servers in a meteorological infrastructure resource pool is used to implement the aforementioned centralized management method for large-scale heterogeneous servers in a meteorological infrastructure resource pool.

[0023] It includes a configuration and task loading module, a host connection module, a hardware management module, a command execution module, a complex task execution module, a file transfer module, and a graphics display module;

[0024] The configuration and task loading module is used to encrypt and store host information and parse configuration files and task files.

[0025] The host connection module is used to enable SSH protocol interaction between the management terminal and the managed host, and to log in to the managed host;

[0026] The hardware management module is used to match the corresponding hardware instructions according to the hardware model, and realize hardware status query and hardware device control, including power on, power off and restart;

[0027] The command execution module is used to execute one or more native Linux commands in batches and return the execution results.

[0028] The complex task execution module is used to implement complex system management tasks with sequential logical relationships based on the command execution module and task file information.

[0029] The file transfer module is used to enable batch uploading and downloading of files.

[0030] The graphical display module is used to display system status data graphically, including bar charts, pie charts, and heat maps.

[0031] The meteorological infrastructure resource pool centralized management system for large-scale heterogeneous hosts of this invention supports host management tasks with complex logical judgments, such as logging into thousands of hosts one by one based on a whitelist, comparing the whitelisted users of each host, filtering out non-system-level users not in the whitelist, and performing deletion operations; it supports graphical display to visualize the usage of host resources such as CPU, memory, and system load; and it also supports batch control of server hardware, including server shutdown and restart operations.

[0032] The technical solution of the present invention achieves the following beneficial technical effects:

[0033] 1. This invention presents a centralized batch host management method independently developed based on the Linux system and SSH protocol. It features remote connection to multiple hosts, batch system configuration, batch program deployment, batch command execution, and batch file upload and download. It supports both password and key authentication methods, and secure mechanisms such as encrypted password storage. This invention primarily targets the management of large-scale heterogeneous hosts, focusing on centralized management of server hardware and operating system software (hereinafter referred to as hosts), and is mainly aimed at system administrators. Compared to open-source or commercial software on the market, it has advantages such as lightweight design, full compatibility with Linux commands (no need to learn or memorize additional syntax), native Linux format remote echo for easy secondary processing, and no need for the controlled end to install an agent.

[0034] 2. The present invention provides a centralized management method for large-scale heterogeneous hosts in a meteorological infrastructure resource pool. It adopts a flat management approach, allowing the management terminal to directly access each managed host without the need for intermediate management nodes or mutual trust relationships between the management terminal and the managed hosts. Access between the management terminal and the managed hosts uses username and password authentication. This application system with only a single node does not require mutual trust relationships with other hosts, resulting in higher security. At the same time, the flat management approach also improves efficiency.

[0035] 3. This invention, a centralized management system for large-scale heterogeneous hosts in a meteorological infrastructure resource pool, enables centralized and batch management of large-scale heterogeneous hosts, solving the problems of low efficiency and error-proneness in traditional manual management. This system is the first self-developed centralized batch management software for meteorological infrastructure resource management. It is easy to deploy, simple to use, and consumes few resources. It can manage over 3000 hosts, with only one Linux host required for centralized management and control. Both physical and virtual machines can be used without hardware configuration requirements. Written in native Python, it does not rely on or require the deployment of large software frameworks, is fully compatible with Linux commands, and eliminates the need to learn complex specialized syntax. Similarly, no agent programs need to be installed on the managed hosts. Relying on Python's powerful function libraries, it achieves high development efficiency while enabling more complex functions.

[0036] 3. This invention is a centralized management system for large-scale heterogeneous hosts of meteorological infrastructure resource pools. The software is designed with security mechanisms to ensure the encrypted storage of critical and sensitive information such as accounts and passwords. Furthermore, it is entirely independently developed, controllable, and self-reliant, without dependence on any foreign technology or commercial companies, aligning with the national strategy of encouraging technological self-reliance. Moreover, its functions can be continuously improved and perfected based on work needs and experience, and the script library can be enriched. Functions can be continuously enriched and improved according to new needs and situations arising in business use. Scripts can be customized and developed based on complex application scenarios, enriching and improving the script library. After several years of iteration, the software has become more versatile and robust, covering common business needs in daily work.

[0037] 4. The meteorological infrastructure resource pool large-scale heterogeneous host centralized management system of the present invention can be widely used in the construction of meteorological informatization projects and the daily operation and maintenance of meteorological basic resource pools, improve the quality of project implementation, realize the unification and planning of system configuration, improve the timeliness of fault handling, and enhance the system stability and the level of operation and maintenance automation. Detailed Implementation

[0038] This embodiment of the centralized management method for large-scale heterogeneous mainframes of meteorological infrastructure resource pool includes the following steps:

[0039] Step (1): The program runs on a Linux system and uses the command line to obtain the host information that needs to be operated in batches through the configuration file. The configuration file includes the host IP address, login account, user login password, SSH port and hardware device model / operating system type. The host IP address supports a single IP and IP address range (e.g., 1.1.1.1-1.1.1.12). The configuration file supports text, excel and other formats. The configuration file uses base64 encoding to encrypt and store the information.

[0040] Configuration file format: Contains 7 columns including host IP address, login user, user login password, SSH port, hardware device model, and operating system type. Each line contains one IP address or one IP range. The program reads the configuration file line by line, parses each field, obtains the relevant information required for logging into each host, and stores the decoded information into an array.

[0041] Step (2): The program first reads and decodes the configuration file, loads the list of single or batch commands to be executed based on the decoded information, and logs into the target host to execute them; it obtains the list of login host IPs and the corresponding account, password, SSH port, device type / operating system type, etc., and then loads the list of single or batch commands to be executed. The command format is consistent with the format of Linux built-in commands. Subsequently, for command execution operations, the SSH protocol is used to log into the target host and execute single or multiple commands sequentially; for file transfer operations, the SFTP protocol is used to log into the target host and execute file upload or download operations. Username / password and key authentication methods are supported during login; for heterogeneous hardware models / operating system types, the specific command format to be executed on each host is selected by matching the "device model / operating system type" field in the configuration file.

[0042] Commands are executed using a group command approach: if there are no dependencies between commands, the command execution module is called multiple times; if there are dependencies between commands, the execution status of the previous command is first checked to determine whether it was successful or failed, and the next command is executed only after the previous command is confirmed to have been successfully executed; if information sharing is required between commands, that is, the execution parameters of the next command need to be determined based on the execution result and auxiliary information of the previous command, then it is necessary to confirm the successful execution of the previous command and obtain the execution result, or it is also necessary to read the auxiliary information in the task file to determine the execution parameters of the next command.

[0043] Step (3): Detect the host with login errors (including authentication errors, login timeouts and other login failures) and give an error message, then skip to the next host;

[0044] Step (4): Read the standard output and standard error output of the target host terminal to obtain the execution results of each command, and display them in the remote centralized management console in categories, while recording in the log file; the log file supports text and excel formats. Since the program runs on the command line terminal of the Linux system, the log file can be directly searched or further processed to obtain more accurate information and improve management efficiency.

[0045] The aforementioned centralized management method for large-scale heterogeneous hosts in the meteorological infrastructure resource pool is implemented using a centralized management system for large-scale heterogeneous hosts in the meteorological infrastructure resource pool. This system includes: a configuration and task loading module, a host connection module, a hardware management module, a command execution module, a complex task execution module, a file transfer module, and a graphics display module; wherein:

[0046] Configuration and task loading module: used to encrypt and store host information, parse configuration files and task files, and use base64 encoding to implement encrypted storage of usernames and passwords;

[0047] The host connection module is used to enable SSH protocol interaction between the management terminal and the managed host, log in to the managed host, and realize the connection and authentication between the management terminal and the managed host; it uses Paramiko to implement SSH protocol interaction between the management terminal and the managed host.

[0048] The hardware management module is used to match the corresponding hardware commands according to the hardware model in the configuration file, and realize the viewing and control of hardware device status based on the IPMI protocol, including power on, power off, restart, etc.

[0049] The command execution module is used to execute one or more native Linux commands in batches and return the execution results. For non-interactive commands, that is, commands that do not require human interaction during execution, the command to be executed is passed as a string parameter to the exec()_command() function, which finally returns the standard output and standard error output of the executed command to the management terminal. For interactive commands, such as password input, the invoke_shell() function is used to simulate a shell terminal session to complete the interactive execution of the command.

[0050] The complex task execution module, based on the command execution module and task file information, implements complex system management tasks with sequential logical relationships. It first obtains the current system status, and based on the execution result of the current command, combined with auxiliary information in the task file, determines the execution parameters for subsequent commands. For example, for a task of cleaning up users on a batch of hosts based on a whitelist, it first queries all current user information in the system, obtains the list of users to be retained for each host through the IP column of the task file, compares it with the current user information one by one, filters out usernames that are not on the whitelist and are not system users (UID≥1000), and finally executes the user deletion operation. The process then continues to the next host.

[0051] The file transfer module enables batch file upload and download based on the SFTP protocol.

[0052] The graphics display module uses the Matplotlib plotting library to display system status data graphically in the form of bar charts, pie charts, heatmaps, etc.

[0053] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of the claims of this patent application.

Claims

1. A centralized management method for large-scale heterogeneous mainframes of meteorological infrastructure resource pools, characterized in that: Includes the following steps: Step (1): Based on the Linux system, obtain the host information that needs to be batch operated through the configuration file using the command line method; Step (2): Decode the obtained configuration file, load the list of single or batch commands to be executed according to the decoded information, and log in to the target host to execute them; Step (3): Detect the host with login failure and give an error message, then skip to the next host; Step (4): Read the standard output and standard error output of the target host terminal to obtain the execution results of each command, and display them in the remote centralized management console in categories, while recording them in the log file.

2. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 1, characterized in that, In step (1), the configuration file includes the host IP address, login account, user login password, SSH port, and hardware device model / operating system type.

3. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 2, characterized in that, The host IP address is a single IP or a range of IP addresses. The configuration file is in text or Excel format and uses base64 encoding to encrypt and store the information.

4. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 1, characterized in that, In step (2), the format of the loaded command is consistent with the format of the built-in command of the Linux system; when the command is executed: for command execution operations, the SSH protocol is used to log in to the target host and execute one or more commands in sequence; for file transfer operations, the SFTP protocol is used to log in to the target host and perform file upload or download operations.

5. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 4, characterized in that, In step (2), the command is executed using the group command method: If there are no dependencies between commands, the command execution module can be called multiple times. If there are dependencies between commands, first check whether the execution status of the previous command was successful or failed. Only after confirming that the previous command was successfully executed will the next command be executed. If information sharing is required between commands, that is, if the execution parameters of the next command need to be determined based on the execution result and auxiliary information of the previous command, then it is necessary to confirm that the previous command was successfully executed and obtain the execution result, or it is necessary to read the auxiliary information in the task file to determine the execution parameters of the next command.

6. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 4, characterized in that, In step (2), for heterogeneous hardware models / operating system types, the specific command format executed by each host is selected by matching the "device model / operating system type" field in the configuration file.

7. The method for centralized management of large-scale heterogeneous mainframes of meteorological infrastructure resource pools according to claim 4, characterized in that, In step (2), when logging into the target host, the authentication method of username and password or key is used; abnormal login situations include authentication error and login timeout.

8. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 1, characterized in that, In step (4), the log file is a text or excel format file.

9. The method for centralized management of large-scale heterogeneous mainframes in meteorological infrastructure resource pools according to claim 1, characterized in that, It also includes the following steps: performing advanced retrieval or further processing on the log files to obtain more accurate information.

10. A centralized management system for a large-scale heterogeneous host of meteorological infrastructure resource pools, characterized in that: The method for centralized management of large-scale heterogeneous hosts in meteorological infrastructure resource pools as described in any one of claims 1-9 includes a configuration and task loading module, a host connection module, a hardware management module, a command execution module, a complex task execution module, a file transfer module, and a graphics display module. The configuration and task loading module is used to encrypt and store host information and parse configuration files and task files. The host connection module is used to enable SSH protocol interaction between the management terminal and the managed host, and to log in to the managed host; The hardware management module is used to match the corresponding hardware instructions according to the hardware model, and realize hardware status query and hardware device control, including power on, power off and restart; The command execution module is used to execute one or more native Linux commands in batches and return the execution results. The complex task execution module is used to implement complex system management tasks with sequential logical relationships based on the command execution module and task file information. The file transfer module is used to enable batch uploading and downloading of files. The graphical display module is used to display system status data graphically, including bar charts, pie charts, and heat maps.