High-availability Linux server resource information acquisition method

Abstract

The invention discloses a high-availability Linux server resource information collection method which comprises the following steps: S1, carrying out collection task configuration, determining definition of a standard interface, and filling collection content by a user according to an interface specification; s2, cluster configuration is carried out, and collection tasks and dynamic distribution of the collection tasks are managed; s3, executing the acquisition task, and executing the acquisition task according to the rule and the parameter set by the acquisition task configuration module; and S4, storing the collected data, and quickly storing and distributing the collected data. Through cooperative work of acquisition task configuration, cluster configuration, acquisition task execution and acquisition data storage, comprehensive acquisition and effective management of Linux server resource information are realized.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a method for collecting resource information of a high-availability Linux server. Background Art

[0002] Linux servers have advantages such as good stability, reliable integrity, and perfect network functions, so Linux servers have become the mainstream choice for software development. Software deployed and running on Linux servers requires the use of information collection systems to collect changes in server resources during operation, so as to monitor the software service operation status in real time, effectively prevent sudden server failures, solve server problems in a timely manner, and reduce losses.

[0003] Currently, most of the collection systems are independently deployed and run on Linux servers. There are few modules for customizing collection server resources, and most collection systems do not have cluster disaster recovery capabilities. They have problems such as poor flexibility, low availability, and high maintenance costs. Summary of the invention

[0004] In view of the above problems, the present invention provides a method for collecting resource information of a high-availability Linux server.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for collecting resource information of a high-availability Linux server comprises the following steps:

[0007] S1, configure the collection task, establish the definition of the standard interface, and the user fills in the collection content according to the interface specification;

[0008] S2, performs cluster configuration, manages collection tasks, and dynamically allocates collection tasks;

[0009] S3, executing the collection task, executing the collection task according to the rules and parameters set by the collection task configuration module;

[0010] S4, collects data for storage, and quickly stores and distributes the collected data.

[0011] In one possible implementation, the collection task configuration described in S1 establishes the definition of a standard interface, and the user fills in the collection content according to the interface specification, including: setting the collection frequency, the range of the target server, the type of resources to be collected, and the data filtering and preprocessing rules; providing a data reading interface specification; providing a standard task execution strategy configuration interface for each collection task; supporting dynamic customization of extended task execution strategies, and configuring them specifically according to the type of task.

[0012] In one possible implementation, cluster configuration is performed in S2 to manage collection tasks and dynamically allocate collection tasks, including: setting a collection task execution unit as a process for executing the collection task, deploying multiple collection task execution units on one machine to form a small cluster, and when an executor unit fails, the election unit in the cluster configuration finds other collection task execution units in the cluster to wake up.

[0013] In a possible implementation manner, the election unit in the cluster configuration finds other collection task execution units in the cluster to perform the wake-up operation, including:

[0014] Use MySQL database to establish the system executor unit information table and clear the voter information;

[0015] Obtain the executor unit information list in real time through the service monitoring unit. The executor units in the executor unit information list are the executor unit clusters.

[0016] A random algorithm is used to select an executor unit service participating in the cluster and update it to the executor unit information table, indicating that the election is complete;

[0017] The cluster services participating in the election query the election results, and those selected as executor units start the task operation to execute the collection task, while non-executors wait for the next election;

[0018] If the executor service status is abnormal, the service monitoring unit notifies the executor election unit to conduct the next election.

[0019] In a possible implementation manner, the collection task execution described in S3 includes task list maintenance, task start and stop management, and task status monitoring according to the rules and parameters set by the collection task configuration module.

[0020] In a possible implementation manner, the task list maintenance is used to obtain N executable task information according to the acquisition task configuration module, and store the task information and the execution strategy in a container.

[0021] In one possible implementation, the task start and stop management includes task startup and task shutdown, wherein task startup uses efficient multi-threading technology and optimized network communication protocols to apply for an executable thread from Linux, quickly query the collected content in the thread, and return the query results; task shutdown is used to stop the collection task and release related resources.

[0022] In one possible implementation, the task status monitoring is used to maintain and mark the status of the task list in the task start and stop management execution process; the task list is divided into two parts: a list of tasks to be executed with configured execution strategies and a list of tasks that have been executed. When the task starts to execute, the task will be read from the list to be executed, executed by the task executor, and transferred to the list of executed tasks; the running status marking is divided into waiting to be run, starting, and running. Waiting to run is a state of no task execution and is mainly used in the case of non-executors. Starting is a stage where tasks in the list to be executed are started one after another and have not yet been fully completed. Running indicates that all tasks to be run have been fully executed and started.

[0023] The present invention has the following beneficial effects: comprehensive collection and effective management of Linux server resource information are achieved through the coordinated work of collection task configuration, cluster configuration, collection task execution and collection data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flowchart of a method for collecting resource information of a high-availability Linux server according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Reference Figure 1 , shown is a flowchart of a method for collecting resource information of a high-availability Linux server according to an embodiment of the present invention, comprising the following steps:

[0027] S1, configure the collection task, establish the definition of the standard interface, and the user fills in the collection content according to the interface specification to achieve the collection goal;

[0028] S2, performs cluster configuration, manages collection tasks and dynamically allocates them, achieves load balancing and high-concurrency processing, and avoids resource waste in cluster situations;

[0029] S3, executing the collection task, executing the collection task according to the rules and parameters set by the collection task configuration module;

[0030] S4, collects data for storage, and quickly stores and distributes the collected data.

[0031] A highly available Linux server resource information collection method according to another embodiment of the present invention, in S1, collection task configuration is performed, and the definition of the standard interface is established. The user fills in the collection content according to the interface specification, including: setting the collection frequency, the scope of the target server, the resource type to be collected (such as CPU usage, memory occupancy, disk space, etc.), and the filtering and preprocessing rules of the data. Common implementations include: CPU usage reading, GPU usage reading, IO usage reading, network bandwidth statistics, etc. A data reading interface specification is provided. According to the data reading interface specification, the user can customize the increase of the read data content. For example, if we want to collect the CPU usage now, we can temporarily define it as the CpuReader class by implementing the interface Reader, and implement the interface read() method, so that the data reading of the CPU usage can be completed. A standard task execution strategy configuration interface is provided for each collection task. The user can fill in the specific collection task execution strategy according to the interface specification, such as collecting the CPU usage every two minutes, collecting the IO usage once every minute, etc. The configuration of the task execution strategy completes the efficient and accurate collection task and allows resources to be more reasonably allocated. It supports dynamic customization of extended task execution strategies, which are configured according to the type of task. Since this system is mainly used for resource collection, the configurations that can be provided include: number of executions and execution time, and the execution time can further include fixed time intervals, fixed execution time and date, automatic delay of task blocking, etc.

[0032] Another embodiment of the present invention is a high-availability Linux server resource information collection method, in which cluster configuration is performed in S2, and the management of collection tasks and the dynamic allocation of collection tasks include: setting a collection task execution unit as a process for collection task execution and operation, deploying multiple collection task execution units on one machine to form a small cluster, and when an executor unit fails, the election unit in the cluster configuration finds other collection task execution units in the cluster to wake up, thereby ensuring the high availability of the system. In the case of multiple executors on one Linux server and multiple Linux servers with multiple executors, availability coordination is performed to avoid resource waste such as repeated collection, where the executor unit refers to a process running by the collection task execution module.

[0033] The service monitoring and discovery unit is mainly aimed at the management of the task execution module itself in the cluster environment. It is mainly implemented by storing its own registration information in a container and checking the compatibility of this container. The storage container can be zookeeper or Mysql. The mysql database can be used to record the node information of all participating clusters, and establish a heartbeat mechanism with each node to update the service operation status in real time. The executor election unit mainly confirms the only executor of the services participating in the cluster. This module is implemented based on the service monitoring module to ensure the high availability of the system and avoid waste of resources in the cluster state. Furthermore, the election unit in the cluster configuration finds other collection task execution units in the cluster to wake up, including:

[0034] Use MySQL database to establish the system executor unit information table and clear the voter information;

[0035] Obtain the executor unit information list in real time through the service monitoring unit. The executor units in the executor unit information list are the executor unit clusters.

[0036] A random algorithm is used to select an executor unit service participating in the cluster and update it to the executor unit information table, indicating that the election is complete;

[0037] The cluster services participating in the election query the election results, and those selected as executor units start the task operation to execute the collection task, while non-executors wait for the next election;

[0038] If the executor service status is abnormal, the service monitoring unit notifies the executor election unit to conduct the next election.

[0039] A highly available Linux server resource information collection method according to another embodiment of the present invention uses efficient multi-threading technology and optimized network communication protocols to execute collection tasks, ensuring that resource information is quickly and accurately acquired without affecting the normal operation of the server. For example, during the collection process, if a short network delay or busyness of the server is encountered, the module will automatically retry and optimize to ensure the integrity and accuracy of the data. Therefore, the collection task execution described in S3 is performed according to the rules and parameters set by the collection task configuration module, including task list maintenance, task start and stop management, and task status monitoring.

[0040] The task list maintenance is used to obtain N executable task information according to the collection task configuration module, and store the task information and execution strategy in a container. Task start and stop management includes task start and task shutdown. Task start uses efficient multi-threading technology and optimized network communication protocol to apply for an executable thread from Linux, quickly query the collection content in this thread, and return the query results; task shutdown is used to stop the collection task and release related resources. Task status monitoring is used to maintain and mark the status of the task list in the execution process of task start and stop management; the task list is divided into two parts: the list of tasks to be executed with the execution strategy configured and the list of tasks that have been executed. When the task starts to execute, the task will be read from the list to be executed through the task executor and transferred to the list of executed tasks; the running status mark is divided into waiting to run, starting, and running. Waiting to run is a state without task execution, which is mainly used in the case of non-executors. Starting is the stage where the tasks in the list to be executed are started one after another and have not yet been fully completed. Running means that all tasks to be executed have been fully executed and started.

[0041] The data collected will be stored in the data collection and storage, so it is necessary to define the data collection and storage interface specifications, including storage and query. Users can implement them according to their actual Linux server resource information needs. Common implementations include: CPU usage data writing, GPU usage data writing, IO usage data writing, network bandwidth statistics data writing, etc. According to the data writing interface specifications, users can customize and add writing implementation content. For example, the CPU data "CPU data object" is read above. Now to implement data writing, you can temporarily define it as CpuWriter class by implementing the interface Writer, and implement the interface write() method to complete the Cpu usage data writing implementation. In addition, it can also provide a variety of external distribution interfaces, such as RabbitMq message push, Kafka message push, etc., to ensure the timeliness of collected data and facilitate external rapid acquisition of data results.

[0042] The highly available Linux server resource information collection method set up above realizes the comprehensive collection and effective management of Linux server resource information through the collaborative work of collection task configuration, cluster configuration, collection task execution and collection data storage.

[0043] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A high-availability Linux server resource information collection method, characterized in that: The following steps are involved: S1, configure the collection task, establish the definition of the standard interface, and the user fills in the collection content according to the interface specification; S2, performs cluster configuration, manages collection tasks, and dynamically allocates collection tasks; S3, executing the collection task, executing the collection task according to the rules and parameters set by the collection task configuration module; S4, collects data for storage, and quickly stores and distributes the collected data.

2. The high-availability Linux server resource information collection method according to claim 1, characterized in that: The collection task configuration described in S1 establishes the definition of the standard interface, and the user fills in the collection content according to the interface specification, including: setting the collection frequency, the range of the target server, the type of resources to be collected, and the filtering and preprocessing rules of the data; providing the data reading interface specification; providing a standard task execution strategy configuration interface for each collection task; supporting dynamic customization of the extended task execution strategy, and configuring it according to the type of task.

3. The high-availability Linux server resource information collection method according to claim 1, characterized in that: In S2, cluster configuration is performed to manage collection tasks and dynamically allocate collection tasks, including: setting a collection task execution unit as a process for collection task execution, deploying multiple collection task execution units on one machine to form a small cluster, and when an executor unit fails, the election unit in the cluster configuration finds other collection task execution units in the cluster to wake them up.

4. The high-availability Linux server resource information collection method according to claim 3, characterized in that: The work of the election unit in the cluster configuration finding other collection task execution units in the cluster to wake up includes: Use MySQL database to establish the system executor unit information table and clear the voter information; Obtain the executor unit information list in real time through the service monitoring unit. The executor units in the executor unit information list are the executor unit clusters. A random algorithm is used to select an executor unit service participating in the cluster and update it to the executor unit information table, indicating that the election is complete; The cluster services participating in the election query the election results, and those selected as executor units start the task operation to execute the collection task, while non-executors wait for the next election; If the executor service status is abnormal, the service monitoring unit notifies the executor election unit to conduct the next election.

5. The high-availability Linux server resource information collection method according to claim 1, characterized in that: The collection task is executed as described in S3, and the collection task is executed according to the rules and parameters set by the collection task configuration module, including task list maintenance, task start and stop management, and task status monitoring.

6. The high-availability Linux server resource information collection method according to claim 5, characterized in that: The task list maintenance is used to obtain N executable task information according to the acquisition task configuration module, and store the task information and execution strategy in a container, where Mysql table data storage can be used.

7. The high-availability Linux server resource information collection method according to claim 5, characterized in that: The task start and stop management includes task start and task shutdown, wherein task start uses efficient multi-threading technology and optimized network communication protocol to apply for an executable thread from Linux, quickly query the collected content in the thread, and return the query results; task shutdown is used to stop the collection task and release related resources.

8. The high-availability Linux server resource information collection method according to claim 5, characterized in that: The task status monitoring is used to maintain and mark the status of the task list in the task start and stop management execution process; the task list is divided into two parts: a list of tasks to be executed with configured execution strategies and a list of tasks that have been executed. When the task starts to execute, the task will be read from the list to be executed, executed by the task executor, and transferred to the list of executed tasks; the running status marking is divided into waiting to run, starting, and running. Waiting to run is a state of no task execution and is mainly used in the case of non-executors. Starting is a stage where tasks in the list to be executed are started one after another and have not yet been fully completed. Running means that all tasks to be run have been fully executed and started.

Images