Server resource collection method and device oriented to complex network scene

Abstract

The invention discloses a server resource collection method and device oriented to a complex network scene, and relates to the technical field of computer networks and data collection. Comprising the following steps of: 1, performing scanning operation on resource information of a server according to a preset time interval on the basis of a Gopoptimization tool, acquiring various resource information of the server, and preprocessing the various resource information; the method comprises the following steps: step 1, transmitting resource information acquired by each server to a data center through WebSocket network communication connection by adopting a corresponding strategy, and step 2, summarizing and sorting all resource information from each server through the data center of a main server, and carrying out unified management and coordination processing on the resource information. And 4, performing persistent storage on the resource information of the server by using a multi-mode database KaiwuDB, wherein the resource information is used for subsequent query, analysis and other related applications.

Description

Technical Field

[0001] The present invention discloses a server resource acquisition method and device for complex network scenarios, and relates to the technical field of computer networks and data acquisition. Background Art

[0002] With the rapid development of information technology, server resource management is crucial to the normal operation of enterprises and organizations. Server resources include CPU, memory, disk, network, etc. Effective collection and management of them is the key to ensuring server performance and business continuity. Although existing resource monitoring solutions can monitor server performance indicators in a relatively comprehensive and real-time manner, they are limited in adaptability when facing complex network connection scenarios. On the one hand, the diversity of server operating systems, such as Windows, Linux, MacOS, etc., are widely used. Different operating systems have obvious differences in resource management and data acquisition methods, making it difficult for existing collection methods to be universal across platforms, increasing the complexity and cost of resource collection, and easily leading to inconsistent and incomplete data. On the other hand, the diversity and uncertainty of network connections are also a major problem. In actual complex networks, the network connections between servers vary. Some can access each other through two-way TCP ports, some can only access one-way, and there are even situations where direct connection is completely impossible. The existing single network transmission mode is difficult to adapt to, and data transmission is prone to poor or lost data, affecting the comprehensive monitoring of server resources. Summary of the invention

[0003] Aiming at the problems faced by server resource collection in complex network scenarios, such as differences in operating systems and complex network connections, the present invention provides a server resource collection method and device for complex network scenarios, effectively solving the deficiencies of the prior art and realizing efficient server resource management.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a server resource acquisition method for complex network scenarios, comprising:

[0006] Step 1: Based on the Gopsutil tool, scan the server's resource information at pre-set time intervals, collect various types of server resource information, and pre-process various types of resource information.

[0007] Step 2: Based on the connectivity between the collection server and the main server, adopt corresponding strategies to transmit the resource information collected by each server to the data center through the WebSocket network communication connection.

[0008] Step 3: All resource information from each server is aggregated and sorted through the data center of the main server, and the resource information is uniformly managed and coordinated.

[0009] Step 4: Use the multi-mode database KaiwuDB to persistently store the server's resource information for subsequent query, analysis, and other related applications.

[0010] Furthermore, in step 1 of the server resource collection method for complex network scenarios, various types of resource information of the server are collected, including: CPU information, memory information, network information and disk information. The CPU information includes the total CPU utilization, the number of CPU cores and the utilization of each CPU core. The memory information includes the total memory, used memory, available memory, memory usage, buffer memory, cache memory, the total swap space and the free swap space. The network information includes the network download speed and upload speed. The disk information includes the total disk capacity, the number of disks and the information of each disk.

[0011] Furthermore, in step 1 of the server resource collection method for complex network scenarios, various types of resource information are preprocessed, including: data format preprocessing, the same group of resource information will be separated by the | symbol, and different groups of resource information will be separated by commas.

[0012] Further, in step 2 of the server resource collection method for complex network scenarios, it includes: judging the connectivity between the collection server and the main server, the connectivity includes four types, namely, the TCP ports of the collection server and the main server can access each other, the collection server can access the TCP port of the main server, the main server can access the TCP port of the collection server, the collection server cannot connect to the main server,

[0013] Take corresponding strategies according to the connectivity situation:

[0014] For the first two connection situations, the acquisition server is started in active connection mode. In this mode, the acquisition server actively accesses the TCP port of the main server and establishes a connection.

[0015] For the third connection situation, the acquisition server is started in passive connection mode. In this mode, the main server takes the task of active polling and polls all acquisition servers started in passive connection mode according to preset rules.

[0016] For the fourth case, the acquisition server polls all secondary servers, which are directly connected to the main server. Through continuous attempts, the acquisition server will successfully connect to a secondary server, and then send the resource information of the acquisition server to the secondary server, which will then forward it to the main server.

[0017] The present invention also provides a server resource collection device for complex network scenarios, including a collection module, a network transmission module, a data center module and a data storage module.

[0018] The collection module is based on the Gopsutil tool. It scans the server's resource information at pre-set time intervals, collects various types of server resource information, and pre-processes various types of resource information.

[0019] The network transmission module adopts corresponding strategies according to the connectivity between the collection server and the main server to transmit the resource information collected by each server to the data center through the WebSocket network communication connection.

[0020] The main server collects and organizes all resource information from various servers through the data center module, and uniformly manages and coordinates the resource information.

[0021] The data storage module uses the multi-mode database KaiwuDB to persistently store the server's resource information for subsequent query, analysis, and other related applications.

[0022] Furthermore, the collection module of the server resource collection device for complex network scenarios collects various resource information of the server, including: CPU information, memory information, network information and disk information. The CPU information includes the total CPU utilization, the number of CPU cores and the utilization of each CPU core. The memory information includes the total memory, used memory, available memory, memory usage, buffer memory, cache memory, the total swap space and the free swap space. The network information includes the network download speed and upload speed. The disk information includes the total disk capacity, the number of disks and the information of each disk.

[0023] Furthermore, the acquisition module of the server resource acquisition device for complex network scenarios preprocesses various types of resource information, including: data format preprocessing, the same group of resource information will be separated by the | symbol, and different groups of resource information will be separated by commas.

[0024] Furthermore, the network transmission module of the server resource collection device for complex network scenarios determines the connectivity between the collection server and the main server. The connectivity includes four types, namely, the TCP ports of the collection server and the main server can access each other, the collection server can access the TCP port of the main server, the main server can access the TCP port of the collection server, the collection server cannot be connected to the main server,

[0025] The network transmission module adopts corresponding strategies according to the connectivity situation:

[0026] For the first two connection situations, the acquisition server is started in active connection mode through the network transmission module. In this mode, the acquisition server actively accesses the TCP port of the main server and establishes a connection.

[0027] For the third connection situation, the acquisition server is started in passive connection mode. In this mode, the main server takes the task of active polling through the network transmission module and polls all acquisition servers started in passive connection mode according to preset rules.

[0028] For the fourth case, the acquisition server polls all secondary servers through the network transmission module. The secondary servers are directly connected to the main server. Through continuous attempts, until the acquisition server successfully connects to a secondary server, the resource information of the acquisition server is first sent to the secondary server, and then forwarded to the main server by the secondary server.

[0029] The benefits of the present invention are:

[0030] The present invention collects and preprocesses resource information of servers of various operating systems at intervals, ensures resource information is transmitted to the data center using WebSocket technology according to the multi-level architecture and different connectivity conditions, summarizes and organizes resource information, and uses the multi-mode database KaiwuDB to store data according to data characteristics. The present invention can meet the needs of server resource collection in complex network scenarios, realize cross-platform collection, stable transmission and efficient storage, help operation and maintenance personnel to grasp the server operation status in real time, effectively respond to changes in complex network environments, and improve the efficiency and quality of server resource management. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the application process of the method of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0033] Example 1

[0034] The present invention provides a server resource acquisition method for complex network scenarios, comprising:

[0035] Step 1: Based on the Gopsutil tool, scan the resource information of the server at a preset time interval, collect various resource information of the server, and pre-process the various resource information.

[0036] The collected server resource information includes: CPU information, memory information, network information and disk information. CPU information includes total CPU utilization, number of CPU cores and utilization of each CPU core. Memory information includes total memory, used memory, available memory, memory utilization, buffer memory, cache memory, total swap space and free swap space. Network information includes network download speed and upload speed. Disk information includes total disk capacity, number of disks and information of each disk.

[0037] Preprocess various resource information, including: preprocessing the data format, the same group of resource information will be separated by the | symbol, and different groups of resource information will be separated by commas.

[0038] The same group of information, such as the CPU information format: CPU-0 utilization, CPU-1 utilization and other related indicators, has the format: total CPU utilization | number of CPU cores | CPU-0 utilization | CPU-1 utilization.

[0039] The memory information format is: total memory|used memory|available memory|memory usage|buffer memory|cache memory|total swap space|free swap space.

[0040] Network information format: download speed | upload speed.

[0041] The disk information record format is: used / total capacity, and the complete format of the overall disk information is: total disk capacity|number of disks|disk 1 information|disk 2 information.

[0042] The format examples between different groups are as follows:

[0043] Time, device ID, CPU information, memory information, network information, disk information.

[0044] Step 2: Based on the connectivity between the collection server and the main server, adopt corresponding strategies to transmit the resource information collected by each server to the data center through the WebSocket network communication connection.

[0045] Specifically, it includes: judging the connectivity between the acquisition server and the main server. There are four types of connectivity, namely, the TCP ports of the acquisition server and the main server can access each other, the acquisition server can access the TCP port of the main server, the main server can access the TCP port of the acquisition server, and the acquisition server cannot connect to the main server.

[0046] Take corresponding strategies according to the connectivity situation:

[0047] For the first two connection situations, the acquisition server is started in active connection mode. In this mode, the acquisition server actively accesses the TCP port of the main server and establishes a connection.

[0048] For the third connection situation, the acquisition server is started in passive connection mode. In this mode, the main server takes the task of active polling and polls all acquisition servers started in passive connection mode according to preset rules.

[0049] For the fourth case, the acquisition server polls all secondary servers, which are directly connected to the main server. Through continuous attempts, the acquisition server will successfully connect to a secondary server, and then send the resource information of the acquisition server to the secondary server, which will then forward it to the main server.

[0050] Step 3: All resource information from each server is summarized and organized through the data center of the main server, and the resource information is uniformly managed and coordinated. The data center can more conveniently uniformly manage and coordinate the server resource information in the entire complex network scenario, providing strong support for subsequent storage and analysis operations.

[0051] Step 4: Use the multi-mode database KaiwuDB to persistently store the server's resource information for subsequent query, analysis, and other related applications. The multi-mode database KaiwuDB is selected as the carrier for data storage. For the server information table, since it mainly records relatively static and clearly related data information such as server IP, operating system, kernel, etc., it is more suitable for storage in a relational database, and can better utilize the advantages of relational databases for data management and query. As for the server resource information record, this information mainly comes from the resource information collected by the acquisition module. Its characteristic is that it has a strong time series and is more suitable for storage in a time series database. The time series database can more accurately reflect the changes in resource information over time, thereby providing a more valuable basis for subsequent analysis and decision-making.

[0052] Example 2

[0053] The present invention also provides a server resource collection device for complex network scenarios, including a collection module, a network transmission module, a data center module and a data storage module.

[0054] The collection module is based on the Gopsutil tool. It scans the server's resource information at pre-set time intervals, collects various types of server resource information, and pre-processes various types of resource information.

[0055] The network transmission module adopts corresponding strategies according to the connectivity between the collection server and the main server to transmit the resource information collected by each server to the data center through the WebSocket network communication connection.

[0056] The main server collects and organizes all resource information from various servers through the data center module, and uniformly manages and coordinates the resource information.

[0057] The data storage module uses the multi-mode database KaiwuDB to persistently store the server's resource information for subsequent query, analysis, and other related applications.

[0058] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0059] Similarly, the device of the present invention collects and pre-processes resource information of servers of various operating systems at intervals, and uses WebSocket technology to ensure that resource information is transmitted to the data center according to the multi-level architecture and different connectivity conditions, summarizes and organizes resource information, and uses the multi-mode database KaiwuDB to store it according to data characteristics. The present invention can meet the needs of server resource collection in complex network scenarios, realize cross-platform collection, stable transmission and efficient storage, help operation and maintenance personnel to grasp the server operation status in real time, effectively respond to changes in complex network environments, and improve the efficiency and quality of server resource management.

[0060] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0061] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A server resource acquisition method for complex network scenarios, characterized by include: Step 1: Based on the Gopsutil tool, scan the server's resource information at pre-set time intervals, collect various types of server resource information, and pre-process various types of resource information. Step 2: Based on the connectivity between the collection server and the main server, adopt corresponding strategies to transmit the resource information collected by each server to the data center through the WebSocket network communication connection. Step 3: All resource information from each server is aggregated and sorted through the data center of the main server, and the resource information is uniformly managed and coordinated. Step 4: Use the multi-mode database KaiwuDB to persistently store the server's resource information for subsequent query, analysis, and other related applications.

2. According to the server resource acquisition method for complex network scenarios according to claim 1, it is characterized by In step 1, various resource information of the server is collected, including: CPU information, memory information, network information and disk information. The CPU information includes the total CPU utilization, the number of CPU cores and the utilization of each CPU core. The memory information includes the total memory, used memory, available memory, memory utilization, buffer memory, cache memory, total swap space and free swap space. The network information includes the network download speed and upload speed. The disk information includes the total disk capacity, the number of disks and the information of each disk.

3. According to the server resource acquisition method for complex network scenarios described in claim 1, it is characterized by In step 1, various resource information is preprocessed, including: data format preprocessing, the same group of resource information is separated by | symbol, and different groups of resource information are separated by commas.

4. The server resource acquisition method for complex network scenarios according to claim 1 is characterized in that Step 2 includes: determining the connectivity between the acquisition server and the main server. The connectivity includes four types, namely, the TCP ports of the acquisition server and the main server can access each other, the acquisition server can access the TCP port of the main server, the main server can access the TCP port of the acquisition server, and the acquisition server cannot connect to the main server. Take corresponding strategies according to the connectivity situation: For the first two connection situations, the acquisition server is started in active connection mode. In this mode, the acquisition server actively accesses the TCP port of the main server and establishes a connection. For the third connection situation, the acquisition server is started in passive connection mode. In this mode, the main server takes the task of active polling and polls all acquisition servers started in passive connection mode according to preset rules. For the fourth case, the acquisition server polls all secondary servers, which are directly connected to the main server. Through continuous attempts, the acquisition server will successfully connect to a secondary server, and then send the resource information of the acquisition server to the secondary server, which will then forward it to the main server.

5. A server resource collection device for complex network scenarios, characterized in that It includes acquisition module, network transmission module, data center module and data storage module. The collection module is based on the Gopsutil tool. It scans the server's resource information at pre-set time intervals, collects various types of server resource information, and pre-processes various types of resource information. The network transmission module adopts corresponding strategies according to the connectivity between the collection server and the main server to transmit the resource information collected by each server to the data center through the WebSocket network communication connection. The main server collects and organizes all resource information from various servers through the data center module, and uniformly manages and coordinates the resource information. The data storage module uses the multi-mode database KaiwuDB to persistently store the server's resource information for subsequent query, analysis, and other related applications.

6. The server resource collection device for complex network scenarios according to claim 5, characterized in that The collection module collects various resource information of the server, including: CPU information, memory information, network information and disk information. CPU information includes total CPU utilization, number of CPU cores and utilization of each CPU core. Memory information includes total memory, used memory, available memory, memory utilization, buffer memory, cache memory, total swap space and free swap space. Network information includes network download speed and upload speed. Disk information includes total disk capacity, number of disks and information of each disk.

7. The server resource collection device for complex network scenarios according to claim 1 is characterized in that Acquisition module Preprocess various resource information, including: preprocessing the data format, the same group of resource information will be separated by the | symbol, and different groups of resource information will be separated by commas.

8. The server resource collection device for complex network scenarios according to claim 1 is characterized in that The network transmission module determines the connectivity between the acquisition server and the main server. There are four types of connectivity, namely, the TCP ports of the acquisition server and the main server can access each other, the acquisition server can access the TCP port of the main server, the main server can access the TCP port of the acquisition server, and the acquisition server cannot connect to the main server. The network transmission module adopts corresponding strategies according to the connectivity situation: For the first two connection situations, the acquisition server is started in active connection mode through the network transmission module. In this mode, the acquisition server actively accesses the TCP port of the main server and establishes a connection. For the third connection situation, the acquisition server is started in passive connection mode. In this mode, the main server takes the task of active polling through the network transmission module and polls all acquisition servers started in passive connection mode according to preset rules. For the fourth case, the acquisition server polls all secondary servers through the network transmission module. The secondary servers are directly connected to the main server. Through continuous attempts, until the acquisition server successfully connects to a secondary server, the resource information of the acquisition server is first sent to the secondary server, and then forwarded to the main server by the secondary server.

Images