Cluster data collection method and apparatus, electronic device, and storage medium

Abstract

The application provides a cluster data collection method and device, electronic equipment and storage medium, relates to the technical field of computers, and is used for adding host attribute information in each host to monitoring data of the corresponding host to obtain target data for reflecting the running condition and host characteristics of each host. The method comprises the following steps: a first server determines to-be-acquired host attribute information, and acquires to-be-acquired host attribute information of each host, wherein one host attribute information is used for reflecting the host characteristics of one host; each host queries the corresponding to-be-acquired host attribute information thereof from the first server, adds the corresponding to-be-acquired host attribute information thereof to host monitoring data of the host, and obtains target data of the host; and a second server collects the target data of each host in a host cluster, wherein the target data of one host is used for reflecting the running condition and host characteristics of the host.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a cluster data collection method and device, electronic equipment and storage medium. BACKGROUND

[0002] In order to ensure normal operation of each host in the host cluster, an operation and maintenance personnel needs to collect monitoring data of each host and monitor the operation of each host. In order to quickly determine an abnormal host when an abnormal situation occurs, the operation and maintenance personnel needs to collect host attribute information of each host, such as host address information and location information of a host room where the host is located, and match the host attribute information of each host with the host monitoring data.

[0003] The prior art can collect the monitoring data of each host by calling a node-exporter collector deployed on each host. In order to obtain the host attribute information matched with the monitoring data of each host, the operation and maintenance personnel needs to write a configuration file for each host, and call a Prometheus system to read the configuration file corresponding to each host, so as to obtain the host attribute information corresponding to each host. However, when the host attribute information to be obtained changes, the operation and maintenance personnel needs to modify the configuration file corresponding to each host one by one, which is a large amount of work and is prone to errors. SUMMARY

[0004] The present application provides a cluster data collection method and device, electronic equipment and storage medium, which are used to add host attribute information in each host to monitoring data of the corresponding host, so as to obtain target data for reflecting the operation of each host and the host characteristics.

[0005] To achieve the above object, the present application adopts the following technical solutions:

[0006] In a first aspect, a cluster data collection method is provided, which is applied to a cluster data collection system. The cluster data collection system includes a host cluster, a first server and a second server. The host cluster includes a plurality of hosts. The first server is in communication connection with each host. The second server is in communication connection with each host. The method includes: the first server determines to-be-obtained host attribute information and obtains the to-be-obtained host attribute information of each host. One host attribute information is used to reflect the host characteristics of one host. Each host queries the to-be-obtained host attribute information corresponding to itself from the first server and adds the to-be-obtained host attribute information corresponding to itself to the host monitoring data of itself, so as to obtain target data of itself. The second server collects the target data of each host in the host cluster. The target data of one host is used to reflect the operation of the host and the host characteristics.

[0007] Optionally, the first server determines the to-be-acquired host attribute information, including: the first server reads the configuration file to obtain an attribute name of the to-be-acquired host attribute information; and the first server acquires the to-be-acquired host attribute information of each host, including: the first server acquires, based on the attribute name of the to-be-acquired host attribute information, the to-be-acquired host attribute information corresponding to each host from a database including a plurality of attribute information of a plurality of hosts.

[0008] Optionally, the method further includes: in a case where a modification operation instruction for the configuration file is received, the first server updates the configuration file in response to the modification operation instruction, reads the updated configuration file to obtain an attribute name of updated to-be-acquired host attribute information, and acquires, based on the attribute name of the updated to-be-acquired host attribute information, the to-be-acquired host attribute information corresponding to each host from a database including a plurality of attribute information of a plurality of hosts.

[0009] Optionally, each host adds the to-be-acquired host attribute information corresponding thereto to host monitoring data corresponding thereto to obtain target data, including: for any one host, the host acquires host monitoring data through a collector pre-deployed on the host; the host monitoring data is used to reflect an operation index of the host; and the host adds the to-be-acquired host attribute information to the host monitoring data through a preset data processing module to obtain the target data.

[0010] Optionally, the second server acquires the target data of each host in the host cluster, including: the second server acquires the target data of each host in the host cluster through a Prometheus system pre-deployed on the second server.

[0011] In a second aspect, a cluster data acquisition system is provided, including a host cluster, a first server, and a second server; the host cluster includes a plurality of hosts, the first server is in communication connection with each host, and the second server is in communication connection with each host; the first server is configured to determine to-be-acquired host attribute information and acquire the to-be-acquired host attribute information of each host; one host attribute information is used to reflect a host feature of one host; the host cluster is configured to query, by each host, the to-be-acquired host attribute information corresponding thereto from the first server, and add the to-be-acquired host attribute information corresponding thereto to host monitoring data corresponding thereto to obtain target data corresponding thereto; and the second server is configured to acquire the target data of each host in the host cluster; the target data of one host is used to reflect an operation condition and a host feature of the one host.

[0012] Optionally, the first server is specifically configured to: read the configuration file to obtain attribute names of the to-be-acquired host attribute information; and acquire the to-be-acquired host attribute information of each host, including: the first server acquires, based on the attribute names of the to-be-acquired host attribute information, the to-be-acquired host attribute information corresponding to each host from a database including a plurality of attribute information of a plurality of hosts.

[0013] Optionally, the first server is further configured to: in a case where a modification operation instruction for the configuration file is received, the first server updates the configuration file in response to the modification operation instruction, and reads the updated configuration file to obtain updated attribute names of the to-be-acquired host attribute information; and the first server acquires, based on the updated attribute names of the to-be-acquired host attribute information, the to-be-acquired host attribute information corresponding to each host from a database including a plurality of attribute information of a plurality of hosts.

[0014] Optionally, the host cluster is specifically configured to: for any one host, the host acquires host monitoring data through a collector pre-deployed on the host; the host monitoring data is used to reflect an operation index of the host; and the host adds the to-be-acquired host attribute information to the host monitoring data through a preset data processing module to obtain target data.

[0015] Optionally, the second server is specifically configured to: collect, through a Prometheus system pre-deployed on the second server, the target data of each host in the host cluster.

[0016] In a third aspect, an electronic device is provided, including: a processor, and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the cluster data acquisition method of the first aspect.

[0017] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores instructions. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the cluster data acquisition method of the first aspect.

[0018] The technical scheme provided by the embodiment of the application brings at least the following beneficial effects: the first server determines the to-be-acquired host attribute information, and acquires the to-be-acquired host attribute information of each host, and one host attribute information is used to reflect the host characteristics of one host; compared with the prior art, the Prometheus system acquires the corresponding host attribute information through the configuration file corresponding to each host, and the application only needs to set one configuration file in the first server, so that the acquisition of the to-be-acquired host attribute information can be completed, which is more convenient; and when the to-be-acquired host attribute information changes, the prior art needs to modify the configuration file corresponding to each host one by one, while the application only needs to respond to the modification operation instruction and update the configuration file in the first server, thereby reducing the workload and making the operation more flexible. Each host queries the corresponding to-be-acquired host attribute information of the first server, adds the corresponding to-be-acquired host attribute information of the first server to the host monitoring data of the first server, obtains the target data of the first server, and then the second server collects the target data of each host in the host cluster, and obtains the running status and host characteristics of each host in the host cluster; compared with the prior art, the host attribute information of each host is added to the monitoring data of the corresponding host in the Prometheus system, and the application completes the operation of adding the host attribute information of each host to the monitoring data of the corresponding host in each host, thereby distributing the processing pressure to the processing service of each host, and reducing the processing pressure of the Prometheus system. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical scheme in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A cluster data acquisition system structure schematic diagram is provided for the embodiment of the application.

[0021] Figure 2 A cluster data acquisition method flowchart is provided for the embodiment of the application. Figure 1 ;

[0022] Figure 3 A to-be-acquired host attribute information acquisition method is provided for the embodiment of the application. Figure 1 ;

[0023] Figure 4 A to-be-acquired host attribute information acquisition method is provided for the embodiment of the application. Figure 2 ;

[0024] Figure 5 A schematic diagram of a target data collection method provided for an embodiment of the present application is shown in FIG. 1.

[0025] Figure 6 A flowchart of a cluster data collection method provided for an embodiment of the present application is shown in FIG. 2. Figure 2

[0026] Figure 7 A flowchart of a cluster data collection method provided for an embodiment of the present application is shown in FIG. 3. Figure 3

[0027] Figure 8 A structural diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0029] It should be noted that in the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean serving as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design schemes. Rather, the words such as "exemplary" or "for example" are intended to present the relevant concept in a specific manner.

[0030] It should also be noted that in the embodiments of the present application, "of", "corresponding" and "corresponding" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent.

[0031] In order to clearly describe the technical solutions in the embodiments of the present application, in the embodiments of the present application, the words such as "first", "second", etc. are used to distinguish the same or similar items with basically the same functions and effects. Those skilled in the art can understand that the words such as "first", "second", etc. are not intended to limit the quantity and execution order.

[0032] Before the embodiments of the present application are explained in detail, some related technical terms and related technologies involved in the embodiments of the present application are introduced.

[0033] ​​The importance of cloud platform to the development of technology is self-evident, and the significance of cloud platform construction is due to the continuous development of economic and social informatization. The government's demand for informatization has also put forward higher and higher requirements, making the stability of the cloud platform very important. In order to ensure the stability of the cloud platform, the operation and maintenance personnel need to monitor the running status of a large number of physical hosts in the cloud platform in real time to ensure the normal operation of the host and the entire platform.

[0034] In the prior art, the running status of a large number of physical hosts in the cloud platform is monitored, the node-exporter collector is used to monitor the running status of the host, the monitoring data of the host is obtained, and the host monitoring data is summarized by the Prometheus system. The Prometheus system is an open source system monitoring and alarm system. In the cloud native monitoring system, the Prometheus system is usually used for monitoring, and a variety of exporters (collectors) are also supported to collect data. The performance of the Prometheus system is sufficient to support a cluster of tens of thousands of servers.

[0035] Among them, the exporter collector is a metric data collection component of the Prometheus system, which is responsible for collecting data from target Jobs (Linux command, used to display the job status of the current session) and converting the collected data into a time series data format supported by the Prometheus system. Unlike traditional metric data collection components, the exporter collector is only responsible for collection and does not send data to the server (Server) end, but waits for the Prometheus Server to actively grab data. The node-exporter collector is used in the present application to collect the running metrics of the Linux system host, including the CPU usage, memory usage, host load, file system usage, network rate and network utilization of the host, and other basic monitoring data.

[0036] In order to perform alarm analysis in the Prometheus system and quickly identify the cause of host anomalies when host anomalies are found, the property information of the host needs to be added to the host monitoring data. In the prior art, the host property information is usually obtained by manually / automatically modifying the Prometheus configuration, and the host property information is added to the host monitoring data.

[0037] The manual modification of the Prometheus configuration to obtain the host attribute information needs to modify the configuration corresponding to each host in turn, which is large in workload and prone to errors; the automatic modification of the Prometheus configuration to obtain the host attribute information cannot correspond the obtained host attribute information to the hosts one by one, that is, the added host attribute information is not fine enough, and the configuration file is too long to read; and when the number of host nodes and the number of monitoring indicators are large, the addition of the host attribute information to the host monitoring data on the Prometheus side has a large pressure on the Prometheus system and affects the performance.

[0038] In view of the above problems, the application provides a cluster data collection method, which is applied to a cluster data collection system. The cluster data collection system includes a host cluster, a first server and a second server. The first server is configured to determine to-be-acquired host attribute information and acquire the to-be-acquired host attribute information of each host. Each host queries the to-be-acquired host attribute information corresponding to the host from the first server at a preset time interval, adds the to-be-acquired host attribute information corresponding to the host to host monitoring data of the host, and obtains target data of the host. After each host in the host cluster obtains the target data of the host, the second server collects the target data of each host in the host cluster.

[0039] The cluster data collection method provided by the embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0040] The cluster data collection method provided by the embodiments of the application can be applied to a cluster data collection system, Figure 1 A structural schematic diagram of the cluster data collection system is shown. As shown in the figure, Figure 1 The cluster data collection system 10 includes a host cluster 101, a first server 102 and a second server 103. The host cluster 101 includes a plurality of hosts, the first server 102 is in communication connection with each host, and the second server 103 is in communication connection with each host.

[0041] The first server 102 is configured to determine to-be-acquired host attribute information and acquire the to-be-acquired host attribute information of each host. The specific acquisition method can be referred to the introduction of the following embodiments, and will not be described here.

[0042] Each host queries the to-be-acquired host attribute information corresponding to the host from the first server 102, adds the to-be-acquired host attribute information corresponding to the host to host monitoring data of the host, and obtains target data of the host. The specific addition method can be referred to the introduction of the following embodiments, and will not be described here.

[0043] The second server 103 collects target data of each host in the host cluster 101, wherein the target data of one host is used to reflect the running condition and the host feature of one host, and the specific collection method can refer to the description of the following embodiments, which will not be repeated here.

[0044] The first server 102 can be a data agent server, and a data agent service is deployed in the first server 102. The configuration file for obtaining the host attribute information can be configured in the data agent service. The specific type of the first server 102 is not limited in the embodiments of the present application.

[0045] The second server 103 can be a single server, and a Prometheus system is deployed in the second server 103. The target data of each host can be collected through the Prometheus system. The specific type of the second server 103 is not limited in the embodiments of the present application.

[0046] Figure 2 is a flowchart of a cluster data collection method according to some example embodiments. In some embodiments, the cluster data collection method described above can be applied to the cluster data collection system as shown in Figure 1 , and can also be applied to other similar devices.

[0047] As shown in Figure 2 , the cluster data collection method provided in the embodiments of the present application includes the following S201-S203.

[0048] S201, the first server determines to obtain host attribute information, and obtains the to-be-obtained host attribute information of each host.

[0049] Wherein, one host attribute information is used to reflect the host feature of one host.

[0050] As a possible implementation manner, the first server sets a configuration file in the first server according to actual business needs, reads the configuration file, and obtains the attribute name of the to-be-obtained host attribute information; based on the attribute name of the to-be-obtained host attribute information, the to-be-obtained host attribute information corresponding to each host is obtained from the database including a plurality of attribute information of a plurality of hosts.

[0051] For example, Figure 3As shown, the data agent server sets a configuration file in the data agent service according to actual business needs (the configuration file is used to set the attribute name of the host attribute information that a user needs to obtain, and instructs the data agent service to obtain the to-be-obtained host attribute information corresponding to each host at a preset time interval), sets the configuration file, and after the configuration file is set, the data agent service reads the configuration file, determines the attribute name of the to-be-obtained host attribute information, and according to the attribute name of the to-be-obtained host attribute that has been determined, queries the configuration management database (CMDB) at a regular time, and obtains the to-be-obtained host attribute information corresponding to each host from the database.

[0052] The complete attribute information of all hosts in the host cluster is stored and managed in the CMDB database. According to the management specification, only in the release window time (the release window time refers to a preset time period during which a change operation can be performed on the CMDB database) can the information stored in the CMDB database be changed by the management system. The host attribute information stored in the CMDB database includes host address information, host device type information, host brand information, host location information, host CPU model, host operating system type information, and host architecture information. The host address information is used to identify a node on a network, is a unique address possessed by each device in the network, and can be used to communicate and communicate with other devices on the network, including Internet Protocol Address (IPAddress) and Media Access Control (MAC) address. The data agent service is used to obtain the to-be-obtained host attribute information corresponding to each host from the database according to the configuration file, and provide a data query interface to the outside, so that each host can obtain the corresponding host attribute information through the data query interface.

[0053] For example, the data agent service reads the configuration file, obtains the attribute name of the to-be-obtained host attribute information, and at the release window time of the CMDB database, queries the CMDB database at a regular time, and obtains the to-be-obtained host attribute information corresponding to each host from the database. For example, the existing host A and host B, the configuration file indicates that the host attribute information to be collected includes host address information, host operating system type information, and host architecture information, then the data agent service obtains the above three kinds of host attribute information corresponding to each host from the CMDB database at a regular time according to the above three kinds of host attribute information to be obtained, such as host address information of host A, operating system type information of host A, and architecture information of host A; and host address information of host B, operating system type information of host B, and architecture information of host B.

[0054] S202: Each host queries the first server for its corresponding host attribute information to be obtained, and adds its corresponding host attribute information to be obtained to its own host monitoring data to obtain its own target data.

[0055] As a possible implementation method, each host obtains the corresponding host monitoring data (host monitoring data is used to reflect the host's operating indicators) through a collector pre-deployed on its own host (such as the node-exporter collector); after obtaining the host monitoring data, each host uses a preset data processing module to periodically query the data query interface of the data proxy service to obtain the host attribute information to be obtained corresponding to each host, and adds the host attribute information to be obtained corresponding to each host to the host monitoring data to obtain the target data (the target data is used to reflect the host's operating status and host characteristics).

[0056] Optionally, host monitoring data may include: CPU usage, CPU load rate (average CPU load within a specified time period), system memory usage (used memory obtained by subtracting free memory from cache), network usage (upload rate, download rate, network fluctuations, etc.), disk reads and writes (disk reads per second, disk writes per second, total disk reads, total disk writes, etc.), and other operating indicators.

[0057] For example, Figure 4 As shown, after the data proxy service obtains the host attribute information to be obtained corresponding to each host from the CMDB database according to the configuration file, host A queries the data query interface of the data proxy service through the data processing service deployed on the host, and obtains the host attribute information corresponding to host A: host address information of host A, operating system type information of host A, and architecture information of host A, and stores the above-mentioned host attribute information in the local cache of the host; host B queries the data query interface of the data proxy service through the data processing service deployed on the host, and obtains the host attribute information corresponding to host B: host address information of host B, operating system type information of host B, and architecture information of host B, and stores the above-mentioned host attribute information in the local cache of the host.

[0058] For example, host A obtains corresponding host monitoring data through the node-exporter collector pre-deployed on its own host, and obtains multiple host operation indicators, including: CPU usage of host A system, host A system startup time, host A current system time, host A system memory usage, host A system load, host A file system usage, host A network fluctuation, host A network speed and other host operation indicators. After host A obtains the corresponding host attribute information of host A and the monitoring data of host A, as shown in the following example: Figure 5As shown, the host A adds the host attribute information corresponding to the host A to each running index of the host monitoring data through the data processing service deployed on the host, obtains target data, that is, obtains multiple processed host monitoring indexes. For example, the host address information of the host A, the operating system type information of the host A and the architecture information of the host A are added to the system memory usage of the host A, to obtain a host monitoring index; the host address information of the host A, the operating system type information of the host A and the architecture information of the host A are added to the file system usage of the host A, to obtain another host monitoring index, and so on.

[0059] Similarly, the host B obtains the corresponding host monitoring data through the node-exporter collector pre-deployed on the host B, to obtain multiple host running indexes, including: the CPU usage of the host B system, the system startup time of the host B, the current system time of the host B, the system memory usage of the host B, the system load of the host B, the file system usage of the host B, the network fluctuation of the host B, the network rate of the host B and the like. Figure 5 As shown, the host B adds the host attribute information corresponding to the host B to each running index of the host monitoring data through the data processing service deployed on the host, obtains target data, that is, obtains multiple processed host monitoring indexes. For example, the host address information of the host B, the operating system type information of the host B and the architecture information of the host B are added to the system memory usage of the host B, to obtain a host monitoring index; the host address information of the host B, the operating system type information of the host B and the architecture information of the host B are added to the system load of the host B, to obtain a host monitoring index, and so on.

[0060] S203, the second server collects target data of each host in the host cluster.

[0061] The target data of one host is used to reflect the running situation and the host characteristics of the host.

[0062] As a possible implementation manner, the second server queries the data query interface of the data processing service on each host through the Prometheus system pre-deployed on the second server, to obtain the target data of each host in the host cluster.

[0063] For example, the data processing services deployed inside each host in the host cluster provide a data query interface to the outside. The data processing service on each host adds the host attribute information to be obtained corresponding to each host to the host monitoring data. After obtaining the target data, the second server queries the data query interface of the data processing service on each host through the Prometheus system pre-deployed on the second server to obtain the target data of each host in the host cluster.

[0064] In one design, in order to facilitate users to change the desired host attribute information, such as Figure 6 As shown, the cluster data collection method provided in the embodiment of the present application may further include the following S301-S303.

[0065] S301: The first server determines updated host attribute information to be obtained, and obtains the updated host attribute information to be obtained for each host.

[0066] As a possible implementation method, when receiving a user's modification operation instruction for a configuration file, the first server updates the configuration file in response to the modification operation instruction, and reads the updated configuration file to obtain the updated attribute name of the host attribute information to be obtained; based on the updated attribute name of the host attribute information to be obtained, the first server obtains the updated host attribute information to be obtained for each host from a database including multiple attribute information of multiple hosts.

[0067] For example, when the host attribute information that the user needs to obtain changes, that is, when receiving the user's modification operation instruction for the configuration file, the data proxy service responds to the modification operation instruction, updates the configuration file, and reads the updated configuration file to obtain the updated attribute name of the host attribute information to be obtained; and based on the updated attribute name of the host attribute information to be obtained, periodically queries the CMDB database to obtain the updated host attribute information to be obtained for each host from the database.

[0068] For example, the existing host A and host B, the original configuration file indicates that the host attribute information to be collected includes host address information, host operating system type information and host architecture information, the modification operation instruction indicates that the host attribute information to be newly obtained includes host CPU model, then the data proxy service updates the configuration file in response to the modification operation instruction, reads the updated configuration file and obtains the attribute name of the updated host attribute information to be obtained. At this time, the updated configuration file indicates that the host attribute information to be collected includes host address information, host operating system type information, host architecture information and host CPU model, then the data proxy service obtains the above four kinds of host attribute information corresponding to each host from the CMDB database, such as host address information of host A, operating system type information of host A, architecture information of host A and CPU model of host A; and host address information of host B, operating system type information of host B, architecture information of host B and CPU model of host B.

[0069] S302, each host queries the updated host attribute information to be obtained from the first server, and adds the updated host attribute information to be obtained of each host to the host monitoring data of each host to obtain the updated target data of each host.

[0070] As a possible implementation manner, each host obtains the corresponding host monitoring data through the node-exporter collector pre-deployed on the host itself, after obtaining the host monitoring data, each host queries the data query interface of the data proxy service through the preset data processing module, obtains the updated host attribute information to be obtained of each host, and adds the updated host attribute information to be obtained of each host to the host monitoring data of each host to obtain the updated target data of each host.

[0071] For example, when the host attribute information that the user needs to obtain changes, the data agent service reads the updated configuration file to obtain the attribute name of the updated host attribute information to be obtained, and based on the attribute name of the updated host attribute information to be obtained, the data agent service obtains the corresponding host attribute information of each host from the database. Host A queries the data query interface of the data agent service through the data processing service deployed on the host to obtain the host attribute information corresponding to host A after the configuration file is updated, i.e., the host address information of host A, the operating system type information of host A, the architecture information of host A, and the CPU model of host A. The data processing service deployed on the host adds the host attribute information corresponding to host A after the configuration file is updated to each running index of the host monitoring data to obtain multiple processed host monitoring indexes. For example, host A adds the host address information of host A, the operating system type information of host A, the architecture information of host A, and the CPU model of host A to the system memory usage of host A through the data processing service to obtain a host monitoring index.

[0072] Similarly, host B queries the data query interface of the data agent service through the data processing service deployed on the host to obtain the host attribute information corresponding to host B after the configuration file is updated, i.e., the host address information of host B, the operating system type information of host B, the architecture information of host B, and the CPU model of host B. The data processing service deployed on the host adds the host attribute information corresponding to host B after the configuration file is updated to each running index of the host monitoring data to obtain multiple processed host monitoring indexes. For example, host B adds the host address information of host B, the operating system type information of host B, the architecture information of host B, and the CPU model of host B to the system load of host B through the data processing service to obtain a host monitoring index.

[0073] S303, the second server collects the updated target data of each host in the host cluster.

[0074] As a possible implementation manner, after the data processing service on each host adds the updated host attribute information to be obtained to the host monitoring data to obtain the updated target data, the second server queries the data query interface of the data processing service on each host through the Prometheus system pre-deployed on the second server to obtain the updated target data of each host in the host cluster.

[0075] As Figure 7As shown, the data proxy server sets a configuration file in the data proxy service according to actual business needs, reads the configuration file after setting the configuration file, determines the attribute name of the host attribute information to be obtained, and queries the CMDB database at a regular time according to the attribute name of the host attribute to be obtained that has been determined, to obtain the host attribute information to be obtained corresponding to each host from the database. When the host attribute information to be obtained changes, the data proxy service updates the configuration file in response to the user's modification operation instruction, reads the updated configuration file to obtain the attribute name of the updated host attribute information to be obtained, and queries the CMDB database at a regular time based on the attribute name of the updated host attribute information to be obtained, to obtain the host attribute information to be obtained corresponding to each host from the database, and provides a data query interface to the outside, so that each host obtains the corresponding host attribute information through the data query interface.

[0076] Further, each host queries the data query interface of the data proxy service at a regular time through the data processing service deployed in each host, obtains the host attribute information to be obtained corresponding to each host, and stores the host attribute information to be obtained corresponding to each host in the local cache of each host. The data processing service of each host adds the host attribute information to be obtained corresponding to each host to the host monitoring data to obtain target data after the host monitoring data is obtained by the node-exporter collector pre-deployed on the host itself, and provides a data query interface to the outside. The second server queries the data query interface of the data processing service on each host through the Prometheus system pre-deployed on the second server to obtain the target data of each host in the host cluster.

[0077] The technical scheme provided by the embodiments of the present application at least brings the following beneficial effects: the first server determines the to-be-acquired host attribute information, and acquires the to-be-acquired host attribute information of each host, one host attribute information being used to reflect the host feature of one host; compared with the prior art, the Prometheus system acquires the corresponding host attribute information through the configuration file corresponding to each host, and the present application only needs to set one configuration file in the first server, so that the acquisition of the to-be-acquired host attribute information can be completed, which is more convenient; and when the to-be-acquired host attribute information needs to be changed, the prior art needs to modify the configuration file corresponding to each host one by one, while the present application only needs to respond to the modification operation instruction and update the configuration file in the first server, thereby reducing the workload and making the operation more flexible. Each host queries the corresponding to-be-acquired host attribute information of itself from the first server, adds the corresponding to-be-acquired host attribute information of itself to the host monitoring data of itself, obtains the target data of itself, and then the second server collects the target data of each host in the host cluster, and obtains the running situation and the host feature of each host in the host cluster; compared with the prior art, the host attribute information of each host is added to the monitoring data of the corresponding host in the Prometheus system, and the present application completes the operation of adding the host attribute information of each host to the monitoring data of the corresponding host in each host, thereby distributing the processing pressure to the processing service of each host, and reducing the processing pressure of the Prometheus system.

[0078] The above embodiments mainly introduce the schemes provided by the embodiments of the present application from the perspective of the device (apparatus). It can be understood that, in order to implement the above method, the device or apparatus contains the corresponding hardware structure and / or software module for executing each method process, which can constitute a material information determination device. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments of the present application, the present application can be realized in the form of hardware or the combination of hardware and computer software. Whether a certain function is executed by hardware or computer software driven hardware depends on the specific application and design constraints of the technical scheme. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0079] The embodiments of the present application can divide the functions of the device or equipment into function modules according to the above method examples. For example, the device or equipment can correspond to each function to divide each function module, or two or more functions can be integrated into one processing module. The integrated module can be realized in the form of hardware or in the form of a software function module. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical function division. There can be another division manner when actually implemented.

[0080] Figure 1 is a structural schematic diagram of a cluster data collection system according to an example embodiment. Referring to Figure 1 The cluster data collection system 10 provided by the embodiments of the present application includes a host cluster 101, a first server 102, and a second server 103.

[0081] The first server 102 is configured to determine to-be-acquired host attribute information, and acquire the to-be-acquired host attribute information of each host. One host attribute information is used to reflect the host characteristics of one host. The host cluster 101 is configured to query the to-be-acquired host attribute information corresponding to each host from the first server, and add the to-be-acquired host attribute information corresponding to each host to the host monitoring data of each host, to obtain target data of each host. The second server 103 is configured to collect the target data of each host in the host cluster. The target data of one host is used to reflect the running situation and the host characteristics of the host.

[0082] Optionally, the first server 102 is specifically configured to read a configuration file to obtain attribute names of the to-be-acquired host attribute information. The first server acquires the to-be-acquired host attribute information of each host by acquiring the to-be-acquired host attribute information corresponding to each host from a database including multiple attribute information of multiple hosts based on the attribute names of the to-be-acquired host attribute information.

[0083] Optionally, the first server 102 is further configured to, in a case where a modification operation instruction for the configuration file is received, update the configuration file in response to the modification operation instruction, read the updated configuration file to obtain attribute names of updated to-be-acquired host attribute information, and acquire the to-be-acquired host attribute information corresponding to each host from the database including multiple attribute information of multiple hosts based on the attribute names of the updated to-be-acquired host attribute information.

[0084] Optionally, the host cluster 101 is specifically configured to, for any one host, acquire host monitoring data by a collector pre-deployed on the host. The host monitoring data is used to reflect the running index of the host. The host adds the to-be-acquired host attribute information to the host monitoring data by a preset data processing module to obtain target data.

[0085] Optionally, the second server 103 is specifically configured to collect target data of each host in the host cluster through a Prometheus system pre-deployed on the second server.

[0086] Figure 8 is a structural schematic diagram of an electronic device provided by the present application. As Figure 8 The electronic device 40 can include at least one processor 401 and a memory 402 for storing processor-executable instructions, wherein the processor 401 is configured to execute the instructions in the memory 402 to implement the cluster data collection method in the above embodiments.

[0087] In addition, the electronic device 40 can further include a communication bus 403 and at least one communication interface 404.

[0088] The processor 401 can be a central processing unit (CPU), a micro processing unit, an ASIC, or one or more integrated circuits for controlling the execution of programs of the present application scheme.

[0089] The communication bus 403 can include a path for transmitting information between the above components.

[0090] The communication interface 404 uses any transceiver-like device for communicating with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0091] The memory 402 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor 401 via a bus. The memory may also be integrated with the processor 401.

[0092] The memory 402 is used to store instructions for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the instructions stored in the memory 402, thereby realizing the functions of the method of the present application.

[0093] As an example, combining Figure 1 The functions implemented by the host cluster 101, the first server 102 and the second server 103 in the cluster data acquisition system 10 are similar to those implemented by Figure 8 The function of the processor 401 in is the same.

[0094] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 in.

[0095] In a specific implementation, as an embodiment, the electronic device 40 may include multiple processors, such as Figure 8 4 and 5. The processors 401 and 407 are shown in FIG. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0096] In a particular implementation, as one example, the electronic device 40 can also include an output device 405 and an input device 406. The output device 405 is in communication with the processor 401 and can take on many forms. For example, the output device 405 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, a projector, or the like. The input device 406 is in communication with the processor 401 and can take on many forms. For example, the input device 406 can be a mouse, a keyboard, a touch screen device, a sensor device, or the like.

[0097] Those skilled in the art can understand that Figure 8 The structure shown in the figure is not intended to limit the electronic device 40, and the electronic device 40 can include more or fewer components than shown, or a combination of some components, or a different arrangement of components.

[0098] In addition, the present application also provides a computer readable storage medium, when the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the cluster data collection method provided by the above-mentioned embodiments.

[0099] In addition, the present application also provides a computer program product, including computer instructions, when the computer instructions run on the electronic device, the electronic device executes the cluster data collection method provided by the above-mentioned embodiments.

[0100] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the features of the application as set forth herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

Claims

1. A method of cluster data collection, characterized by, The application is applied to a cluster data collection system, the cluster data collection system comprising a host cluster, a first server and a second server; the host cluster comprises a plurality of hosts, the first server is in communication connection with each host, and the second server is in communication connection with each host; The first server determines to-be-acquired host attribute information and acquires the to-be-acquired host attribute information of each host; One host attribute information is used to reflect the host characteristics of one host; Each host queries its corresponding to-be-acquired host attribute information from the first server and adds the to-be-acquired host attribute information to its host monitoring data to obtain target data of the host; The second server collects the target data of each host in the host cluster; The target data of one host is used to reflect the running condition and host characteristics of the host; The first server determines to-be-acquired host attribute information, comprising: The first server reads a configuration file to obtain attribute names of the to-be-acquired host attribute information; The first server acquires to-be-acquired host attribute information of each host, comprising: The first server acquires to-be-acquired host attribute information corresponding to each host from a database comprising a plurality of attribute information of a plurality of hosts based on the attribute names of the to-be-acquired host attribute information; The host adds its corresponding to-be-acquired host attribute information to its host monitoring data to obtain target data of the host, comprising: For any host, the host acquires host monitoring data through a collector pre-deployed on the host; the host monitoring data is used to reflect the running index of the host; The host adds the to-be-acquired host attribute information to the host monitoring data through a preset data processing module to obtain the target data.

2. The method of claim 1, wherein, The method further comprises: In a case where a modification operation instruction for the configuration file is received, the first server updates the configuration file in response to the modification operation instruction and reads the updated configuration file to obtain attribute names of updated to-be-acquired host attribute information; The first server acquires to-be-acquired host attribute information corresponding to each host from a database comprising a plurality of attribute information of a plurality of hosts based on the attribute names of the updated to-be-acquired host attribute information.

3. The method of claim 1, wherein, The second server collects the target data of each host in the host cluster, comprising: The second server collects the target data of each host in the host cluster through a Prometheus system pre-deployed on the second server.

4. A cluster data acquisition system characterized by, The cluster data collection system comprises a host cluster, a first server and a second server; the host cluster comprises a plurality of hosts, the first server is in communication connection with each host, and the second server is in communication connection with each host; The first server is configured to determine to-be-acquired host attribute information and acquire the to-be-acquired host attribute information of each host; One host attribute information is used to reflect the host characteristics of one host; The host cluster is configured to: each host queries each corresponding to-be-acquired host attribute information from the first server, and adds each corresponding to-be-acquired host attribute information to each host monitoring data to obtain each target data; The second server is configured to: collect target data of each host in the host cluster; and the target data of one host is used to reflect the running condition and host characteristics of one host; The first server is specifically configured to: read a configuration file to obtain attribute names of the to-be-acquired host attribute information; acquire to-be-acquired host attribute information of each host, including: the first server acquires corresponding to-be-acquired host attribute information of each host from a database including multiple attribute information of multiple hosts based on the attribute names of the to-be-acquired host attribute information; The host cluster is specifically configured to: for any one host, the host acquires host monitoring data through a collector pre-deployed on the host; and the host monitoring data is used to reflect the running index of the host; The host adds the to-be-acquired host attribute information to the host monitoring data through a preset data processing module to obtain the target data.

5. The cluster data acquisition system of claim 4, wherein, The first server is further configured to: in a case where a modification operation instruction for the configuration file is received, the first server updates the configuration file in response to the modification operation instruction, reads the updated configuration file to obtain attribute names of updated to-be-acquired host attribute information, and acquires corresponding to-be-acquired host attribute information of each host from a database including multiple attribute information of multiple hosts based on the attribute names of the updated to-be-acquired host attribute information. The second server is specifically configured to:

6. The cluster data acquisition system of claim 4, wherein, collect target data of each host in the host cluster through a Prometheus system pre-deployed on the second server. including:

7. An electronic device, comprising: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute instructions to implement the cluster data collection method of any one of claims 1-3. When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the cluster data collection method of any one of claims 1-3.

8. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: ​

Images