Data acquisition method, device, system, electronic device and storage medium

Abstract

The embodiments of the present invention provide a data acquisition method, device, system, electronic device and storage medium. The host in a distributed cluster obtains the target data indicated by the specified monitoring index from the operating data of each component deployed in the host; and sends the obtained target data to the management server. For each distributed cluster, the management server receives the target data sent by each host in the distributed cluster; determines the to-be-processed data indicated by the calculation index corresponding to the distributed cluster from the target data sent by each host in the distributed cluster; and aggregates the to-be-processed data corresponding to the distributed cluster to obtain the corresponding aggregation result. Based on the above processing, the operating data of the components deployed in each host in each distributed cluster can be obtained, and the operating data of the components deployed in each host in the distributed cluster can be uniformly managed.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a data acquisition method, device, system, electronic equipment and storage medium. Background Art

[0002] With the rapid development of big data technology, distributed clusters based on big data technology are widely used in various fields. A distributed cluster includes multiple hosts (e.g., servers), and multiple components based on big data technology can be deployed in each host to achieve corresponding functions. For example, Hbase (distributed database) components are deployed on each host in a distributed cluster to achieve the function of distributed storage of data.

[0003] In order to optimize the performance of the distributed cluster, it is necessary to obtain the operating data of each component deployed in each host in the distributed cluster. Subsequently, the performance of the host (for example, the CPU usage of the host, the memory usage of the host, etc.) can be analyzed based on the operating data of each component obtained. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data acquisition method, device, system, electronic device and storage medium, which can acquire the operating data of the components deployed in each host in each distributed cluster, and can uniformly manage the operating data of the components deployed in each host in the distributed cluster. The specific technical solution is as follows:

[0005] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present invention provides a data acquisition system, the system comprising: a distributed cluster and a management server, wherein:

[0006] The host in the distributed cluster is used to obtain data indicated by the specified monitoring indicator from the operating data of each component deployed in the host as target data; and send the obtained target data to the management server;

[0007] The management server is used to receive, for each distributed cluster, target data sent by each host in the distributed cluster; determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the data to be processed corresponding to the distributed cluster; and perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain a corresponding aggregation result, wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0008] In a second aspect, in order to achieve the above-mentioned purpose, an embodiment of the present invention provides a data acquisition method, which is applied to a management server and includes:

[0009] For each distributed cluster, receiving target data sent by each host in the distributed cluster; wherein the target data is: data indicated by a specified monitoring indicator in the operation data of each component deployed in the host;

[0010] Determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster; wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster;

[0011] Aggregate the to-be-processed data corresponding to the distributed cluster to obtain corresponding aggregation results.

[0012] Optionally, before receiving the target data sent by each host in each distributed cluster, the method further includes:

[0013] For each distributed cluster, through the data transmission process, according to the correspondence between the cluster priority and the preset queue recorded in the preset database, the preset queue corresponding to the cluster priority of the distributed cluster is determined as the target queue;

[0014] The step of receiving target data sent by each host in each distributed cluster includes:

[0015] Receiving target data sent by each host in the distributed cluster through the data transmission process;

[0016] After receiving the target data sent by each host in each distributed cluster, the method further includes:

[0017] Adding target data of the distributed cluster to the target queue through the data transmission process;

[0018] The target data of the distributed cluster is obtained from the target queue through the data processing process.

[0019] Optionally, after adding the target data of the distributed cluster to the target queue, the method further includes:

[0020] If the target data of the distributed cluster is not successfully added to the target queue, the target data of the distributed cluster is added to a backup queue; wherein the backup queue is used to recover the data that is not successfully added to the queue.

[0021] Optionally, before determining, in the target data sent from each host in the distributed cluster, data indicated by the calculation indicator corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster, the method further includes:

[0022] Obtaining the calculation index corresponding to the distributed cluster from the preset database through the data processing process;

[0023] The step of determining, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster, includes:

[0024] Through the data processing process, data indicated by the calculation index corresponding to the distributed cluster is determined from the target data sent by each host in the distributed cluster as the data to be processed corresponding to the distributed cluster.

[0025] Optionally, the method further includes:

[0026] When a monitoring strategy update instruction is received, the corresponding relationship between the cluster priority and the preset queue in the preset database and / or the calculation index of each distributed cluster are updated.

[0027] Optionally, the method further includes:

[0028] For each distributed cluster, when receiving a registration request sent by a host in the distributed cluster, obtaining the specified monitoring indicator from the preset database;

[0029] The specified monitoring indicator is sent to the host in the distributed cluster.

[0030] In a third aspect, in order to achieve the above-mentioned purpose, an embodiment of the present invention provides a data acquisition method, which is applied to a host in a distributed cluster, and the method includes:

[0031] Acquire data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data;

[0032] The acquired target data is sent to the management server, so that the management server determines, for each distributed cluster, the data indicated by the calculation index corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; the data to be processed corresponding to the distributed cluster is aggregated to obtain a corresponding aggregation result, wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0033] Optionally, before obtaining data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data, the method further includes:

[0034] Sending a registration request of the distributed cluster to the management server, so that the management server obtains a specified monitoring indicator from a preset database when receiving the registration request, and sends the specified monitoring indicator to the host;

[0035] Receive the specified monitoring indicator sent by the management server.

[0036] Optionally, before obtaining data indicated by the specified monitoring indicator from the operating data of each component deployed in the host as the target data, the method further includes:

[0037] When a component adding message is received, a monitoring interface address of the added component is obtained from the component adding message; wherein the monitoring interface address is used to obtain the operation data of the added component.

[0038] In a fourth aspect, in order to achieve the above-mentioned purpose, an embodiment of the present invention provides a data acquisition device, which is applied to a management server, and includes:

[0039] A receiving module is used to receive target data sent by each host in each distributed cluster; wherein the target data is: data indicated by a specified monitoring indicator in the running data of each component deployed in the host;

[0040] A first determination module is used to determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster; wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster;

[0041] The aggregation module is used to aggregate the to-be-processed data corresponding to the distributed cluster to obtain corresponding aggregation results.

[0042] Optionally, the device further comprises:

[0043] A second determination module is used to determine, for each distributed cluster, a preset queue corresponding to the cluster priority of the distributed cluster as a target queue according to the correspondence between the cluster priority and the preset queue recorded in a preset database through a data transmission process before the receiving module executes for each distributed cluster and receives the target data sent by each host in the distributed cluster;

[0044] The receiving module is specifically used to receive target data sent by each host in the distributed cluster through the data transmission process;

[0045] The device also includes:

[0046] A first adding module is used for adding the target data of the distributed cluster to the target queue through the data transmission process after the receiving module receives the target data sent by each host in the distributed cluster for each distributed cluster;

[0047] The first acquisition module is used to acquire the target data of the distributed cluster from the target queue through a data processing process.

[0048] Optionally, the device further comprises:

[0049] A second adding module is used to add the target data of the distributed cluster to the target queue after the first adding module executes the adding of the target data of the distributed cluster to the target queue, and if the target data of the distributed cluster is not successfully added to the target queue, add the target data of the distributed cluster to the backup queue; wherein the backup queue is used to recover the data that is not successfully added to the queue.

[0050] Optionally, the device further comprises:

[0051] A second acquisition module is used to execute, before the first determination module executes the target data sent from each host in the distributed cluster, to determine the data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster, to obtain the calculation index corresponding to the distributed cluster from the preset database through the data processing process;

[0052] The first determination module is specifically used to determine, through the data processing process, data indicated by the calculation index corresponding to the distributed cluster from the target data sent by each host in the distributed cluster, as the data to be processed corresponding to the distributed cluster.

[0053] Optionally, the device further comprises:

[0054] The updating module is used to update the corresponding relationship between the cluster priority and the preset queue in the preset database and / or the calculation index of each distributed cluster when receiving the monitoring strategy update instruction.

[0055] Optionally, the device further comprises:

[0056] A third acquisition module is used to acquire the specified monitoring indicator from the preset database for each distributed cluster when receiving a registration request sent by a host in the distributed cluster;

[0057] The sending module is used to send the specified monitoring indicator to the host in the distributed cluster.

[0058] In a fifth aspect, in order to achieve the above-mentioned purpose, an embodiment of the present invention provides a data acquisition device, which is applied to a host in a distributed cluster, and the device includes:

[0059] A first acquisition module is used to acquire data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data;

[0060] The first sending module is used to send the acquired target data to the management server, so that the management server can determine, for each distributed cluster, the data indicated by the calculation index corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; and perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain the corresponding aggregation result, wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0061] Optionally, the device further comprises:

[0062] a second sending module, configured to execute sending a registration request for the distributed cluster to the management server before the first obtaining module executes to obtain data indicated by the specified monitoring indicator from the running data of each component deployed in the host as the target data, so that the management server obtains the specified monitoring indicator from a preset database upon receiving the registration request, and sends the specified monitoring indicator to the host;

[0063] The receiving module is used to receive the specified monitoring indicator sent by the management server.

[0064] Optionally, the device further comprises:

[0065] The second acquisition module is used to obtain the data indicated by the specified monitoring indicator from the operating data of each component deployed in the host as the target data before the first acquisition module executes the acquisition. When a component addition message is received, the monitoring interface address of the added component is obtained from the component addition message; wherein the monitoring interface address is used to obtain the operating data of the added component.

[0066] An embodiment of the present invention further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0067] Memory, used to store computer programs;

[0068] The processor is used to implement the data acquisition method steps described in the second aspect or the data acquisition method steps described in the third aspect when executing the program stored in the memory.

[0069] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the data acquisition method steps described in the second aspect or the data acquisition method steps described in the third aspect are implemented.

[0070] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data acquisition method described in the second aspect or the data acquisition method described in the third aspect.

[0071] The technical solution provided by the embodiment of the present invention is that the host in the distributed cluster obtains the data indicated by the specified monitoring index from the operation data of each component deployed in the host as the target data; and sends the obtained target data to the management server. For each distributed cluster, the management server receives the target data sent by each host in the distributed cluster; from the target data sent by each host in the distributed cluster, the data indicated by the calculation index corresponding to the distributed cluster is determined as the data to be processed corresponding to the distributed cluster; the data to be processed corresponding to the distributed cluster is aggregated to obtain the corresponding aggregation result.

[0072] Based on the above processing, the running data of the components deployed in each host in each distributed cluster can be obtained through the management server and the hosts in the distributed cluster, and the running data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0073] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0075] Figure 1A flow chart of a data acquisition method provided by an embodiment of the present invention;

[0076] Figure 2 A flowchart of another data acquisition method provided by an embodiment of the present invention;

[0077] Figure 3 A schematic diagram of a preset queue provided in an embodiment of the present invention;

[0078] Figure 4 A flowchart of another data acquisition method provided by an embodiment of the present invention;

[0079] Figure 5 A flowchart of another data acquisition method provided by an embodiment of the present invention;

[0080] Figure 6 A flowchart of another data acquisition method provided by an embodiment of the present invention;

[0081] Figure 7 A flowchart of a component registration method provided by an embodiment of the present invention;

[0082] Figure 8 A flowchart of another data acquisition method provided by an embodiment of the present invention;

[0083] Fig. 9 A flowchart of another data acquisition method provided by an embodiment of the present invention;

[0084] Fig.10 A structural diagram of a data acquisition device provided by an embodiment of the present invention;

[0085] Fig.11 A structural diagram of another data acquisition device provided by an embodiment of the present invention;

[0086] Fig.12 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0088] In order to optimize the performance of the distributed cluster, it is necessary to obtain the operating data of each component deployed in each host in the distributed cluster. Subsequently, the performance of the host (for example, the CPU usage of the host, the memory usage of the host, etc.) can be analyzed based on the operating data of each component obtained.

[0089] An embodiment of the present invention provides a data acquisition system, which can acquire the operation data of each component deployed in each host in a distributed cluster. The system includes a distributed cluster and a management server, wherein:

[0090] The host in the distributed cluster is used to obtain the data indicated by the specified monitoring indicator from the operating data of each component deployed in the host as the target data; and send the obtained target data to the management server.

[0091] A management server is used to receive, for each distributed cluster, target data sent by each host in the distributed cluster; determine, from the target data sent by each host in the distributed cluster, data indicated by a calculation indicator corresponding to the distributed cluster as pending data corresponding to the distributed cluster; and perform aggregation processing on the pending data corresponding to the distributed cluster to obtain a corresponding aggregation result, wherein the calculation indicator corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0092] Based on the data acquisition system provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be acquired through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0093] For other embodiments of the above data acquisition system, please refer to the subsequent introduction of related methods for distributed clusters and management servers.

[0094] See also Figure 1 , Figure 1 A flowchart of a data acquisition method provided by an embodiment of the present invention, which can be applied to the management server in the above data acquisition system, can include the following steps:

[0095] S101: For each distributed cluster, receiving target data sent by each host in the distributed cluster.

[0096] The target data is: data indicated by a specified monitoring indicator in the operation data of each component deployed in the host.

[0097] S102: Determine, from target data sent by each host in the distributed cluster, data indicated by a calculation indicator corresponding to the distributed cluster as to-be-processed data corresponding to the distributed cluster.

[0098] The calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0099] S103: Aggregate the to-be-processed data corresponding to the distributed cluster to obtain a corresponding aggregation result.

[0100] Based on the data acquisition method provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be obtained through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0101] With respect to step S101, for each distributed cluster, multiple components can be deployed in each host in the distributed cluster to achieve corresponding functions. Each component will generate corresponding operation data during operation. In order to analyze the performance of the host in the distributed cluster, each host in the distributed cluster can obtain the data indicated by the specified monitoring indicator from the operation data of each component deployed in the host, obtain the target data of the distributed cluster, and send the target data of the distributed cluster to the management server. The operation data of a distributed cluster may include the operation log of the distributed cluster.

[0102] The specified monitoring indicators may include monitoring indicators corresponding to different performances of the host, for example, monitoring indicators for determining the memory status of the host, monitoring indicators for determining the status of each thread in the host, and monitoring indicators for determining various events in the host.

[0103] Exemplarily, the specified monitoring indicators corresponding to the memory status may include: MemNonHeapUsedM (the size of non-heap memory currently used by the JVM (Java Virtual Machine)), MemNonHeapMaxM (the size of non-heap memory configured for the JVM), MemHeapUsedM (the size of heap memory currently used by the JVM), MemHeapMaxM (the size of heap memory configured for the JVM), and MemMaxM (the maximum memory that can be used by the JVM during runtime), etc.

[0104] The specified monitoring indicators corresponding to the status of each thread may include: ThreadsNew (the number of threads currently in the New state), ThreadsRunnable (the number of threads currently in the Runnable state), ThreadsBlocked (the number of threads currently in the Blocked state), and ThreadsWaiting (the number of threads currently in the Waiting state), etc.

[0105] The specified monitoring indicators corresponding to various types of events may include: LogError (the number of Error events within a fixed time interval), LogWarn (the number of Warn events within a fixed time interval), and LogInfo (the number of Info events within a fixed time interval), etc.

[0106] For the specific processing method of the host in the distributed cluster, please refer to the relevant introduction of the subsequent embodiments.

[0107] Accordingly, for each distributed cluster, the management server may receive target data sent by each host in the distributed cluster.

[0108] In one embodiment of the present invention, Figure 1 Based on Figure 2 Before step S101, the method may further include the following steps:

[0109] S104: For each distributed cluster, through the data transmission process, according to the correspondence between the cluster priority and the preset queue recorded in the preset database, determine the preset queue corresponding to the cluster priority of the distributed cluster as the target queue.

[0110] Accordingly, step S101 may include the following steps:

[0111] S1011: Receive target data sent by each host in the distributed cluster through a data transmission process.

[0112] Accordingly, after step S101, the method may further include the following steps:

[0113] S105: adding the target data of the distributed cluster to the target queue through the data transmission process.

[0114] S106: Obtain target data of the distributed cluster from the target queue through the data processing process.

[0115] The higher the cluster priority of a distributed cluster, the more important the distributed cluster is. The number of calculation indicators of a distributed cluster with a high cluster priority is greater than the number of calculation indicators of a distributed cluster with a low cluster priority, that is, a distributed cluster with a high priority obtains more running data.

[0116] The cluster priority of a distributed cluster may be determined according to the function of the distributed cluster. For example, the cluster priority of a distributed cluster used for processing services is higher than the cluster priority of a distributed cluster used for testing.

[0117] Multiple queues can be preset in the management server, and different preset queues are used to store target data of distributed clusters with different cluster priorities.

[0118] The preset database may be Redis, and the preset database stores the corresponding relationship between the cluster priority and the preset queue.

[0119] The management server can obtain the correspondence between the cluster priority and the preset queue from the preset database through the data transmission process. Then, for each distributed cluster, the management server can determine the preset queue corresponding to the cluster priority of the distributed cluster as the target queue according to the above correspondence through the data transmission process.

[0120] Furthermore, when receiving the target data sent by each host in the distributed cluster, the target data of the distributed cluster can be added to the target queue through the data transmission process. Subsequently, the management server can obtain the target data of the distributed cluster from the target queue through the data processing process.

[0121] In addition, for each distributed cluster, if there is no preset queue corresponding to the cluster priority of the distributed cluster in the correspondence between the cluster priority and the preset queue, it indicates that the target data of the distributed cluster may be tampered with by a malicious attacker. The data transmission process can discard the target data of the distributed cluster, thereby improving the security of the management server.

[0122] Based on the above processing, when a large amount of target data is received from each distributed cluster, the management server may not be able to process all the target data at the same time. After storing the target data of each distributed cluster in the preset queue, the management server can obtain the target data from the preset queue in turn, and process the obtained target data in turn, which can reduce the processing pressure of the management server.

[0123] In addition, if a large amount of data is added to the preset queue corresponding to the cluster priority of a distributed cluster, and the target data of the distributed cluster may not be able to be added to the preset queue corresponding to the cluster priority of the distributed cluster, a new queue can be added to the management server, and the target data of the distributed cluster can be added to the new queue, which can avoid data loss to a certain extent.

[0124] In one embodiment of the present invention, after step S105, the method may further include the following steps: if the target data of the distributed cluster is not successfully added to the target queue, adding the target data of the distributed cluster to a backup queue.

[0125] The backup queue is used to recover data that has not been successfully added to the queue.

[0126] In one implementation, for each distributed cluster, when adding the target data of the distributed cluster to the target queue corresponding to the cluster priority of the distributed cluster, the target data of the distributed cluster may not be successfully added to the target queue due to an abnormality in the target queue (for example, a large amount of data is stored in the target queue).

[0127] If the target data of the distributed cluster is not successfully added to the target queue, in order to avoid data loss, the target data of the distributed cluster may be added to a backup queue.

[0128] Subsequently, when the time it takes for the target data of the distributed cluster to be added to the backup queue reaches a preset time, the target queue may have returned to normal, and the management server can obtain the target data of the distributed cluster from the backup queue and add the target data of the distributed cluster back to the target queue corresponding to the distributed cluster.

[0129] For example, see Figure 3 , the distributed clusters include: cluster A, cluster B and cluster C. The preset queues may include: high priority queue, general queue and backup queue.

[0130] For each distributed cluster, after receiving the target data of the distributed cluster, the management server can determine the preset queue corresponding to the cluster priority of the distributed cluster from multiple preset queues through the data transmission process, and add the target data of the distributed cluster to the target queue corresponding to the cluster priority of the distributed cluster. For example, the cluster priority of cluster A is higher, and the management server can add the target data of cluster A to the high priority queue through the data transmission process. The cluster priority of cluster B is lower, and the management server can add the target data of cluster B to the general queue through the data transmission process.

[0131] When adding the target data of cluster A to the high priority queue, if the target data of cluster A is not successfully added to the high priority queue, in order to avoid data loss, the target data of cluster A can be added to the backup queue. Later, when the time for adding the target data of cluster A to the backup queue reaches a preset time, the management server can obtain the target data of cluster A from the backup queue and add the target data of cluster A to the high priority queue again.

[0132] For step S102, for each distributed cluster, the target data of the distributed cluster is the data indicated by the specified monitoring index. In order to uniformly manage multiple distributed clusters, the specified monitoring indexes corresponding to each distributed cluster are the same, and the number of the specified monitoring indexes is greater than the number of the calculation indexes corresponding to each distributed cluster. Therefore, for each distributed cluster, there is unnecessary data in the target data of the distributed cluster.

[0133] Accordingly, after receiving the target data of the distributed cluster, the management server can obtain the calculation index corresponding to the distributed cluster. Furthermore, the management server can determine the data indicated by the calculation index corresponding to the distributed cluster from the target data sent by each host in the distributed cluster, and obtain the to-be-processed data corresponding to the distributed cluster.

[0134] In one embodiment of the present invention, Figure 2 Based on Figure 4 Before step S102, the method may further include the following steps:

[0135] S107: Obtain the calculation index corresponding to the distributed cluster from the preset database through the data processing process.

[0136] Accordingly, step S102 may include the following steps:

[0137] S1021: Determine, through a data processing process, data indicated by a calculation indicator corresponding to the distributed cluster from target data sent by each host in the distributed cluster as data to be processed corresponding to the distributed cluster.

[0138] The management server can obtain the calculation index of each distributed cluster from the preset database through the data processing process, and process the target data of the distributed cluster according to the calculation index of the distributed cluster, so as to realize targeted processing of different distributed clusters.

[0139] With respect to step S103 , for each distributed cluster, after determining the data to be processed of the distributed cluster, the management server may perform aggregation processing on the data to be processed of the distributed cluster to obtain a corresponding aggregation result.

[0140] Exemplarily, for each distributed cluster, the calculation index in the distributed cluster includes: LogError, then the data to be processed of the distributed cluster includes: the number of Error events of each component deployed in each host in the distributed cluster within a fixed time interval. Furthermore, the management server can aggregate the data to be processed of the distributed cluster, and obtain the total number of Error events of each component deployed in each host in the distributed cluster within a fixed time interval.

[0141] After obtaining the aggregation result of the distributed cluster, the management server can store the aggregation result of the distributed cluster locally. Later, when receiving a query request for the distributed cluster sent by the client, the aggregation result of the distributed cluster can be sent to the client. Correspondingly, the client can display the aggregation result of the distributed cluster to the user.

[0142] In addition, for each distributed cluster, the management server can also issue an alarm based on the aggregation result of the distributed cluster. For example, for the above embodiment, when the total number of Error events of each component deployed in each host in the distributed cluster within a fixed time interval reaches a preset threshold, the management server can send an alarm message to the client, and the alarm message can carry the identifier of the distributed cluster and the aggregation result of the distributed cluster.

[0143] In one embodiment of the present invention, the method may further include the following steps: when a monitoring strategy update instruction is received, the corresponding relationship between the cluster priority and the preset queue in the preset database, and / or the calculation index of each distributed cluster is updated.

[0144] In one implementation, the user can set the correspondence between the cluster priority and the preset queue and the calculation index of each distributed cluster. When the above information needs to be modified, the technician can send a monitoring policy update instruction to the management server through the client. The monitoring policy update instruction can carry the correspondence between the cluster priority and the preset queue, and / or the calculation index of each distributed cluster. Upon receiving the monitoring policy update instruction, the management server can update the correspondence between the cluster priority and the preset queue in the preset data, and / or the calculation index of each distributed cluster.

[0145] For example, if the computing indicators of a distributed cluster include: indicator A, indicator B and indicator C, and the monitoring policy update instruction carries the computing indicators of the distributed cluster including: D, upon receiving the monitoring policy update instruction, the management server can update the computing indicators of the distributed cluster to: indicator A, indicator B, indicator C and indicator D.

[0146] Based on the above processing, the user can configure the corresponding relationship between the cluster priority and the preset queue and the calculation index of each distributed cluster, which can meet the personalized needs of the user.

[0147] In one embodiment of the present invention, the method may further include the following steps:

[0148] Step 1: for each distributed cluster, when a registration request sent by a host in the distributed cluster is received, a specified monitoring indicator is obtained from a preset database.

[0149] Step 2: Send the specified monitoring indicator to the host in the distributed cluster.

[0150] In one implementation, for each distributed cluster, each host in the distributed cluster may send a registration request to the management server before obtaining the target data of the distributed cluster. The registration request may carry the name of the distributed cluster, the name of the host in the distributed cluster, and the name of each component deployed in the host.

[0151] Correspondingly, when receiving the registration request, the management server may obtain the specified monitoring indicator from the preset database, and send the specified monitoring indicator to the host in the distributed cluster.

[0152] In addition, the designated monitoring indicator may be set by a technical staff, and the above-mentioned monitoring policy update instruction may also carry the designated monitoring indicator. Furthermore, when receiving the monitoring policy update instruction, the management server may also update the designated monitoring indicator.

[0153] See also Figure 5 , Figure 5 A flowchart of a data acquisition method provided by an embodiment of the present invention, which can be applied to a host in a distributed cluster in the above data acquisition system, can include the following steps:

[0154] S501: Acquire data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data.

[0155] S502: Send the acquired target data to the management server, so that the management server can determine, for each distributed cluster, the data indicated by the calculation indicator corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain the corresponding aggregation result.

[0156] The calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0157] Based on the data acquisition method provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be obtained through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0158] With respect to step S501, for each distributed cluster, multiple components can be deployed in each host in the distributed cluster to achieve corresponding functions. Each component will generate corresponding operation data when running. In order to analyze the performance of the host in the distributed cluster, each host in the distributed cluster can obtain data indicated by the specified monitoring indicator from the operation data of each component deployed in the host to obtain the target data of the distributed cluster.

[0159] In one embodiment of the present invention, before obtaining the target data, the management server may also obtain the specified monitoring index from the management server. Figure 5 Based on Figure 6 Before step S501, the method may further include the following steps:

[0160] S503: Send a registration request of the distributed cluster to the management server, so that the management server obtains the specified monitoring indicator from the preset database when receiving the registration request, and sends the specified monitoring indicator to the host.

[0161] S504: Receive the specified monitoring indicator sent by the management server.

[0162] In one implementation, for each distributed cluster, each host in the distributed cluster may send a registration request to the management server before obtaining the target data of the distributed cluster. Accordingly, upon receiving the registration request, the management server may obtain the specified monitoring indicator from the preset database and send the specified monitoring indicator to the host in the distributed cluster.

[0163] With respect to step S502 , after acquiring the target data, each host in the distributed cluster may send the acquired target data to the management server.

[0164] Correspondingly, after receiving the target data of each distributed cluster, the management server can determine the data indicated by the calculation indicator corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; and perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain the corresponding aggregation result.

[0165] For the specific processing method of the management server, please refer to the relevant introduction of the above embodiment.

[0166] In one embodiment of the present invention, before step S501, the method may further include the following steps: when a component adding message is received, acquiring a monitoring interface address of the added component from the component adding message.

[0167] The monitoring interface address is used to obtain the operating data of the added component.

[0168] The monitoring interface address of the component may be a JMX (a framework for embedding management functions into applications, devices, systems, etc.) interface address of the component, or a JVM interface address of the component.

[0169] For example, see Figure 7 , the components deployed in the host in the distributed cluster include component A and component B. If a new component (for example, component C) is deployed in the host, the host receives a component addition message through the data collection process, and the component addition message carries the JMX interface address of component C. Subsequently, the data collection process can obtain the running data of component C according to the JMX interface address of component C.

[0170] For example, when component C is Hadoop (a distributed system infrastructure), the JMX interface address of Hadoop is: http: / / host:60010 / jmx.

[0171] Based on the above processing, when a new component is deployed in a host in a distributed cluster, the host can obtain the operation data of the new component through the new component monitoring interface address, thereby realizing dynamic expansion of the component.

[0172] See also Figure 8 , Figure 8 A flowchart of another data acquisition method provided by an embodiment of the present invention, the data acquisition method is applied to a host and a management server in a distributed cluster. The distributed cluster includes: distributed cluster 1 and distributed cluster 2. The components deployed in the host in each distributed cluster include: Hadoop, Hdfs (distributed file system), Hbase (an open source non-relational database), Hive (a data warehouse tool based on Hadoop), and Presto (a data query engine).

[0173] Each host in the distributed cluster sends a registration request to the management server through the data collection process. After receiving the registration request, the management server can send the specified monitoring index to the host in the distributed cluster through the data management process.

[0174] The host in the distributed cluster can obtain the target data indicated by the specified monitoring indicator from the running data of each component deployed in the host, and send the obtained target data to the Data-transfer (data transfer process) in the management server. The data transfer process is developed based on Openresty+LUA (a web platform based on the Lua language).

[0175] Correspondingly, after receiving the target data sent by each distributed cluster, the management server can obtain the correspondence between the cluster priority and the preset queue from Redis through the data transmission process, and determine the target queue corresponding to the cluster priority of the distributed cluster based on the above correspondence, and add the target data of the distributed cluster to the target queue.

[0176] Then, the management server can obtain the target data of a distributed cluster from the preset queue through the data processing process, and determine the to-be-processed data indicated by the calculation index of the distributed cluster from the target data of the distributed cluster. Then, the management server can aggregate the to-be-processed data of the distributed cluster to obtain the aggregated result of the distributed cluster, and store the aggregated result of the distributed cluster in the aggregated result storage database.

[0177] In addition, the management server can also receive monitoring policy update instructions sent by users through the data management process. The management server can communicate with the client through a specified API (Application Programming Interface), and the user can enter the monitoring policy update instruction on the client, and the client sends the monitoring policy update instruction to the management server through the API. Alternatively, the management server can configure a user access page, and the user can enter the monitoring policy update instruction to the management server through the user access page.

[0178] The monitoring strategy update instruction can carry the correspondence between cluster priority and preset queue, and the management server can store the received correspondence to Redis through the data management process. The monitoring strategy update instruction can also carry the specified monitoring indicators and the calculation indicators of each distributed cluster, and the management server can store the received specified monitoring indicators and the calculation indicators of each distributed cluster locally through the data management process.

[0179] Based on the data acquisition method provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be obtained through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0180] See also Fig. 9 , Fig. 9 This is a flow chart of another data acquisition method provided in an embodiment of the present invention. The method may include the following steps:

[0181] S901: Register monitoring components through the monitoring collection module.

[0182] The monitoring and collection module is the data collection process in the aforementioned embodiment. When a host in a distributed cluster receives a component adding message, it determines the monitoring interface address of the component carried in the component adding message. Furthermore, the host in the distributed cluster can obtain the target data indicated by the specified monitoring indicator from the running data of each component through the monitoring interface address of each component.

[0183] S902: Monitoring data is reported to Openresty.

[0184] The monitoring data is the target data in the aforementioned embodiment. When the target data is obtained, the host in the distributed cluster can upload the target data to the data transmission process in the management server, and the data transmission process is developed based on Openresty.

[0185] S903: Determine whether there is a Topic strategy, if yes, execute step S905, if no, execute step S904.

[0186] The topic strategy is the preset queue corresponding to the cluster priority of the distributed cluster in the above embodiment. For each distributed cluster, the management server determines whether there is a target queue corresponding to the cluster priority of the distributed cluster through the data transmission process, and makes corresponding processing according to the determination result.

[0187] S904: discard the indicator.

[0188] For each distributed cluster, if the management server determines through the data transmission process that there is no target queue corresponding to the cluster priority of the distributed cluster, the target data of the distributed cluster is discarded.

[0189] S905: Upload to the preset queue.

[0190] For each distributed cluster, if the management server determines through the data transmission process that there is a target queue corresponding to the cluster priority of the distributed cluster, the target data of the distributed cluster is added to the target queue corresponding to the cluster priority of the distributed cluster.

[0191] S906: Obtain calculation indicators through the calculation module.

[0192] The calculation module is the data processing process in the aforementioned embodiment. For each distributed cluster, the management server obtains the calculation index of the distributed cluster through the data processing process, and determines the data to be processed indicated by the calculation index of the distributed cluster from the target data of the distributed cluster. Then, the management server aggregates the data to be processed of the distributed cluster through the data processing process to obtain the corresponding aggregation result.

[0193] S907: Calculation results are stored.

[0194] The calculation result is the aggregation result in the aforementioned embodiment. For each distributed cluster, when the aggregation result of the distributed cluster is obtained, the management server may also store the aggregation result of the distributed cluster locally.

[0195] S908: The monitoring management center configures new component monitoring indicators.

[0196] The monitoring management center is the data management process in the aforementioned embodiment. The management server can receive monitoring policy update instructions through the data management process, and update the correspondence between cluster priority and preset queues, and / or the calculation indicators of each distributed cluster according to the received monitoring policy update instructions.

[0197] Based on the data acquisition method provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be obtained through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0198] and Figure 1 Corresponding to the method embodiment of Fig.10 , Fig.10 A structural diagram of a data acquisition device provided in an embodiment of the present invention, wherein the device is applied to a management server and comprises:

[0199] The receiving module 1001 is used to receive target data sent by each host in each distributed cluster; wherein the target data is data indicated by a specified monitoring indicator in the operation data of each component deployed in the host;

[0200] The first determination module 1002 is used to determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster; wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster;

[0201] The aggregation module 1003 is used to perform aggregation processing on the to-be-processed data corresponding to the distributed cluster to obtain corresponding aggregation results.

[0202] Optionally, the device further comprises:

[0203] A second determination module is used to determine, for each distributed cluster, a preset queue corresponding to the cluster priority of the distributed cluster as a target queue according to the correspondence between the cluster priority and the preset queue recorded in a preset database through a data transmission process before the receiving module 1001 executes for each distributed cluster and receives the target data sent by each host in the distributed cluster;

[0204] The receiving module 1001 is specifically used to receive target data sent by each host in the distributed cluster through the data transmission process;

[0205] The device also includes:

[0206] A first adding module is used for adding the target data of the distributed cluster to the target queue through the data transmission process after the receiving module 1001 receives the target data sent by each host in the distributed cluster for each distributed cluster;

[0207] The first acquisition module is used to acquire the target data of the distributed cluster from the target queue through a data processing process.

[0208] Optionally, the device further comprises:

[0209] A second adding module is used to add the target data of the distributed cluster to the target queue after the first adding module executes the adding of the target data of the distributed cluster to the target queue, and if the target data of the distributed cluster is not successfully added to the target queue, add the target data of the distributed cluster to the backup queue; wherein the backup queue is used to recover the data that is not successfully added to the queue.

[0210] Optionally, the device further comprises:

[0211] A second acquisition module is used for obtaining the calculation index corresponding to the distributed cluster from the preset database through the data processing process before the first determination module 1002 executes the target data sent from each host in the distributed cluster to determine the data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster;

[0212] The first determination module 1002 is specifically used to determine, through the data processing process, data indicated by the calculation index corresponding to the distributed cluster from the target data sent by each host in the distributed cluster, as the data to be processed corresponding to the distributed cluster.

[0213] Optionally, the device further comprises:

[0214] The updating module is used to update the corresponding relationship between the cluster priority and the preset queue in the preset database and / or the calculation index of each distributed cluster when receiving the monitoring strategy update instruction.

[0215] Optionally, the device further comprises:

[0216] A third acquisition module is used to acquire the specified monitoring indicator from the preset database for each distributed cluster when receiving a registration request sent by a host in the distributed cluster;

[0217] The sending module is used to send the specified monitoring indicator to the host in the distributed cluster.

[0218] Based on the data acquisition device provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be acquired through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0219] and Figure 5 Corresponding to the method embodiment of Fig.11 , Fig.11 A structural diagram of a data acquisition device provided in an embodiment of the present invention, the device is applied to a host in a distributed cluster, and the device includes:

[0220] The first acquisition module 1101 is used to acquire data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data;

[0221] The first sending module 1102 is used to send the acquired target data to the management server, so that the management server can determine, for each distributed cluster, the data indicated by the calculation indicator corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; and perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain a corresponding aggregation result, wherein the calculation indicator corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

[0222] Optionally, the device further comprises:

[0223] A second sending module is used for executing sending a registration request for the distributed cluster to the management server before the first obtaining module 1101 executes to obtain data indicated by the specified monitoring indicator from the running data of each component deployed in the host as the target data, so that the management server obtains the specified monitoring indicator from the preset database when receiving the registration request, and sends the specified monitoring indicator to the host;

[0224] The receiving module is used to receive the specified monitoring indicator sent by the management server.

[0225] Optionally, the device further comprises:

[0226] The second acquisition module is used to obtain the data indicated by the specified monitoring indicator from the operating data of each component deployed in the host before the first acquisition module 1101 executes as the target data, and when a component addition message is received, obtain the monitoring interface address of the added component from the component addition message; wherein the monitoring interface address is used to obtain the operating data of the added component.

[0227] Based on the data acquisition device provided by the embodiment of the present invention, the operation data of the components deployed in each host in each distributed cluster can be acquired through the management server and the host in the distributed cluster, and the operation data of the components deployed in each host in the distributed cluster can also be uniformly managed through the management server. In addition, the target data of the distributed cluster can be processed according to the calculation index corresponding to the cluster priority of each distributed cluster, so that targeted processing can be achieved for different distributed clusters.

[0228] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.

[0229] Memory 1203, used for storing computer programs;

[0230] The processor 1201 is used to implement the steps of the data acquisition method applied to the management server in the above embodiment, or the steps of the data acquisition method applied to the host in the distributed cluster in the above embodiment when executing the program stored in the memory 1203.

[0231] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0232] The communication interface is used for communication between the above electronic device and other devices.

[0233] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0234] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0235] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of the data acquisition method applied to the management server in the above embodiment are implemented, or the steps of the data acquisition method applied to the host in the distributed cluster in the above embodiment are implemented.

[0236] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer is run on the computer, the computer executes the data acquisition method applied to the management server in the above embodiment, or the data acquisition method applied to the host in the distributed cluster in the above embodiment.

[0237] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0238] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0239] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, system, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0240] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data acquisition system, characterized in that: The system comprises: a distributed cluster and a management server, wherein: The host in the distributed cluster is used to obtain data indicated by the specified monitoring indicator from the operating data of each component deployed in the host as target data; and send the obtained target data to the management server; The management server is used to receive, for each distributed cluster, target data sent by each host in the distributed cluster; determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the data to be processed corresponding to the distributed cluster; and perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain a corresponding aggregation result, wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

2. A data acquisition method, characterized in that: The method is applied to a management server, and the method comprises: For each distributed cluster, receiving target data sent by each host in the distributed cluster; wherein the target data is: data indicated by a specified monitoring indicator in the operation data of each component deployed in the host; Determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster; wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster; Aggregate the to-be-processed data corresponding to the distributed cluster to obtain corresponding aggregation results.

3. The method according to claim 2, characterized in that Before receiving the target data sent by each host in each distributed cluster, the method further includes: For each distributed cluster, through the data transmission process, according to the correspondence between the cluster priority and the preset queue recorded in the preset database, the preset queue corresponding to the cluster priority of the distributed cluster is determined as the target queue; The step of receiving target data sent by each host in each distributed cluster includes: Receiving target data sent by each host in the distributed cluster through the data transmission process; After receiving the target data sent by each host in each distributed cluster, the method further includes: Adding target data of the distributed cluster to the target queue through the data transmission process; The target data of the distributed cluster is obtained from the target queue through the data processing process.

4. The method according to claim 3, characterized in that After adding the target data of the distributed cluster to the target queue, the method further includes: If the target data of the distributed cluster is not successfully added to the target queue, the target data of the distributed cluster is added to a backup queue; wherein the backup queue is used to recover the data that is not successfully added to the queue.

5. The method according to claim 3, characterized in that: Before determining, from the target data sent from each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster, the method further includes: Obtaining the calculation index corresponding to the distributed cluster from the preset database through the data processing process; The step of determining, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster, includes: Through the data processing process, data indicated by the calculation index corresponding to the distributed cluster is determined from the target data sent by each host in the distributed cluster as the data to be processed corresponding to the distributed cluster.

6. The method according to claim 5, characterized in that The method further comprises: When a monitoring strategy update instruction is received, the corresponding relationship between the cluster priority and the preset queue in the preset database and / or the calculation index of each distributed cluster are updated.

7. The method according to claim 6, characterized in that The method further comprises: For each distributed cluster, when receiving a registration request sent by a host in the distributed cluster, obtaining the specified monitoring indicator from the preset database; The specified monitoring indicator is sent to the host in the distributed cluster.

8. A data acquisition method, characterized in that: The method is applied to a host in a distributed cluster, and the method comprises: Acquire data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data; The acquired target data is sent to the management server, so that the management server determines, for each distributed cluster, the data indicated by the calculation index corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; the data to be processed corresponding to the distributed cluster is aggregated to obtain a corresponding aggregation result, wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

9. The method according to claim 8, characterized in that Before obtaining data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data, the method further includes: Sending a registration request of the distributed cluster to the management server, so that the management server obtains a specified monitoring indicator from a preset database when receiving the registration request, and sends the specified monitoring indicator to the host; Receive the specified monitoring indicator sent by the management server.

10. The method according to claim 8, characterized in that Before obtaining data indicated by the specified monitoring indicator from the operation data of each component deployed in the host as target data, the method further includes: When a component adding message is received, a monitoring interface address of the added component is obtained from the component adding message; wherein the monitoring interface address is used to obtain the operation data of the added component.

11. A data acquisition device, characterized in that: The device is applied to a management server, and comprises: A receiving module is used to receive target data sent by each host in each distributed cluster; wherein the target data is: data indicated by a specified monitoring indicator in the running data of each component deployed in the host; A first determination module is used to determine, from the target data sent by each host in the distributed cluster, data indicated by the calculation index corresponding to the distributed cluster as the to-be-processed data corresponding to the distributed cluster; wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster; The aggregation module is used to aggregate the to-be-processed data corresponding to the distributed cluster to obtain corresponding aggregation results.

12. A data acquisition device, characterized in that: The device is applied to a host in a distributed cluster, and the device includes: A first acquisition module is used to acquire data indicated by a specified monitoring indicator from the operation data of each component deployed in the host as target data; The first sending module is used to send the acquired target data to the management server, so that the management server can determine, for each distributed cluster, the data indicated by the calculation index corresponding to the distributed cluster from the target data sent by the host in the distributed cluster, as the data to be processed corresponding to the distributed cluster; and perform aggregation processing on the data to be processed corresponding to the distributed cluster to obtain the corresponding aggregation result, wherein the calculation index corresponding to the distributed cluster is determined according to the cluster priority of the distributed cluster.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when used to execute a program stored in a memory, implements the method steps described in any one of claims 2 to 7 or claims 8 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 2 to 7 or claims 8 to 10 are implemented.

Images