Data processing method, device, computer equipment and storage medium for monitoring system

Abstract

The embodiment of the present application discloses a data processing method, device, computer equipment and storage medium for a monitoring system. The method is applied to a monitoring system, including: obtaining multiple monitoring data; determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node; if there is a first abnormal service node whose monitoring data amount is greater than the corresponding monitoring data amount threshold, then determining multiple first service modules with different node granularity labels according to the first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node; deploying the corresponding first service module in each first candidate service node to obtain multiple second target service nodes; continuing to determine whether there is an overloaded first abnormal service node until there is no first abnormal service node, and processing the monitoring data through the first target service node. This embodiment can improve the processing efficiency of the monitoring data.

Description

Technical Field

[0001] The present application relates to the technical field of system monitoring, and in particular to a data processing method, apparatus, computer equipment, and storage medium for a monitoring system. Background Art

[0002] With the development of the Internet, in order to ensure the stability of enterprise systems, more and more applications and services in enterprises need to be monitored so as to timely discover and solve potential problems. Therefore, the demand for monitoring systems is also increasing.

[0003] As the scale and scope of the monitoring system continue to expand, the amount of monitoring data that the monitoring system needs to monitor also increases. If the amount of monitoring data obtained by the monitoring system is too large, the time it takes for the monitoring system to process and analyze the monitoring data will also increase, resulting in a slower response time for the monitoring system, thereby affecting the monitoring system's processing efficiency of the monitoring data. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, apparatus, computer equipment, and storage medium for a monitoring system, which can improve the processing efficiency of monitoring data by the monitoring system.

[0005] In a first aspect, an embodiment of the present application provides a data processing method for a monitoring system, the method being applied to the monitoring system, the method comprising:

[0006] Acquire a plurality of monitoring data, wherein the monitoring data carries a data granularity tag, and the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data;

[0007] Determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, wherein a plurality of first target service nodes with different node granularity labels are deployed in the monitoring system;

[0008] Based on the correspondence between the first target service node and the monitoring data volume threshold, determining whether there is a first abnormal service node among the plurality of first target service nodes, the monitoring data volume of which is greater than the corresponding monitoring data volume threshold;

[0009] If the first abnormal service node exists, determining a plurality of second target service nodes and node granularity labels corresponding to the second target service nodes according to a preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, wherein the number of the second target service nodes is greater than the number of the first target service nodes;

[0010] Taking the second target service node as the first target service node, returning to the step of determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, until the first abnormal service node no longer exists;

[0011] The plurality of monitoring data are processed by a plurality of the first target service nodes.

[0012] In a second aspect, an embodiment of the present application further provides a data processing device for a monitoring system, comprising: the data processing device for the monitoring system is disposed in the monitoring system, and the data processing device for the monitoring system comprises:

[0013] a transceiver unit, configured to obtain a plurality of monitoring data, wherein the monitoring data carries a data granularity tag, and the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data;

[0014] A processing unit is configured to determine the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node, wherein multiple first target service nodes with different node granularity labels are deployed in the monitoring system; based on the correspondence between the first target service node and the monitoring data amount threshold, determine whether there is a first abnormal service node among the multiple first target service nodes whose monitoring data amount is greater than the corresponding monitoring data amount threshold; if the first abnormal service node exists, determine multiple second target service nodes and the node granularity label corresponding to each second target service node based on the preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, wherein the number of the second target service nodes is greater than the number of the first target service nodes; use the second target service node as the first target service node, and return to execute the step of determining the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node until the first abnormal service node no longer exists; and process the multiple monitoring data through the multiple first target service nodes.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above method can be implemented.

[0017] The embodiment of the present application provides a data processing method, device, computer equipment and storage medium for a monitoring system. Wherein, the method is applied to a monitoring system, and the method includes: after obtaining a plurality of monitoring data from different source objects (monitored objects), determining the amount of monitoring data corresponding to each first target service node according to the relationship between the data granularity label carried by the monitoring data and the node granularity label of each first target service node, if it is detected that there is a first abnormal service node (the first abnormal service node is overloaded) whose monitoring data amount is greater than the corresponding monitoring data amount threshold among the plurality of first target service nodes, it is necessary to adjust the number of first service modules (i.e., the number of first target service nodes) and the node granularity label of each first service module (i.e., the node granularity label corresponding to the first target service node) according to the first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, and determine the number of first service modules. and node granularity labels, select the first candidate service node corresponding to each first service module in the node resource pool, and deploy the corresponding first service module in the corresponding first candidate service node to obtain multiple second target service nodes, and use the second target service node as the first target service node, and continue to detect whether there is a first abnormal service node in each first target service node until there is no first service abnormal node, and use the adjusted first target service node to process multiple monitoring data; it can be seen that in the embodiment of the present application, when it is found that there is an abnormal service node with excessive load in the monitoring system, the number of service nodes and node granularity labels in the monitoring system can be dynamically adjusted, thereby reducing the problem of low efficiency in monitoring data processing due to excessive load on the service node, and improving the processing efficiency of the monitoring system for monitoring data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A schematic diagram of an application scenario of the data processing method of the monitoring system provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a data processing method for a monitoring system provided in an embodiment of the present application;

[0021] Figure 3 A schematic block diagram of a data processing device of a monitoring system provided in an embodiment of the present application;

[0022] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0025] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] Embodiments of the present application provide a data processing method, apparatus, computer equipment, and storage medium for a monitoring system.

[0028] The executor of the data processing method of the monitoring system may be the data processing device of the monitoring system provided in the embodiment of the present application, or a computer device in which the data processing device of the monitoring system is integrated, wherein the data processing device of the monitoring system may be implemented in hardware or software, and the computer device may be a terminal or a server, in which the monitoring system is deployed.

[0029] See also Figure 1 , Figure 1 Schematic diagram of the application scenario of the data processing method of the monitoring system provided in the embodiment of the present application. The data processing method of the monitoring system is applied to Figure 1In the monitoring system 10, in some embodiments, the monitoring system 10 obtains multiple monitoring data from a monitored object 20 (i.e., a source object of the monitoring data) in each of multiple environments in the target system (such as environment A and environment B in the figure). After obtaining the multiple monitoring data, the amount of monitoring data corresponding to each first target service node is determined based on the association between the data granularity label carried by the monitoring data and the node granularity label of each first target service node. Then, based on the monitoring data amount threshold corresponding to each first target service node, it is determined whether there is a first abnormal node with an excessive load among the multiple first target service nodes. If there is a first abnormal node with an excessive load, it is necessary to dynamically adjust the number of the first target service nodes and the node granularity label. The specific adjustment steps are as follows:

[0030] According to the preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, the quantity of the first service modules and the node granularity label of each first service module are adjusted. After the quantity of the first service modules and the node granularity label are determined, the first candidate service node corresponding to each first service module is selected in the node resource pool, and the corresponding first service module is deployed in the corresponding first candidate service node to obtain multiple second target service nodes, and the second target service node is used as the first target service node. Then, continue to detect whether there is a first abnormal service node in each first target service node until there is no first service abnormal node, and use the adjusted first target service node to process multiple monitoring data.

[0031] For ease of understanding, some terms mentioned in this embodiment are explained below:

[0032] Data granularity label: The data granularity label indicates the minimum granularity source object of the corresponding monitoring data. For example, when the monitoring system is used to monitor the target system, the target system includes multiple environments, each environment includes multiple business lines, such as environment A, environment B and environment C, where environment A includes business line A1, business line A2 and business line A3; environment B includes business line B1, business line B2 and business line B3; environment C includes business line C1 and business line C2. If the business line is the minimum granularity source object (monitoring object) in the target system, the data granularity label of each acquired monitoring data needs to be Specifically for business line labels, for example, multiple monitoring data include monitoring data 1, monitoring data 2, monitoring data 3 and monitoring data 4. The data granularity label of monitoring data 1 is: B-B1, indicating that the monitoring data 1 comes from business line B1 in environment B; the data granularity label of monitoring data 2 is: B-B2, indicating that the monitoring data 2 comes from business line B2 in environment B; the data granularity label of monitoring data 3 is: C-C2, indicating that the monitoring data 3 comes from business line C2 in environment C; the data granularity label of monitoring data 4 is: A-A1, indicating that the monitoring data 4 comes from business line A1 in environment A.

[0033] It should be noted that the minimum granularity source object can also be the equipment under the business line, where each business line includes multiple devices, or other granularity objects. This embodiment does not limit the minimum granularity source object. The embodiment of this application only uses the minimum granularity source object as an example of the business line.

[0034] Node granularity label: The granularity label set for the target service node in the monitoring system, which represents the granularity and source object of the monitoring data that the target service node can process. For example, if the node granularity label corresponding to a target service node is B, then the target service node needs to process the monitoring data from environment B, and the monitoring system needs to assign the acquired monitoring data with a data granularity label of B-B1, B-B2, or B-B3 to the target service node; for another example, if the node granularity label corresponding to a target service node is C-C1, then the target service node needs to process the monitoring data from business line C1 in environment C, and the monitoring system needs to assign the acquired monitoring data with a data granularity label of C-C1 to the target service node.

[0035] Target service nodes: such as the first target service node, the second target service node, the third target service node and the fourth target service node, are nodes on which service modules are deployed. The service module can be a Prometheus service module. In this case, the monitoring system is a Prometheus monitoring system. The target service nodes on which service modules are deployed have the ability to process monitoring data. Different target service nodes are set with different node granularity labels and process monitoring data corresponding to the node granularity labels.

[0036] Monitoring data volume threshold: the maximum monitoring data volume that the corresponding target service node can process. After determining the first target service node, it is necessary to establish a correspondence between the first target service node and the monitoring data volume threshold. The monitoring data volume threshold corresponding to each first target service node can be set by the user or automatically determined according to the performance of the corresponding service node. When the acquired monitoring data includes monitoring data of multiple data types, the monitoring data volume threshold can be a data volume threshold of a specified data volume type. For example, the monitoring data acquired by a certain target service node includes 100 CPU usage indicators and 50 memory usage indicators. The set monitoring data volume threshold is the data volume threshold of the CPU usage indicator. At this time, it is only necessary to judge the data volume size of the CPU usage indicator.

[0037] Among them, if the amount of monitoring data obtained by the target service node exceeds the monitoring data volume threshold, it means that the target service node is overloaded and is an abnormal service node. At this time, it is necessary to dynamically adjust the number of target service nodes and node granularity labels in the monitoring system to solve the problem of service node overload.

[0038] Node resource pool: includes multiple candidate service nodes. The candidate service nodes in the node resource pool are computing resources that can deploy service modules, for example, computing power cards.

[0039] Figure 2 FIG. 1 is a flow chart of a data processing method for a monitoring system provided in an embodiment of the present application. Figure 2 As shown, the method includes the following steps S110-S160.

[0040] S110 : Acquire multiple monitoring data, where the monitoring data carries a data granularity label.

[0041] The data granularity tag indicates the minimum granularity source object of the corresponding monitoring data.

[0042] In this embodiment, the monitoring system needs to detect and analyze the flow of data in real time, including analyzing the amount of monitoring data acquired by each first target service node, and determining the data processing requirements and status of the current monitoring system based on the acquired amount of monitoring data.

[0043] In an embodiment of the present application, a plurality of first target service nodes with different node granularity labels are deployed in the monitoring system.

[0044] S120: Determine the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node.

[0045] In this embodiment, each piece of monitoring data obtained carries a corresponding data granularity label, and each first target service node in the monitoring system is provided with a corresponding node granularity label, wherein the data granularity label indicates the minimum granularity source object of the corresponding monitoring data, and the node granularity label represents the granularity size and source object of the monitoring data that the target service node can process.

[0046] This embodiment uses the example of the target system's monitored objects including environment A, environment B, and environment C, where environment A includes business line A1, business line A2, and business line A3; environment B includes business line B1, business line B2, and business line B3; and environment C includes business line C1 and business line C2, where the business line is the smallest granularity source object in the target system.

[0047] At the same time, the multiple monitoring data obtained include at least one monitoring data 1, at least one monitoring data 2, at least one monitoring data 3 and at least one monitoring data 4, wherein the data granularity label of monitoring data 1 is: B-B1 (indicating that the monitoring data 1 comes from business line B1 in environment B); the data granularity label of monitoring data 2 is: B-B2 (indicating that the monitoring data 2 comes from business line B2 in environment B); the data granularity label of monitoring data 3 is: C-C2 (indicating that the monitoring data 3 comes from business line C2 in environment C); the data granularity label of monitoring data 4 is: A-A1 (indicating that the monitoring data 4 comes from business line A1 in environment A) for illustration.

[0048] If the first target service node in the monitoring system includes target service node 1, target service node 2, target service node 3 and target service node 4, the node granularity label corresponding to target service node 1 is A (indicating that the monitoring data from environment A can be assigned to the service node, including monitoring data with data granularity labels A-A1 and A-A2), the corresponding node granularity label of target service node 2 is B (indicating that the monitoring data from environment B can be assigned to the service node, including monitoring data with data granularity labels B-B1 and B-B2), the corresponding node granularity label of target service node 3 is C-C1 (indicating that the monitoring data from business line C1 in environment C can be assigned to the service node), and the corresponding node granularity label of target service node 3 is C-C2 (indicating that the monitoring data from business line C2 in environment C can be assigned to the service node).

[0049] At this time, the monitoring system will determine the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node.

[0050] For example, the monitoring system allocates monitoring data 1 with data granularity label B-B1 and monitoring data 2 with data granularity label B-B2 to target service node 2 with node granularity label B. The total quantity of monitoring data 1 and monitoring data 2 is the amount of monitoring data corresponding to target service node 2; allocates monitoring data 3 with data granularity label C-C2 to target service node 3 with node granularity label C-C2. The quantity of monitoring data 3 is the amount of monitoring data corresponding to target service node 3; allocates monitoring data 4 with data granularity label A-A1 to target service node 1 with node granularity label A. The quantity of monitoring data 4 is the amount of monitoring data corresponding to target service node 4. After the allocation is completed, the amount of monitoring data corresponding to each target service node is determined.

[0051] S130. Based on the correspondence between the first target service node and the monitoring data volume threshold, determine whether there is a first abnormal service node among the multiple first target service nodes whose monitoring data volume is greater than the corresponding monitoring data volume threshold; if so, execute step S140; if not, execute step S160.

[0052] In this embodiment, each first target service node has its corresponding monitoring data volume threshold, which can be automatically determined according to the performance of the corresponding service node (the higher the performance, the higher the threshold), or can be set by the user.

[0053] In some embodiments, the monitoring data volume threshold can be a data volume threshold of a specified data volume type. For example, the monitoring data obtained by a target service node includes 100 CPU usage indicators and 50 memory usage indicators. The set monitoring data volume threshold is the data volume threshold of the CPU usage indicator. At this time, it is only necessary to judge the data volume size of the CPU usage indicator.

[0054] In other embodiments, the monitoring data volume threshold is a total monitoring data volume, which needs to be determined based on the data type of the acquired monitoring data, the monitoring data volume corresponding to each data type, and the weight corresponding to each data type. For example, the data type includes CPU usage indicator and memory usage indicator. The monitoring data received by a service node includes M CPU usage indicators and N memory usage indicators. The weight corresponding to the CPU usage indicator is 0.7, and the weight corresponding to the memory usage indicator is 0.3. At this time, the total monitoring data volume = 0.7×M+0.3×N.

[0055] S140: Determine a plurality of second target service nodes and a node granularity label corresponding to each of the second target service nodes according to a preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node.

[0056] The number of the second target service nodes is greater than the current number of the first target service nodes.

[0057] In this embodiment, if it is detected that there is currently a first abnormal service node with excessive load in the monitoring system, it is necessary to dynamically adjust the number of service nodes and node granularity labels in the monitoring system, and determine the first service modules with multiple different node granularity labels based on the preset first node number adjustment strategy and the node granularity label corresponding to the first abnormal node.

[0058] In some embodiments, step S140 is specifically implemented by the following steps:

[0059] According to the first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, multiple first service modules and the node granularity label corresponding to each first service module are determined; the first candidate service node corresponding to each first service module is determined in a preset node resource pool, and the corresponding first service module is deployed in each first candidate service node to obtain multiple second target service nodes, and the node granularity label of the first service module is used as the node granularity label of the corresponding second target service node.

[0060] Furthermore, the first node quantity adjustment strategy is a one-by-one incremental strategy. For example, if the current number of first target service nodes is 10, then it is necessary to determine the number of first service modules (shards) to be 11, that is, to add a new first service module (one service module is deployed in each of the original first target service nodes), and determine the node granularity label of each first service module based on the node granularity label corresponding to the first abnormal node. For the first abnormal node, it is necessary to reduce the granularity of the node, and load the reduced granularity into the newly added first service module. For example, if the node granularity label corresponding to the first abnormal node is B, at this time, the node granularity label of the service module corresponding to the first abnormal node is adjusted to B-B1, and the reduced granularity is added to the new first service module. The granularity label of the new first service module is B-B2. The granularity labels corresponding to the service modules in other service nodes that are not overloaded remain unchanged, thereby determining multiple first service modules with different node granularity labels.

[0061] In this embodiment, after determining the first service modules with multiple different granularity labels, in order to ensure the integrity and consistency of monitoring data processing, the monitoring system needs to redeploy all service nodes in each first candidate service node after determining the first service modules with multiple different node granularity labels, that is, determine the first candidate service nodes corresponding to each of the first service modules in the preset node resource pool, and deploy the corresponding first service modules in each of the first candidate service nodes to obtain multiple second target service nodes.

[0062] Specifically, determining the first candidate service nodes corresponding to each of the first service modules in a preset node resource pool includes: obtaining the available resource amount of each candidate service node in the node resource pool; and determining the first candidate service nodes corresponding to each of the first service modules from the node resource pool based on the available resource amount.

[0063] For example, after obtaining the available resource amount of each candidate service node in the node resource pool, multiple candidate service nodes with the highest available resource amount in the node resource pool are respectively determined as the first candidate service nodes corresponding to each first service module.

[0064] After determining the first candidate service nodes corresponding to each first service module, the corresponding first service module is deployed in each first candidate service node, the deployment of the first service module is completed in each first candidate service node, and the first candidate service node where the first service module is deployed is determined as the second target service node.

[0065] In this embodiment, the service modules mentioned, including the first service module, the second service module and the third service module, are all monitoring service modules, such as Prometheus service modules. Nodes deployed with the service modules have the ability to process monitoring data.

[0066] S150: Set the second target service node as the first target service node, and return to step S120.

[0067] In this embodiment, after determining the second target service node, the second target service node is determined as the second target service node, and the step of determining the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node is returned to execute until the first abnormal service node no longer exists.

[0068] Wherein, if there is no first abnormal service node, it means that all first target service nodes in the monitoring system are not overloaded.

[0069] S160: Process the plurality of monitoring data through the plurality of first target service nodes.

[0070] In this embodiment, when it is determined that the first abnormal service node does not exist, multiple monitoring data are processed by multiple first target service nodes.

[0071] Specifically, according to the association relationship between the data granularity label and the node granularity label, the multiple monitoring data are respectively allocated to the corresponding first target service nodes, and the monitoring data corresponding to each first target service node is determined; then the corresponding monitoring data are respectively analyzed and processed by each first target service node.

[0072] For example, the monitoring system assigns monitoring data 1 with a data granularity label of B-B1 and monitoring data 2 with a data granularity label of B-B2 to the target service node 2 with a node granularity label of B; assigns monitoring data 3 with a data granularity label of C-C2 to the target service node 3 with a node granularity label of C-C2; and assigns monitoring data 4 with a data granularity label of A-A1 to the target service node 1 with a node granularity label of A; after the assignment is completed, the amount of monitoring data corresponding to each target service node is determined.

[0073] In some embodiments, the monitoring system needs to monitor the resource utilization of each first target service node in real time during the process of processing monitoring data. If the resource utilization is too high (for example, the CPU utilization exceeds 80%), the load pressure of the corresponding service node needs to be reduced. If the resource utilization is too low (for example, the CPU utilization is less than 20%), the load pressure of the corresponding service node needs to be increased or the load of the node needs to be transferred to other service nodes to reduce the number of service nodes.

[0074] Specifically, whether the resource utilization rate of the service node is too high is detected in the following manner: the resource utilization rate of each first target service node is obtained; if there is a second abnormal service node among the multiple first target service nodes whose resource utilization rate is greater than the first preset resource utilization rate, then it is determined whether there is a third target service node among the multiple first target service nodes whose resource utilization rate is less than the second preset resource utilization rate, and the second preset resource utilization rate is less than the first preset resource utilization rate; if the third target service node exists, the node granularity label of the second abnormal service node and the node granularity label of the third target service node are adjusted according to the preset label adjustment strategy.

[0075] Among them, the value of the first preset resource utilization rate can be 80%, the value of the second preset resource utilization rate can be 50%, and the types of the first and second preset resource utilization rates can be CPU utilization rates. This embodiment does not limit the specific values ​​and types of the preset resource utilization rates.

[0076] For example, if it is detected that there is a service node in the detection system with a resource utilization rate as high as 85%, it means that the service node is a second abnormal service node and the pressure on the second abnormal service node needs to be reduced. At this time, if there is a third target service node with a resource utilization rate less than 50%, at this time, the load pressure of the second abnormal service node can be transferred to the third target service node by adjusting the node granularity labels of the third target service node and the second abnormal service node.

[0077] In some embodiments, if the third target service node does not exist, multiple fourth target service nodes and node granularity labels corresponding to each of the fourth target service nodes are determined based on the preset second node quantity adjustment strategy and the node granularity label corresponding to the second abnormal node, and the number of the fourth target service nodes is greater than the number of the first target service nodes; then the fourth target service node is used as the first target service node.

[0078] Specifically, first, a plurality of second service modules with different node granularity labels are determined through a preset second node quantity adjustment strategy and the node granularity label corresponding to the second abnormal node; the second candidate service nodes corresponding to each second service module are determined in the node resource pool, and the corresponding second service module is deployed in each second candidate service node to obtain a plurality of fourth target service nodes, and the node granularity label of the second service module is used as the label of the corresponding fourth target service node.

[0079] Among them, the second node quantity adjustment strategy can be a one-by-one incremental strategy. The specific implementation steps of determining multiple fourth target service nodes and the node granularity labels corresponding to each of the fourth target service nodes according to the preset second node quantity adjustment strategy and the node granularity labels corresponding to the second abnormal node can refer to the implementation steps of determining multiple second target service nodes and the node granularity labels corresponding to each of the second target service nodes in step S140, and the details will not be repeated here.

[0080] Among them, the specific implementation steps of determining the second candidate service nodes corresponding to each second service module in the node resource pool and deploying the corresponding second service module in each second candidate service node to obtain multiple fourth target service nodes can refer to the implementation steps of determining the first candidate service nodes corresponding to each first service module in the preset node resource pool and deploying the corresponding first service module in each first candidate service node in step S140, and the details are not repeated here.

[0081] In some embodiments, whether the resource utilization rate of the service node is too low is detected in the following manner: if there is a third abnormal service node (resource utilization rate is too low) among the multiple first target service nodes whose resource utilization rate is less than the third preset resource utilization rate, then multiple fifth target service nodes and the node granularity label of each fifth target service node are determined according to the preset third node quantity adjustment strategy and the node granularity label corresponding to the third abnormal node, the number of the fifth target service nodes is less than the number of the first target service nodes, and the third preset resource utilization rate is less than the second preset resource utilization rate; the fifth target service node is used as the first target service node.

[0082] Specifically, first, multiple third service modules and the node granularity label of each third service module are determined according to the preset third node quantity adjustment strategy and the node granularity label corresponding to the third abnormal node, and then the third candidate service nodes corresponding to each of the third service modules are determined in the node resource pool, and the corresponding third service modules are deployed in each of the third candidate service nodes to obtain multiple fifth target service nodes, and the node granularity label of the third service module is used as the node granularity label of the corresponding fifth service module.

[0083] The value of the third preset resource usage rate may be 20%, and the type of the third preset resource usage rate may be CPU usage rate. This embodiment does not limit the specific value and type of the third preset resource usage rate.

[0084] Specifically, this embodiment distributes the load of the third abnormal service node to other service nodes in the monitoring system by adjusting the node granularity label. Furthermore, it can be determined whether there is a fifth target service node in the monitoring system whose resource utilization rate is higher than the third preset resource utilization rate and lower than the fourth preset resource utilization rate. If there is a fifth target service node, the node granularity label of the third abnormal service node is transferred to the fifth target service node, and the third abnormal service node is removed.

[0085] The fourth preset resource usage rate may be 40%, and the type of the fourth preset resource usage rate may be CPU usage rate. This embodiment does not limit the specific value and type of the fourth preset resource usage rate.

[0086] In some embodiments, this embodiment also obtains historical monitoring data within a preset historical period; generates monitoring data volume change trend information based on the historical monitoring data; and sends the monitoring data volume change trend information to the early warning terminal.

[0087] Among them, the preset historical period can be the previous week. Users can use the data volume change trend information to estimate the data growth in the future. For example, 4 pm every day is the peak data volume period, and resources for candidate service nodes need to be prepared in advance.

[0088] To sum up, in the embodiments of the present application, when an abnormal service node with excessive load is found in the monitoring system, the number of service nodes and node granularity labels in the monitoring system can be dynamically adjusted, thereby reducing the problem of low efficiency in monitoring data processing due to excessive load on the service nodes and improving the monitoring system's processing efficiency of monitoring data.

[0089] Figure 3 This is a schematic block diagram of a data processing device for a monitoring system provided in an embodiment of the present application. Figure 3 As shown, corresponding to the above data processing method of the monitoring system, the present application also provides a data processing device 300 of the monitoring system. The data processing device 300 of the monitoring system is set in the monitoring system, and the data processing device 300 of the monitoring system includes: a transceiver unit 301 and a processing unit 302:

[0090] The transceiver unit 301 is configured to obtain a plurality of monitoring data, wherein the monitoring data carries a data granularity tag, and the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data;

[0091] The processing unit 302 is used to determine the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node, where multiple first target service nodes with different node granularity labels are deployed in the monitoring system; based on the correspondence between the first target service node and the monitoring data amount threshold, determine whether there is a first abnormal service node among the multiple first target service nodes whose monitoring data amount is greater than the corresponding monitoring data amount threshold; if the first abnormal service node exists, determine multiple second target service nodes and the node granularity label corresponding to each second target service node based on the preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, and the number of the second target service nodes is greater than the number of the first target service nodes; use the second target service node as the first target service node, and return to execute the step of determining the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node until the first abnormal service node no longer exists; and process the multiple monitoring data through the multiple first target service nodes.

[0092] In some embodiments, when executing the step of determining multiple second target service nodes and the node granularity labels corresponding to each of the second target service nodes according to the preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, the processing unit 302 is specifically configured to:

[0093] Determine, according to the first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, a plurality of first service modules and a node granularity label corresponding to each of the first service modules;

[0094] Determine the first candidate service nodes corresponding to each of the first service modules in a preset node resource pool, deploy the corresponding first service module in each of the first candidate service nodes, obtain multiple second target service nodes, and use the node granularity label of the first service module as the node granularity label of the corresponding second target service node.

[0095] In some embodiments, when executing the step of determining the first candidate service nodes corresponding to the first service modules in the preset node resource pool, the processing unit 302 is specifically configured to:

[0096] Obtain available resource amounts of each candidate service node in the node resource pool; and determine the first candidate service nodes corresponding to each first service module from the node resource pool according to the available resource amounts.

[0097] In some embodiments, when executing the step of processing the plurality of monitoring data through the plurality of first target service nodes, the processing unit 302 is specifically configured to:

[0098] According to the association between the data granularity label and the node granularity label, the plurality of monitoring data are respectively allocated to the corresponding first target service nodes, and the monitoring data corresponding to each first target service node is determined; and data analysis and processing are performed on the corresponding monitoring data through each first target service node.

[0099] In some embodiments, after executing the step of processing the plurality of monitoring data by the plurality of first target service nodes, the processing unit 302 is further configured to:

[0100] Obtain the resource utilization rate of each first target service node; if there is a second abnormal service node among the multiple first target service nodes whose resource utilization rate is greater than the first preset resource utilization rate, determine whether there is a third target service node among the multiple first target service nodes whose resource utilization rate is less than the second preset resource utilization rate, and the second preset resource utilization rate is less than the first preset resource utilization rate; if the third target service node exists, adjust the node granularity label of the second abnormal service node and the node granularity label of the third target service node according to the preset label adjustment strategy.

[0101] In some embodiments, after executing the step of determining whether there is a third target service node whose resource usage rate is less than the second preset resource usage rate among the plurality of second target service nodes, the processing unit 302 is further configured to:

[0102] If the third target service node does not exist, multiple fourth target service nodes and node granularity labels corresponding to each of the fourth target service nodes are determined according to the preset second node quantity adjustment strategy and the node granularity label corresponding to the second abnormal node, and the number of the fourth target service nodes is greater than the number of the first target service nodes; the fourth target service node is used as the first target service node.

[0103] In some embodiments, after executing the step of obtaining the resource usage rate of each first target service node, the processing unit 302 is further configured to:

[0104] If there is a third abnormal service node among the multiple first target service nodes whose resource utilization rate is less than the third preset resource utilization rate, then multiple fifth target service nodes and the node granularity label of each of the fifth target service nodes are determined according to the preset third node quantity adjustment strategy and the node granularity label corresponding to the third abnormal node, the number of the fifth target service nodes is less than the number of the first target service nodes, and the third preset resource utilization rate is less than the second preset resource utilization rate; the fifth target service node is used as the first target service node.

[0105] To sum up, the data processing device 300 of the monitoring system in the embodiment of the present application can dynamically adjust the number of service nodes and node granularity labels in the monitoring system when it is found that there is an abnormal service node with excessive load in the monitoring system, thereby reducing the problem of low efficiency in monitoring data processing caused by excessive load of the service node and improving the processing efficiency of the monitoring system for monitoring data.

[0106] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the data processing device and each unit of the above-mentioned monitoring system can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of the description, it will not be repeated here.

[0107] The data processing device of the monitoring system can be implemented in the form of a computer program. Figure 4 Runs on the computer equipment shown.

[0108] See also Figure 4 , Figure 4 4 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 400 can be a terminal or a server, and a monitoring system is installed in the computer device 400.

[0109] See Figure 4 The computer device 400 includes a processor 402 , a memory, and a network interface 405 connected via a system bus 401 , wherein the memory may include a non-volatile storage medium 403 and an internal memory 404 .

[0110] The non-volatile storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, which, when executed, can enable the processor 402 to execute a data processing method for a monitoring system.

[0111] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device 400.

[0112] The internal memory 404 provides an environment for the operation of the computer program 4032 in the non-volatile storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute a data processing method for a monitoring system.

[0113] The network interface 405 is used to communicate with other devices through the network. Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 400 to which the solution of the present application is applied. The specific computer device 400 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0114] The processor 402 is configured to execute a computer program 4032 stored in the memory to implement the following steps:

[0115] Acquire a plurality of monitoring data, wherein the monitoring data carries a data granularity tag, and the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data;

[0116] Determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, wherein a plurality of first target service nodes with different node granularity labels are deployed in the monitoring system;

[0117] Based on the correspondence between the first target service node and the monitoring data volume threshold, determining whether there is a first abnormal service node among the plurality of first target service nodes, the monitoring data volume of which is greater than the corresponding monitoring data volume threshold;

[0118] If the first abnormal service node exists, determining a plurality of second target service nodes and node granularity labels corresponding to the second target service nodes according to a preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, wherein the number of the second target service nodes is greater than the number of the first target service nodes;

[0119] Taking the second target service node as the first target service node, returning to the step of determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, until the first abnormal service node no longer exists;

[0120] The plurality of monitoring data are processed by a plurality of the first target service nodes.

[0121] It should be understood that in the embodiment of the present application, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0122] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0123] Therefore, the present application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the following steps:

[0124] Acquire a plurality of monitoring data, wherein the monitoring data carries a data granularity tag, and the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data;

[0125] Determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, wherein a plurality of first target service nodes with different node granularity labels are deployed in the monitoring system;

[0126] Based on the correspondence between the first target service node and the monitoring data volume threshold, determining whether there is a first abnormal service node among the plurality of first target service nodes, the monitoring data volume of which is greater than the corresponding monitoring data volume threshold;

[0127] If the first abnormal service node exists, determining a plurality of second target service nodes and node granularity labels corresponding to the second target service nodes according to a preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal node, wherein the number of the second target service nodes is greater than the number of the first target service nodes;

[0128] Taking the second target service node as the first target service node, returning to the step of determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, until the first abnormal service node no longer exists;

[0129] The plurality of monitoring data are processed by a plurality of the first target service nodes.

[0130] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0131] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0133] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0135] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method for a monitoring system, characterized in that: The method is applied to a monitoring system, wherein the monitoring system is used to monitor a target system, and the method includes: Acquire multiple monitoring data, wherein the monitoring data carries a data granularity tag, the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data, and the data granularity tag is a tag composed of an environment and a business line in the target system; Determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, wherein a plurality of first target service nodes with different node granularity labels are deployed in the monitoring system; Based on the correspondence between the first target service node and the monitoring data volume threshold, determining whether there is a first abnormal service node among the plurality of first target service nodes, the monitoring data volume of which is greater than the corresponding monitoring data volume threshold; If the first abnormal service node exists, determining a plurality of second target service nodes and node granularity labels corresponding to the second target service nodes according to a preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal service node, wherein the number of the second target service nodes is greater than the number of the first target service nodes; Taking the second target service node as the first target service node, returning to the step of determining the amount of monitoring data corresponding to each first target service node according to the data granularity label of each monitoring data and the node granularity label of each first target service node, until the first abnormal service node no longer exists; The plurality of monitoring data are processed by a plurality of the first target service nodes.

2. The method according to claim 1, characterized in that The determining, according to the preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal service node, a plurality of second target service nodes and a node granularity label corresponding to each of the second target service nodes, includes: Determine, according to the first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal service node, a plurality of first service modules and a node granularity label corresponding to each of the first service modules; Determine the first candidate service nodes corresponding to each of the first service modules in a preset node resource pool, deploy the corresponding first service module in each of the first candidate service nodes, obtain multiple second target service nodes, and use the node granularity label of the first service module as the node granularity label of the corresponding second target service node.

3. The method according to claim 2, characterized in that The determining, in a preset node resource pool, first candidate service nodes corresponding to the first service modules, respectively, includes: Obtaining the available resources of each candidate service node in the node resource pool; The first candidate service nodes corresponding to the first service modules are determined from the node resource pool according to the available resource amount.

4. The method according to claim 1, wherein The processing of the plurality of monitoring data by the plurality of first target service nodes includes: Allocate the plurality of monitoring data to the corresponding first target service nodes according to the association relationship between the data granularity label and the node granularity label, and determine the monitoring data corresponding to each of the first target service nodes; The corresponding monitoring data is analyzed and processed respectively by each of the first target service nodes.

5. The method according to any one of claims 1 to 4, characterized in that After the plurality of monitoring data are processed by the plurality of first target service nodes, the method further includes: Obtaining resource usage of each of the first target service nodes; If there is a second abnormal service node whose resource usage rate is greater than a first preset resource usage rate among the multiple first target service nodes, determining whether there is a third target service node whose resource usage rate is less than a second preset resource usage rate among the multiple first target service nodes, where the second preset resource usage rate is less than the first preset resource usage rate; If the third target service node exists, the node granularity label of the second abnormal service node and the node granularity label of the third target service node are adjusted according to a preset label adjustment policy.

6. The method according to claim 5, characterized in that After determining whether there is a third target service node among the plurality of first target service nodes whose resource utilization rate is less than a second preset resource utilization rate, the method further includes: If the third target service node does not exist, determining a plurality of fourth target service nodes and node granularity labels corresponding to the fourth target service nodes according to the preset second node quantity adjustment strategy and the node granularity label corresponding to the second abnormal service node, wherein the number of the fourth target service nodes is greater than the number of the first target service nodes; The fourth target service node is used as the first target service node.

7. The method according to claim 5, characterized in that After obtaining the resource usage rate of each first target service node, the method further includes: If there is a third abnormal service node among the multiple first target service nodes whose resource utilization rate is less than the third preset resource utilization rate, then determine multiple fifth target service nodes and the node granularity label of each of the fifth target service nodes according to the preset third node quantity adjustment strategy and the node granularity label corresponding to the third abnormal service node, the number of the fifth target service nodes is less than the number of the first target service nodes, and the third preset resource utilization rate is less than the second preset resource utilization rate; The fifth target service node is used as the first target service node.

8. A data processing device for a monitoring system, characterized in that: The data processing device of the monitoring system is provided in the monitoring system, and the monitoring system is used to monitor the target system. The data processing device of the monitoring system includes: a transceiver unit, configured to obtain a plurality of monitoring data, wherein the monitoring data carries a data granularity tag, wherein the data granularity tag indicates a minimum granularity source object of the corresponding monitoring data, and wherein the data granularity tag is a tag composed of an environment and a business line in the target system; A processing unit is configured to determine the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node, wherein multiple first target service nodes with different node granularity labels are deployed in the monitoring system; based on the correspondence between the first target service node and the monitoring data amount threshold, determine whether there is a first abnormal service node among the multiple first target service nodes whose monitoring data amount is greater than the corresponding monitoring data amount threshold; if the first abnormal service node exists, determine multiple second target service nodes and the node granularity label corresponding to each second target service node based on the preset first node quantity adjustment strategy and the node granularity label corresponding to the first abnormal service node, wherein the number of the second target service nodes is greater than the number of the first target service nodes; use the second target service node as the first target service node, and return to execute the step of determining the amount of monitoring data corresponding to each first target service node based on the data granularity label of each monitoring data and the node granularity label of each first target service node until the first abnormal service node no longer exists; and process the multiple monitoring data through the multiple first target service nodes.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data processing method of the monitoring system according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the data processing method of the monitoring system according to any one of claims 1 to 7.

Images