Data monitoring control system, method, device, electronic device and storage medium

Abstract

The present application relates to the field of computer technology, and in particular to a data monitoring control system, method, device, electronic device and storage medium for improving data monitoring efficiency. The system includes: multiple monitoring nodes, a time series database and a coordinator; each monitoring node is used to report each monitoring target obtained through service discovery to the coordinator; receive the monitoring target set assigned by the coordinator, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database; the time series database is used to summarize the monitoring data captured by each monitoring node. The coordinator is used to allocate a corresponding monitoring target set to each monitoring node according to the data scale of the monitoring target reported by each monitoring node and the pre-configured sharding rules, and notify each monitoring node. In the present application, the coordinator can dynamically adjust the allocation of monitoring targets in real time, which is conducive to improving data monitoring efficiency.

Description

Background Art

[0002] In the monitoring target related technologies, when deploying a monitoring service cluster, taking the Prometheus cluster as an example, one way is to deploy multiple Prometheus, each Prometheus has an independent configuration file, and each configuration file only contains a part of the collection configuration to achieve the sharding effect. However, this method requires pre-evaluation of the scale of the collection object before evenly dividing it into multiple configurations for different Prometheus to collect. If the scale of the collection object is difficult to evaluate or changes frequently, this method is not feasible. In addition, there is the problem of manual expansion and contraction.

[0003] That is to say, in the monitoring service cluster deployment method in the related art, the allocation of monitoring targets is fixed, and the data monitoring efficiency is low. Summary of the invention

[0004] The embodiments of the present application provide a data monitoring control system, method, device, electronic device and storage medium, which are used to propose a cluster deployment method of monitoring nodes, which can dynamically adjust the allocation of monitoring targets in real time and improve data monitoring efficiency.

[0005] A data monitoring and control system provided in an embodiment of the present application includes: a plurality of monitoring nodes, a time series database and a coordinator;

[0006] Each of the multiple monitoring nodes is used to obtain a monitoring target through service discovery, and report each obtained monitoring target to the coordinator; receive a monitoring target set assigned by the coordinator, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database;

[0007] The time series database is used to aggregate the monitoring data captured by each monitoring node;

[0008] The coordinator is used to obtain the data scale of each monitoring target reported by each monitoring node, allocate a corresponding monitoring target set to each monitoring node according to the data scale of each monitoring target and a pre-configured sharding rule, and notify each monitoring node.

[0009] The first data monitoring control method provided in the embodiment of the present application is applied to a monitoring node, including:

[0010] Obtain monitoring targets through service discovery, and report each obtained monitoring target to the coordinator, so that the coordinator allocates a corresponding monitoring target set to each monitoring node according to the data scale of each obtained monitoring target and the pre-configured sharding rules;

[0011] Receive the monitoring target set assigned by the coordinator, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database, so that the time series database summarizes the monitoring data captured by each monitoring node.

[0012] The second data monitoring control method provided in the embodiment of the present application is applied to the coordinator, including:

[0013] Obtain the data scale of each monitoring target reported by each monitoring node;

[0014] According to the data scale of each monitoring target and the pre-configured sharding rules, a corresponding monitoring target set is assigned to each monitoring node, and each monitoring node is notified so that after each monitoring node receives the monitoring target set assigned by the coordinator, it captures data for each monitoring target in the monitoring target set, and sends the captured monitoring data to the time series database, which summarizes the monitoring data captured by each monitoring node.

[0015] The first data monitoring control device provided in the embodiment of the present application is applied to a monitoring node, including:

[0016] A data reporting unit, used to obtain monitoring targets through service discovery, and report each obtained monitoring target to the coordinator, so that the coordinator can allocate a corresponding monitoring target set to each monitoring node according to the data scale of each acquired monitoring target and the pre-configured sharding rule;

[0017] The data capture unit is used to receive the monitoring target set assigned by the coordinator, and capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database so that the time series database can summarize the monitoring data captured by each monitoring node.

[0018] The second data monitoring control device provided in the embodiment of the present application is applied to a coordinator, including:

[0019] A data acquisition unit, used to acquire the data scale of each monitoring target reported by each monitoring node;

[0020] The target allocation unit is used to allocate a corresponding monitoring target set to each monitoring node according to the data scale of each monitoring target and the pre-configured sharding rules, and notify each monitoring node so that after each monitoring node receives the monitoring target set allocated by the coordinator, it captures data for each monitoring target in the monitoring target set, and sends the captured monitoring data to the time series database, which summarizes the monitoring data captured by each monitoring node.

[0021] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned data monitoring control methods.

[0022] An embodiment of the present application provides a computer-readable storage medium, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of any one of the above-mentioned data monitoring control methods.

[0023] The beneficial effects of this application are as follows:

[0024] The embodiments of the present application provide a data monitoring control method, device, electronic device and storage medium. The embodiments of the present application propose a coordinator that can realize dynamic allocation of monitoring targets. Specifically, the monitoring node reports the data scale of the monitoring target to the coordinator, and then the coordinator can summarize the actual data scale of each monitoring target reported by each monitoring node, and allocate it to each monitoring node according to the actual data scale of the monitoring target. Compared with the related art, the allocation of monitoring targets in the embodiments of the present application is based on the data scale of each monitoring target, rather than being averagely distributed according to the scale obtained by pre-evaluation. It can be dynamically adjusted when the system is running, and the problem of excessive load on a single shard will not occur, which is conducive to improving data monitoring efficiency.

[0025] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of the structure of a data monitoring and control system in an embodiment of the present application;

[0029] Figure 3 This is an overall architecture diagram of a data monitoring and control system in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a configuration injection method in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a data capture request proxy method in an embodiment of the present application;

[0032] Figure 6 A flowchart of a method for allocating monitoring targets based on sharding rules in an embodiment of the present application;

[0033] Figure 7 This is a flow chart of the first data monitoring control method in an embodiment of the present application;

[0034] Figure 8 Schematic diagram of the flow of the second data monitoring control method in the embodiment of the present application;

[0035] Fig. 9 This is a schematic diagram of an optional interaction implementation timing flow in an embodiment of the present application;

[0036] Fig.10 This is a schematic diagram of the composition structure of the first data monitoring control device in the embodiment of the present application;

[0037] Fig.11 This is a schematic diagram of the structure of the second data monitoring control device in the embodiment of the present application;

[0038] Fig.12 The present invention is a schematic diagram of the structure of an electronic device to which the present invention is applied. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0040] The following is an introduction to some concepts involved in the embodiments of the present application.

[0041] Monitoring server: a server used for data monitoring, such as a Prometheus server. Prometheus can provide flexible PromSql for querying monitoring data. Among them, PromSql is the database query syntax of Prometheus, which can easily find the value of the monitoring indicator and do some mathematical operations to quickly and conveniently collect statistics and set alarm thresholds. Moreover, Promsql queries the time series database, so even if it queries historical data, it is very fast. The monitoring node in the embodiment of the present application includes a Prometheus server (also referred to as Prometheus in this article).

[0042] Prometheus job: Prometheus divides the collection tasks into jobs. Each job has its own name, service discovery configuration, indicator filtering configuration, etc. Among them, indicator filtering means that for personnel with different functions (development, testing, operation and maintenance), they may only care about part of the monitoring data. They may deploy their own Prometheus Server to monitor the indicator data they are concerned about. Unnecessary data needs to be filtered out to avoid wasting resources. Similar configuration can be made.

[0043] Monitoring target: refers to the monitoring targets obtained by the job through service discovery, specifically the crawling objects of Prometheus. Prometheus will periodically send Get (receive) requests to all monitoring targets to crawl monitoring data.

[0044] Service discovery: Simply put, it is to find the instance address and port that provides the service through the service name, which is mainly used to solve the problem of how to obtain the service instance address. With the rise of container technology, a large number of services are scattered throughout the system, and calls between services need to be implemented through service discovery. Service discovery is an indispensable key component in distributed systems and is often used to build open source frameworks for service discovery solutions. In the embodiment of the present application, Prometheus obtains the monitoring target through service discovery.

[0045] Sidecar: It is a special container process in Pod (container group), which usually does not contain core business functions, but assists the core business container process to complete work, such as network proxy, configuration file generation, etc. In the embodiment of the present application, the container process server can be a Sidecar server. Each Prometheus in the present application corresponds to a Sidecar. The proxy configuration is injected into the original configuration file through the Sidecar, so that Prometheus will proxy all crawling requests to the Sidecar in the http (Hyper Text Transfer Protocol) protocol, and the Sidecar will perform data crawling.

[0046] Pod: It is the smallest scheduling unit in Kubernetes (K8s for short). A Pod consists of one or more container processes. These container processes share network space and can also share some directories through configuration. Among them, Kubernetes is a system for running and coordinating containerized applications on a group of hosts, providing mechanisms for application deployment, planning, updating and maintenance. The application runs on the kubernetes cluster to achieve service expansion and reduction, perform rolling updates, and schedule traffic between different versions of applications to test functions or roll back problematic deployments. In an embodiment of the present application, Sidecar, Prometheus and coordinator can be deployed in Pod mode or directly on a physical machine. Among them, a physical machine is a name for a physical computer relative to a virtual machine. The physical machine provides a hardware environment for the virtual machine.

[0047] TSDB (Time Series Database) is a database service that integrates efficient reading and writing of time series data, compressed storage, and real-time computing capabilities. It is mainly used to store time series data and can be widely used in the field of Internet of Things and the Internet to achieve real-time monitoring of devices and business services, and real-time prediction and alarm. In the embodiment of this application, TSDB is used to aggregate the monitoring data of each Prometheus.

[0048] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0049] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0050] The embodiments of the present application mainly relate to the technical field of cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0051] The data monitoring and control system provided in the embodiment of the present application supports the cloud-based Prometheus service for ultra-large-scale data collection to realize the cloud-hosted Prometheus service. Among them, Sidecar, Prometheus and coordinator can be deployed in Pod mode, and one Pod corresponds to one Prometheus and Sidecar. In addition, Sidecar, Prometheus and coordinator can also be deployed directly on a physical machine. The monitoring server and container process server in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0052] The following is a brief introduction to the design concept of the embodiment of the present application:

[0053] Prometheus is a monitoring and alarm system based on a time series database. It has very low business intrusion and excellent performance in recording pure numbers based on time series. This makes it suitable for monitoring hardware indicators such as servers, as well as for monitoring highly dynamic service-oriented architectures.

[0054] However, native Prometheus does not provide a clustering solution. At present, when deploying a Prometheus cluster, in addition to the method introduced in the background technology, there is also a common implementation method, which is to add a hash modulus (hashmod) method to the configuration file to scatter the monitoring targets obtained by service discovery according to the value of a certain label (label), so that different Prometheus can collect different sets of monitoring targets to achieve a sharding effect. This method also invades the configuration file. It is necessary to add hashmod to the original configuration, and it is necessary to select a label for hashmod for each job. In addition, the hashmod method can only be used to shard according to the value of the label, and it is impossible to shard according to the actual monitoring target data scale, so the load of each shard may vary greatly, resulting in a performance bottleneck in a single shard. In addition, there is also the problem of manual expansion and contraction.

[0055] In view of this, the embodiments of the present application propose a data monitoring control system, method, device, electronic device and storage medium. In the embodiments of the present application, a coordinator that can realize dynamic allocation of monitoring targets is proposed. Specifically, the monitoring node reports the data scale of the monitoring target to the coordinator, and then the coordinator can summarize the actual data scale of each monitoring target reported by each monitoring node, and allocate it to each monitoring node according to the actual data scale of the monitoring target. Compared with the related art, the allocation of monitoring targets in the embodiments of the present application is based on the data scale of each monitoring target, rather than being averagely distributed according to the scale obtained by pre-evaluation. It can be dynamically adjusted when the system is running, and the problem of excessive load on a single shard will not occur, which is conducive to improving data monitoring efficiency.

[0056] In addition, the monitoring node in the embodiment of the present application may include a monitoring server and a container process server, wherein the monitoring server is mainly illustrated by taking the Prometheus server as an example, and the container process server is mainly illustrated by taking the Sidecar server as an example (hereinafter referred to as Sidecar). First, a Sidecar is added to the Prometheus Pod, and the Pod is called a shard. The Sidecar injects the proxy configuration into the original configuration file (i.e., the existing configuration file), so that Prometheus will proxy all crawling requests to the Sidecar using the http protocol. The Sidecar will choose to use the network protocol and certificate information in the original configuration file to send a crawling request to the real monitoring target or return an empty result based on the list of monitoring targets that this shard is responsible for, thereby realizing a Prometheus clustering method based on configuration file injection and monitoring target filtering.

[0057] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0058] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120, and the terminal devices 110 and the server 120 can communicate through a communication network. Each terminal device corresponds to a user, wherein the user corresponding to the terminal device on the left is user A, and the user corresponding to the terminal device on the right is user B.

[0059] In an optional implementation, the communication network is a wired network or a wireless network. The terminal device 110 and the server 120 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0060] In the embodiment of the present application, the terminal device 110 is an electronic device used by the user, which can be a personal computer, mobile phone, tablet computer, notebook, e-book reader, smart home, etc., which has certain computing power and runs instant messaging software and websites or social software and websites. Each terminal device 110 is connected to the server 120 through a wireless network. The server 120 can be an independent physical server, or a server 120 cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. For example, the coordinator, Prometheus, and Sidecar in the embodiments of the present application may be deployed on a Pod, in which case the coordinator, Prometheus server, and Sidecar server may be cloud servers; in addition, the coordinator, Prometheus, and Sidecar may also be directly deployed on a physical machine, in which case the coordinator, Prometheus server, and Sidecar server may be physical servers, and the Prometheus server and Sidecar server may be two independent physical servers or the same physical server, etc., without specific limitation herein.

[0061] Among them, an application can be installed in each terminal device. The application involved in the embodiment of the present application can be software or a client such as a web page or a mini program. The server is a server corresponding to the software or web page, mini program, etc., and the specific type of the client is not limited.

[0062] In the embodiment of the present application, the monitoring target of Prometheus can be an application. Prometheus provides SDKs (Software Development Kits) in various languages, which users can easily integrate into applications to effectively monitor the internal state of applications. In addition, Prometheus can also be used to monitor container resources such as k8s and docker (application container engine). In the embodiment of the present application, the monitoring target of the monitoring node is not specifically limited.

[0063] See also Figure 2 As shown, it is a schematic diagram of the composition structure of a data monitoring and control system 200 provided in an embodiment of the present application, the system includes: a plurality of monitoring nodes 210, a time series database 220 and a coordinator 230;

[0064] Each monitoring node 210 in the plurality of monitoring nodes 210 is used to obtain a monitoring target through service discovery, and report each obtained monitoring target to the coordinator 230; receive a monitoring target set assigned by the coordinator 230, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database 220;

[0065] Among them, after the monitoring node obtains the monitoring target through service discovery, when reporting these monitoring targets to the coordinator for the first time, it reports the basic information of these monitoring targets, such as the network address and configuration of the monitoring target. Then the coordinator detects based on the basic information of each monitoring target, obtains the data scale of each monitoring target, and allocates the monitoring target set to the monitoring node. The monitoring node captures data for each monitoring target in the monitoring target set and updates the data scale of these monitoring targets. Afterwards, the monitoring node reports the updated data scale of each monitoring target to the coordinator.

[0066] The time series database 220 is used to aggregate the monitoring data captured by each monitoring node 210;

[0067] The coordinator 230 is used to obtain the data scale of each monitoring target reported by each monitoring node 210, allocate a corresponding monitoring target set to each monitoring node 210 according to the data scale of each monitoring target and a pre-configured sharding rule, and notify each monitoring node 210.

[0068] The first data scale of all monitoring targets is detected by the coordinator, and then these monitoring targets are assigned to a monitoring node by the coordinator. The subsequent data scale of these monitoring targets is updated in real time by the monitoring node each time it captures the monitoring data. The subsequent monitoring node can report the latest data scale to the coordinator.

[0069] Therefore, when the coordinator obtains the data scale of each monitoring target reported by each monitoring node, it can be divided into the following two cases:

[0070] Case 1: receiving each monitoring target reported by each monitoring node, and detecting the data scale of each monitoring target; after allocating a corresponding monitoring target set to each monitoring node, sending the data scale of each monitoring target in the monitoring target set to the corresponding monitoring node.

[0071] Case 2: receiving the updated data scale of each monitoring target reported by each monitoring node.

[0072] In the above implementation, the monitoring node reports the data scale of the monitoring target to the coordinator, and then the coordinator can summarize the actual data scale of each monitoring target reported by each monitoring node, and allocate it to each monitoring node according to the actual data scale of the monitoring target. Compared with the related art, the allocation of monitoring targets in the embodiment of the present application is based on the data scale of each monitoring target, rather than being evenly distributed according to the scale obtained by pre-evaluation. It can be dynamically adjusted when the system is running, and the problem of excessive load on a single shard will not occur, which is conducive to improving data monitoring efficiency.

[0073] Optionally, each monitoring node 210 includes a monitoring server 211 and a container process server 212. Figure 2 As shown, where:

[0074] The monitoring server 211 is used to initiate a data capture request for a monitoring target to be captured, and proxy the data capture request to the corresponding container process server 212 based on a pre-configured target configuration file; and receive a data capture result for the data capture request returned by the container process server 212;

[0075] The container process server 212 is used to report the data scale of each monitoring target obtained to the coordinator; after receiving the data capture request, it captures data according to the monitoring target set, obtains the data capture result for the data capture request, and returns the data capture result to the monitoring server 211.

[0076] See also Figure 3As shown, it is an overall architecture diagram of a data monitoring and control system listed in the embodiment of the present application. Among them, the data monitoring and control system represents a monitoring service cluster, which includes 3 monitoring nodes, 1 coordinator and 1 time series database. In the embodiment of the present application, taking the monitoring service cluster such as the Prometheus cluster as an example, one Prometheus cluster corresponds to one coordinator, and each monitoring node can be used as a shard (hereinafter referred to as a shard, i.e., a monitoring node in the embodiment of the present application), for example Figure 3 Shard 1, Shard 2, and Shard 3 in the diagram. The monitoring server shown in the figure can be a Prometheus server, and the container process server is the Sidecar server.

[0077] in, Figure 3 The following is an example of three monitoring targets, namely, monitoring target 1, monitoring target 2, and monitoring target 3. It should be noted that there may be more monitoring nodes and monitoring targets. In addition, new monitoring targets may be discovered over time, and additional monitoring nodes may need to be deployed. This is just an example, and there is no specific limit on the number of monitoring targets, monitoring nodes, etc. Figure 3 The above process is introduced in detail.

[0078] The coordinator collects information such as the data size of the monitoring targets from each Sidecar, and assigns each Sidecar a list of monitoring targets that it needs to collect. Figure 3 As shown in the figure, in the first cycle, each Sidecar obtains the monitoring target based on service discovery by calling the corresponding Prometheus, and then reports the basic information of the monitoring target to the coordinator. For example, "I found monitoring target 1, monitoring target 2, and monitoring target 3". In subsequent cycles, each Sidecar will capture data, record the data scale of the monitoring target, and then report it to the coordinator.

[0079] For example, after each sidecar reports the data scale of the three monitoring targets to the coordinator, the coordinator collects the data scale of the monitoring targets reported by all sidecars and summarizes them. Then, the coordinator knows the latest data scale of the monitoring targets being collected, etc. If new monitoring targets appear, the coordinator knows which monitoring targets need to be detected, and then further detects the data scale of the new monitoring targets. Figure 3The three monitoring targets are used as examples for the introduction. Based on the data scale of monitoring target 1, monitoring target 2, and monitoring target 3, and the pre-configured sharding rules (described in detail below), the coordinator allocates a corresponding monitoring target set to each monitoring node. The monitoring target set contains at least one monitoring target for which the monitoring node is responsible for capturing monitoring data. Afterwards, the coordinator notifies each monitoring node of the allocation result.

[0080] exist Figure 3 In the example, the monitoring target assigned by the coordinator to shard 1 is monitoring target 1, the monitoring target assigned to shard 2 is monitoring target 2, and the monitoring target assigned to shard 3 is monitoring target 3. When crawling data, Prometheus periodically initiates data crawling requests to all monitoring targets. Each Sidecar proxies the corresponding Prometheus data crawling request to itself, and chooses to collect monitoring data or return a null value to Prometheus according to the list of monitoring targets it is responsible for, thereby achieving the effect of monitoring target sharding. Finally, all Prometheus sends the acquired monitoring data to Thanos, which is summarized by Thanos.

[0081] In an optional implementation, when Sidecar chooses to collect monitoring data or return a null value to Prometheus based on the list of monitoring targets it is responsible for, it can be divided into the following two cases:

[0082] Case 1: When the monitoring target to be captured is in the monitoring target set, the container process server sends a data capture request to the monitoring target to be captured; captures monitoring data from the monitoring target to be captured, and returns the captured monitoring data to the monitoring server.

[0083] Taking shard 1 as an example, if Prometheus in shard 1 initiates a data crawling request to monitoring target 1, the monitoring target to be crawled is monitoring target 1. At this time, Prometheus proxies the data crawling request to Sidecar in shard 1. At this time, the monitoring target list responsible for Sidecar includes monitoring target 1, so Sidecar sends the data crawling request to monitoring target 1, and then crawls the monitoring data from monitoring target 1 and returns the crawled monitoring data to the monitoring server.

[0084] Case 2: When the monitoring target to be captured is not in the monitoring target set, the container process server returns a null value to the monitoring server.

[0085] Still taking shard 1 as an example, if Prometheus in shard 1 initiates a data crawling request to monitoring target 2, the monitoring target to be crawled is monitoring target 2. At this time, Prometheus proxies the data crawling request to Sidecar in shard 1. At this time, the monitoring target list responsible for Sidecar includes monitoring target 1, but not monitoring target 2. Therefore, Sidecar no longer crawls monitoring data from monitoring target 2, and directly returns a null value to the monitoring server, that is, a null result.

[0086] That is to say, when Sidecar crawls data based on the monitoring target set, it only crawls data for the monitoring targets that are currently in the monitoring target set, that is, it is only responsible for monitoring targets in the monitoring target set. When Prometheus initiates a data crawling request, it may initiate a data crawling request for all discovered monitoring targets, so Sidecar may return an empty result to Prometheus.

[0087] It should be noted that in the embodiment of the present application, the monitoring target set assigned by the coordinator to each monitoring node will be dynamically adjusted during operation, such as periodically collecting information such as the data scale of the monitoring target and assigning the monitoring target set to each monitoring node. The monitoring target set, data capture request, etc. listed in this article refer to the current cycle.

[0088] In addition, considering that the data originally collected by a single Prometheus is dispersed into various shards in the embodiment of the present application, the data of any shard is incomplete, and thus a time series database is used to aggregate the data, perform unified queries, and obtain a global view.

[0089] Figure 3 The time series database shown may be Thanos. In the embodiment of the present application, Thanos is a time series database used to aggregate and query the data of multiple Prometheus to obtain a single data view, and supports uploading the data of Prometheus from the local to the object storage for long-term storage. Its main function is to achieve data persistence and historical data extraction.

[0090] In addition, other TSDB products such as cortex (processor), influxdb (time series library) etc. may be used instead of Thanos. For the embodiment of the present application, it is only necessary to add a remote writer (remote writing program) to the configuration file.

[0091] After introducing the above embodiment, the container process server in the embodiment of the present application will be introduced in detail below.

[0092] The Prometheus cluster deployment methods listed in the related art are all quite intrusive to the configuration files. For example, each Prometheus uses an independent configuration file, and the monitoring targets that Prometheus needs to collect are pre-configured in each Prometheus configuration file. In this way, the configuration files are divided into multiple parts, and the management is relatively complicated.

[0093] Considering that Sidecar can realize network proxy, configuration file generation, etc., Sidecar is introduced in the embodiment of the present application. A Sidecar is added to each Prometheus Pod. The Sidecar injects the proxy configuration into the existing configuration file, so that Prometheus will proxy all data crawling requests to Sidecar via the http protocol.

[0094] Optionally, when the Prometheus server proxies the data crawling request to its corresponding Sidecar server, it is mainly implemented based on a pre-configured target configuration file, which includes injection information for implementing the data crawling request proxy, specifically including proxy indication information and the network address information of the Sidecar server. Based on the Sidecar server, the injection information for implementing the data crawling request proxy is added to the original configuration file provided in the related technology, and the target configuration file for the Prometheus server is generated based on the injection information.

[0095] Combine the following Figure 4 and Figure 5 The proxy process of data crawling requests is introduced in detail:

[0096] in, Figure 4 Schematic diagram of a configuration injection method in an embodiment of the present application; wherein, Figure 4 The left part is the original configuration file, which includes the crawling interval, evaluation interval, monitoring target configuration, static configuration, etc.

[0097] The crawling interval is 15 seconds and the evaluation interval is 15 seconds. The monitoring target configuration includes the crawling task name, for example Figure 4 The name shown is test, and it also includes the configuration of the request protocol type. In the original configuration file, the data capture request is https (Hypertext Transfer Protocol Secure); the static configuration specifically includes the network address information of the monitoring target, etc. For example, "10.110.110.110:8080" means that the IP (Internet Protocol) address of the monitoring target is 10.110.110.110, and the port number is 8080.

[0098] Specifically, Prometheus crawls monitoring data from the monitoring target with a crawling interval (15s) rule period. The crawling interval can be defined on a global basis or on a single metric basis; then the monitoring data is stored. Prometheus performs regular calculations on the alarm rules with another independent rule period of an evaluation interval (15s). The evaluation interval only has a global value; then the alarm status is updated. It should be noted that the alarm part is not introduced in detail in the embodiments of this application, and no specific limitation is made. It is just an example here.

[0099] In addition, you can define the data crawling request as https format through "scheme:https". Among them, scheme is a page jump protocol, which is a very good implementation mechanism. By defining your own scheme protocol, you can easily jump to various pages in the app (Appliaction).

[0100] In an embodiment of the present application, Sidecar can generate a new configuration file that Prometheus actually uses on the shard by injecting necessary information based on the original configuration file, that is, the target configuration file, such as Figure 4 As shown in the figure, the newly injected information of Sidecar in the original configuration file includes: 1. proxy instruction information for proxying data crawling requests to Sidecar and network address information of Sidecar; 2. changing https to http; 3. saving task information, etc. Among them, all schemes are set to http, so that data crawling requests are proxied to Sidecar in http format; the job name is added to the new configuration file to ensure that detailed distinction can be made when receiving data crawling requests.

[0101] It should be noted that in the embodiment of the present application, the configuration injection of Sidecar does not change the original configuration file, but regenerates a configuration for Prometheus, which is non-intrusive to the original configuration file. The user still modifies and sees the original configuration file, and the configuration file seen by all shards is the same, so there is no problem of complex configuration management.

[0102] See also Figure 5As shown, it is a schematic diagram of a data capture request proxy method listed in an embodiment of the present application; wherein, the original configuration file is the original configuration file that needs to be added by the Sidecar to inject information, and the new configuration file is the target configuration file newly generated by the Sidecar based on the injected information. In the embodiment of the present application, the original configuration files loaded by each Sidecar are unified, but different Sidecars add different injection information to the original configuration file, and the target configuration files generated based on this are also different, and each target configuration file has its own corresponding Prometheus server.

[0103] In an embodiment of the present application, the original configuration file includes the network address information of the monitoring target to be captured; for example, "monitoring target: 10.10.10.10:8080" means the network address information of a monitoring target to be captured. When the Prometheus server proxies the data capture request to the Sidecar, it specifically proxies the data capture request to the Sidecar server corresponding to the injection information in the target configuration file via the http protocol; the Sidecar server is specifically used to capture data according to the network address information of the monitoring target to be captured in the original configuration file after receiving the data capture request.

[0104] Still Figure 5 Take the original configuration file and target configuration file shown as examples, where the IP address of Sidecar is 127.0.0.1 and the port number is 8008; the IP address of the monitoring target to be captured is 10.10.10.10 and the port number is 8080.

[0105] First, Sidecar loads the original configuration file, adds injection information to the original configuration file, and generates the target configuration file. Then Prometheus loads the target configuration file, and sends the data crawling request to the Sidecar with the network address information of 127.0.0.1:8008 in http mode according to the network address information and job name of Sidecar in the target configuration file. Sidecar then determines whether to use the original configuration file for actual crawling based on whether this shard is responsible for this monitoring target (network address information: 10.10.10.10:8080). If the monitoring data is crawled from the monitoring target, the data size of the monitoring target is counted and the monitoring data is returned to Prometheus.

[0106] In the above implementation, the Sidecar injects the proxy configuration into the original configuration file, so that Prometheus will proxy all data crawling requests to the Sidecar using the http protocol. The Sidecar will choose to use the network protocol and certificate information in the original configuration file to send crawling requests to the real monitoring target or return an empty result based on the list of monitoring targets that this shard is responsible for.

[0107] After introducing the above embodiment, the coordinator in the embodiment of the present application will be described in detail. In the embodiment of the present application, the coordinator is responsible for distributing the monitoring target list allowed to be collected to all slices. In order to make the load of each slice as balanced as possible, the following is combined with Figure 6 The sharding method in the embodiment of the present application is introduced in detail.

[0108] See also Figure 6 As shown, it is a flow chart of a method for allocating monitoring targets based on sharding rules listed in an embodiment of the present application, which specifically includes the following steps:

[0109] First, let's introduce the pre-steps: the coordinator collects monitoring target information from all shards (i.e. monitoring nodes), where the monitoring target information mainly includes the following two parts:

[0110] 1. All monitoring targets obtained by each shard through service discovery;

[0111] 2. The monitoring targets that each shard is collecting and the data size of each monitoring target that has been recently counted, such as the average of the data size of the last three statistics.

[0112] In an embodiment of the present application, the coordinator can obtain the total data size of each monitoring target currently captured by each monitoring node through the previous step. Next, the monitoring target is segmented mainly based on this information and preset segmentation rules.

[0113] Optionally, a data scale threshold is preset, referred to as the preset threshold in this article. Based on the size relationship between the preset threshold and the sum of the data scales of each monitoring node, the monitoring nodes are divided into two categories, namely the first category of monitoring nodes and the second category of monitoring nodes. Among them, the first category of monitoring nodes refers to the monitoring nodes whose sum of the data scales of all monitoring targets currently being collected exceeds the preset threshold; the second category of monitoring nodes refers to the monitoring nodes whose sum of the data scales of all monitoring targets currently being collected does not exceed the preset threshold. For these two types of monitoring nodes, an optional sharding rule is:

[0114] For the first-category monitoring nodes whose total data scale exceeds the preset threshold, after selecting at least one candidate monitoring target from the monitoring targets currently captured by the first-category monitoring node, the set consisting of all remaining monitoring targets is used as the monitoring target set of the first-category monitoring node, wherein the sharding rule includes that the total data scale of each monitoring target in the monitoring target set of the first-category monitoring node is not less than the preset threshold.

[0115] For example, for shard 1, the monitoring targets currently being collected by the shard are monitoring target 1, monitoring target 2, monitoring target 3, monitoring target 4, and monitoring target 5. The total data size of these five monitoring targets exceeds the preset threshold. At this time, shard 1 belongs to the first type of monitoring node, and at least one monitoring target can be selected from these five monitoring targets as a candidate monitoring target. Afterwards, the set composed of the remaining monitoring targets is used as the monitoring target set corresponding to shard 1.

[0116] Among them, when selecting at least one monitoring target from these five monitoring targets as a candidate monitoring target, it is necessary to meet such a sharding rule, that is, except for the selected monitoring target, the total data scale of the remaining monitoring targets is still not less than the preset threshold.

[0117] Assuming that at least one monitoring target is selected from the above five monitoring targets as a candidate monitoring target, there are three possible situations:

[0118] Case 1: When monitoring target 3 is selected, the total data size T1 of the remaining 4 monitoring targets is not less than the preset threshold. At this time, monitoring target 3 can be used as a candidate monitoring target.

[0119] Case 2: when monitoring target 4 is selected, the total data size T2 of the remaining 4 monitoring targets is also not less than the preset threshold. At this time, monitoring target 4 can be used as a candidate monitoring target.

[0120] Case 3: When monitoring target 1 and monitoring target 2 are selected, the total data size T3 of the remaining three monitoring targets is not less than the preset threshold. At this time, monitoring target 1 and monitoring target 2 can also be used as candidate monitoring targets.

[0121] When there are multiple situations listed above, the greedy algorithm can be used to find the optimal solution and select the candidate monitoring target corresponding to the situation that meets the optimal solution. Among them, the greedy algorithm (also known as the greedy algorithm) means that when solving a problem, it always makes the best choice at the moment. In other words, it does not consider the overall optimal solution, and it only makes a local optimal solution in a certain sense. The greedy algorithm cannot obtain the overall optimal solution for all problems, but it can produce the overall optimal solution or an approximate solution to the overall optimal solution for many problems with a wide range.

[0122] For example, in the embodiment of the present application, the optimal solution indicates that the sum of the data scales of the monitoring targets taken out is the largest. At this time, if T1>T2>T3, the sum of the data scales of the monitoring targets taken out in the first case is the largest, so monitoring target 3 is selected as the candidate monitoring target. For another example, if the optimal solution indicates that the number of monitoring targets taken out is the largest, at this time, the number of monitoring targets taken out in the third case is the largest, so monitoring target 1 and monitoring target 2 are selected as candidate monitoring targets.

[0123] Assuming that the candidate monitoring targets are selected using the method of situation 3, the monitoring target set corresponding to shard 1 is {monitoring target 3, monitoring target 4, monitoring target 5}.

[0124] It should be noted that the above-listed method of selecting at least one candidate monitoring target from the monitoring targets currently captured by the first type of monitoring node is only an example, and the specific method may be determined in reference to the actual situation and is not specifically limited here.

[0125] In an embodiment of the present application, for the second-category monitoring node, since the total data scale of each monitoring target currently captured by it does not exceed the preset threshold, the candidate monitoring target selected by the above process can be pre-allocated to the second-category monitoring node. If the sharding rules in the embodiment of the present application are met after pre-allocation, that is, the total data scale of each monitoring target in the monitoring target set of the second-category monitoring node does not exceed the preset threshold, and when the pre-allocated candidate monitoring node is added, the total data scale of each monitoring target and the candidate monitoring target does not exceed the preset threshold, then the candidate monitoring target can be actually allocated to the corresponding second-category monitoring node. At this time, the set consisting of each monitoring target currently captured by the second-category monitoring node and at least one candidate monitoring target is used as the monitoring target set of the second-category monitoring node.

[0126] If it is not satisfied, that is, when the sum of the data scales currently corresponding to all second-class monitoring nodes exceeds the preset threshold after adding the candidate monitoring target, the candidate monitoring target is added to the monitoring target set corresponding to the original first-class monitoring node. At this time, the set composed of the monitoring targets currently captured by the second-class monitoring node is used as the monitoring target set of the second-class monitoring node.

[0127] For example, for shard 2, the monitoring targets currently being collected by the shard include monitoring target 6, monitoring target 7, and monitoring target 8, wherein the sum of the data scales of these three monitoring targets is less than the preset threshold. For shard 3, the monitoring targets currently being collected by the shard include monitoring target 9 and monitoring target 10, wherein the sum of the data scales of these two monitoring targets is also less than the preset threshold. At this point, shard 2 and shard 3 belong to the second type of monitoring nodes. Further, the candidate monitoring nodes monitoring target 1 and monitoring target 2 selected from shard 1 can be pre-allocated.

[0128] For example, when monitoring target 1 is pre-allocated to shard 2, the total data scale of the four monitoring targets, monitoring target 1, monitoring target 6, monitoring target 7, and monitoring target 8, still does not exceed the preset threshold. When monitoring target 2 is pre-allocated to shard 3, the total data scale of the three monitoring targets, monitoring target 2, monitoring target 9, and monitoring target 10, also does not exceed the preset threshold. At this time, it can be determined that the set of monitoring targets allocated to shard 2 is {monitoring target 1, monitoring target 6, monitoring target 7, monitoring target 8}, and the set of monitoring targets allocated to shard 3 is {monitoring target 2, monitoring target 9, monitoring target 10}. Correspondingly, the set of monitoring targets corresponding to shard 1 is {monitoring target 3, monitoring target 4, monitoring target 5}.

[0129] However, there may be another situation. Suppose that when monitoring target 1 is pre-allocated to shard 2, the total data scale of the four monitoring targets, monitoring target 1, monitoring target 6, monitoring target 7 and monitoring target 8, still does not exceed the preset threshold. However, when monitoring target 2 is also pre-allocated to shard 2, the total data scale of the five monitoring targets, monitoring target 1, monitoring target 2, monitoring target 6, monitoring target 7 and monitoring target 8, exceeds the preset threshold. Moreover, when monitoring target 2 is pre-allocated to shard 3, the total data scale of the three monitoring targets, monitoring target 2, monitoring target 9 and monitoring target 10, also exceeds the preset threshold. This indicates that the total data scale currently corresponding to all second-category monitoring nodes exceeds the preset threshold after adding the candidate monitoring target. At this time, monitoring target 2 can be put back to the original shard 1. Finally, the monitoring target set corresponding to shard 1 is {monitoring target 2, monitoring target 3, monitoring target 4, monitoring target 5}, the monitoring target set corresponding to shard 2 is {monitoring target 1, monitoring target 6, monitoring target 7, monitoring target 8}, and the monitoring target set corresponding to shard 3 is {monitoring target 9, monitoring target 10}.

[0130] It should be noted that the above implementation methods are only examples, and any optional distribution method is applicable to the embodiments of the present application. They will not be listed one by one here. The specific method can be determined by reference to the actual situation, such as preferably preferring a more load-balanced distribution method.

[0131] For the specific process of allocating monitoring targets for the above two types of monitoring nodes, please refer to Figure 6 Steps one and two are shown.

[0132] Step 1: For each existing shard, if the total scale of its current monitoring targets exceeds the preset threshold, try to remove the redundant monitoring targets from the shard. The premise for removal is that the total scale of the monitoring targets after removal is not less than the preset threshold.

[0133] Step 2: Try to allocate all the monitoring targets just taken out to other shards, provided that the total size of the monitoring targets of the allocated shards does not exceed the preset threshold after adding the pre-allocated monitoring targets. If no shard meets the requirements, put the monitoring targets back to the original shards.

[0134] Specifically, in step one, the premise for taking out is that "the total size of the monitored targets after taking out is not less than the preset threshold", which can effectively control the memory. Because once it becomes a non-overflow state (that is, the total size does not exceed the preset threshold), the gap between this shard and the preset threshold is unknown, and it is possible to add other monitoring targets later. Once this is done, the memory will increase dramatically.

[0135] In the above implementation, the coordinator can periodically obtain all monitoring target configurations and data scale information from all Sidecars, and distribute the monitoring targets with known data scales and captured by the monitoring nodes to all Sidecars based on the above sharding rules. Dynamic adjustment is made during runtime, and the problem of excessive load on a single shard will not occur.

[0136] Optionally, the monitoring targets in the allocation methods listed in the above embodiments are all monitoring targets being collected by the monitoring nodes. For the monitoring targets that are not currently captured by any monitoring node, the coordinator is also used to: detect whether the data scale of these monitoring targets has been detected; if so, allocate the monitoring target to the monitoring node with the smallest total data scale of each monitoring target in the currently corresponding monitoring target set, wherein the monitoring target set corresponding to the monitoring node with the smallest total data scale has a total data scale of each monitoring target that does not exceed a preset threshold after the monitoring target is added; otherwise, the monitoring target will not be allocated in this cycle.

[0137] For details of the above process, please refer to Figure 6 Step 3 in:

[0138] For monitoring targets that are not currently collected by any shards, first check whether their data scale has been detected.

[0139] If it is not detected, the coordinator will perform detection later, and the monitoring target will not participate in the allocation in this cycle.

[0140] If it has been detected, try to assign it to the shard with the smallest data size of the current monitoring target, provided that the data size of the shard does not exceed the preset threshold after adding the monitoring target.

[0141] For example, for the newly added monitoring target 11, the monitoring target is not currently collected by any monitoring node. At this time, the coordinator first needs to check whether the data scale of the monitoring target has been detected. Assuming that the data scale of the monitoring target 11 is detected, it can be assigned to the one with the smallest data scale among the three shards, shard 1, shard 2 and shard 3. For example, the monitoring target set corresponding to shard 1 is {monitoring target 2, monitoring target 3, monitoring target 4, monitoring target 5}, and the sum of the data scales of all monitoring targets in this set is T4; the monitoring target set corresponding to shard 2 is {monitoring target 1, monitoring target 6, monitoring target 7, monitoring target 8}, and the sum of the data scales of all monitoring targets in this set is T5; the monitoring target set corresponding to shard 3 is {monitoring target 9, monitoring target 10}, and the sum of the data scales of all monitoring targets in this set is T6.

[0142] Among them, T4>T5>T6, the shard 3 has the smallest total data size, and after adding monitoring target 11 to shard 3, the total data size is T7, and T7 is less than the preset threshold, so the monitoring target can be assigned to shard 3, and finally the {monitoring target 9, monitoring target 10, monitoring target 11} corresponding to shard 3 can be determined. If T7 exceeds the preset threshold, monitoring target 11 may not be assigned in this cycle.

[0143] In summary, based on the above-listed prerequisite steps, step one, step two and step three, we can finally obtain: the monitoring target set corresponding to shard 1 is {monitoring target 2, monitoring target 3, monitoring target 4, monitoring target 5}; the monitoring target set corresponding to shard 2 is {monitoring target 1, monitoring target 6, monitoring target 7, monitoring target 8}; the monitoring target set corresponding to shard 3 is {monitoring target 9, monitoring target 10, monitoring target 11}.

[0144] After that, you can execute Figure 6 In step 4 shown, the allocation result is sent to the Sidecar of each shard. Then the Sidecar can refer to the corresponding monitoring target combination to capture data, as shown in the above embodiment.

[0145] Based on the above sharding rules, monitoring targets with known data scale but not captured by any monitoring node can be further allocated. Dynamic adjustment is performed at runtime to ensure shard load balancing. In addition, combined with the above sharding rules, all monitoring targets with known data scale can be allocated to all Sidecars, achieving accurate and fast allocation of monitoring targets.

[0146] In addition, in the embodiment of the present application, the coordinator can automatically expand or shrink the capacity without manual intervention. For example, when the number of Sidecars is insufficient, the capacity is expanded.

[0147] Optionally, the coordinator of the embodiment of the present application can control the number of multiple monitoring nodes (the number of shards) according to the total data size of all acquired monitoring targets and the data capture capability of each monitoring node.

[0148] The total data size of all monitoring targets obtained by the coordinator generally refers to the total data size of the monitoring targets reported by the Sidecar. Figure 6 As shown in step 3, it is known that the data size of the monitoring target (such as monitoring target 11) detected by the coordinator itself may also be included. Figure 6 For example, in step 5, by calculating the sum of the data scales of all monitoring targets, the expected number of shards is set to: (sum of data scales / scale that can be collected by each shard) + 1. It should be noted that the scale in the embodiments of the present application refers to the data scale. Taking the case where the scale of data that can be collected by each shard listed in the embodiments of the present application is the same as an example, the scale that can be collected by each shard can be determined based on its corresponding data capture capability.

[0149] Based on the above formula, it can be calculated whether new shards are needed in the next cycle during automatic expansion and contraction, and generally no contraction is required. For example, the expected number of shards in the next cycle should be set to 4. In the embodiment of the present application, a parameter P for indicating the number of shards can be set. Specifically, when setting the expected number of shards, for example, the number of shard nodes plus 1, the parameter P can be directly adjusted to increase the number of shards.

[0150] In the above implementation, the expansion and contraction is automatically performed by the coordinator without manual intervention, which is more convenient and efficient than the method of manual expansion and contraction in the related art.

[0151] It should be noted that the above-listed method of calculating the expected number of shards is only an example, which can also be set to +2, +3, etc., without specific limitation. In addition, it should be noted that in the sharding algorithm of the coordinator listed in the above embodiment, specific supportable load information of the shards can also be added, instead of the load conditions of all shards being the same. This is more conducive to ensuring load balancing of the shards.

[0152] See also Figure 7 As shown, it is an implementation flow chart of a data monitoring control method provided in an embodiment of the present application, which is applied to a monitoring node. The specific implementation process of the method is as follows:

[0153] S71: Obtain monitoring targets through service discovery, and report each obtained monitoring target to the coordinator, so that the coordinator allocates a corresponding monitoring target set to each monitoring node according to the acquired data scale of each monitoring target and the pre-configured sharding rule;

[0154] S72: Receive the monitoring target set assigned by the coordinator, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database, so that the time series database summarizes the monitoring data captured by each monitoring node.

[0155] Optionally, the monitoring node includes a monitoring server and a container process server;

[0156] Capture data for each monitoring target in the monitoring target set, including:

[0157] The monitoring server initiates a data capture request for the monitoring target to be captured, and based on the pre-configured target configuration file, proxies the data capture request to the corresponding container process server;

[0158] After receiving the data capture request, the container process server captures data according to the monitoring target set, obtains the data capture result for the data capture request, and returns the data capture result to the monitoring server;

[0159] The monitoring server receives the data capture results returned by the container process server.

[0160] Optionally, data is captured according to the monitoring target set, a data capture result for the data capture request is obtained, and the data capture result is returned to the monitoring server, specifically including:

[0161] According to the data capture request of the monitoring server for the monitoring target to be captured, when the monitoring target to be captured is in the monitoring target set, sending a data capture request to the monitoring target to be captured; capturing monitoring data from the monitoring target to be captured, and returning the captured monitoring data to the monitoring server; or

[0162] When the monitoring target to be captured is not in the monitoring target set, a null value is returned to the monitoring server.

[0163] Optionally, the target configuration file includes proxy indication information and network address information of the container process server; based on the pre-configured target configuration file, the data capture request is proxied to the corresponding container process server, specifically including:

[0164] According to the proxy instruction information in the target configuration file and the network address information of the container process server, the data crawling request is proxied to the container process server via the http protocol, wherein the target configuration file is obtained and updated by the container process server.

[0165] In the above implementation, the Sidecar configuration injection does not change the original configuration file, but regenerates a configuration for Prometheus, which is non-intrusive to the original configuration file and does not cause the problem of complex configuration management.

[0166] See also Figure 8 As shown, it is an implementation flow chart of a data monitoring control method provided in an embodiment of the present application, which is applied to a coordinator. The specific implementation process of the method is as follows:

[0167] S81: Obtain the data scale of each monitoring target reported by each monitoring node;

[0168] S82: According to the data scale of each monitoring target and the pre-configured sharding rules, a corresponding monitoring target set is assigned to each monitoring node, and each monitoring node is notified so that after each monitoring node receives the monitoring target set assigned by the coordinator, it captures data for each monitoring target in the monitoring target set, and sends the captured monitoring data to the time series database, which summarizes the monitoring data captured by each monitoring node.

[0169] Optionally, obtain the data scale of each monitoring target reported by each monitoring node, including:

[0170] Receive each monitoring target reported by each monitoring node, detect the data scale of each monitoring target; after allocating a corresponding monitoring target set to each monitoring node, send the data scale of each monitoring target in the monitoring target set to the corresponding monitoring node; or

[0171] Receive the updated data scale of each monitoring target reported by each monitoring node.

[0172] Optionally, the method further includes:

[0173] The number of monitoring nodes is controlled based on the total data size of all monitoring targets and the data capture capability of each monitoring node.

[0174] Optionally, according to the data scale of each monitoring target and the pre-configured sharding rules, a corresponding monitoring target set is allocated to each monitoring node, specifically including:

[0175] Get the total data size of each monitoring target currently captured by each monitoring node;

[0176] For the first-class monitoring nodes whose total data scale exceeds the preset threshold, after selecting at least one candidate monitoring target from the monitoring targets currently captured by the first-class monitoring nodes, the set consisting of all the remaining monitoring targets is used as the monitoring target set of the first-class monitoring nodes, wherein the sharding rule includes that the total data scale of each monitoring target in the monitoring target set of the first-class monitoring nodes is not less than the preset threshold;

[0177] For the second-category monitoring nodes whose total data scale does not exceed the preset threshold, the set consisting of the various monitoring targets currently captured by the second-category monitoring nodes is used as the monitoring target set of the second-category monitoring nodes, wherein the sharding rule includes that the total data scale of each monitoring target in the monitoring target set of the second-category monitoring nodes does not exceed the preset threshold; or the set consisting of each monitoring target currently captured by the second-category monitoring node and at least one candidate monitoring target is used as the monitoring target set of the second-category monitoring nodes, wherein the sharding rule includes that the total data scale of each monitoring target and the candidate monitoring target in the monitoring target set of the second-category monitoring nodes does not exceed the preset threshold.

[0178] Optionally, according to the data scale of each monitoring target and the pre-configured sharding rules, a corresponding monitoring target set is allocated to each monitoring node, specifically including:

[0179] When assigning candidate monitoring targets to second-class monitoring nodes, if the sum of the data sizes currently corresponding to all second-class monitoring nodes exceeds the preset threshold after adding the candidate monitoring targets, the candidate monitoring targets are added to the monitoring target set corresponding to the original first-class monitoring nodes.

[0180] Optionally, the method further includes:

[0181] For the monitoring targets that are not captured by any monitoring nodes at present, check whether the data scale of the monitoring targets has been detected;

[0182] If yes, the monitoring target is assigned to the monitoring node with the smallest total data scale of each monitoring target in the current corresponding monitoring target set, wherein the monitoring target set corresponding to the monitoring node with the smallest total data scale does not exceed the preset threshold after the monitoring target is added;

[0183] Otherwise, no monitoring target is assigned in this cycle.

[0184] In the above implementation, the allocation of monitoring targets is based on the data size of each monitoring target, and will be dynamically adjusted during runtime, so there will be no problem of excessive load on a single shard. In addition, the expansion and contraction is automatically performed by the coordinator without manual intervention.

[0185] See also Fig. 9 As shown in FIG. 1 , it is an interactive timing diagram for realizing data monitoring control based on a data monitoring control system. The specific implementation process of the method is as follows:

[0186] Step S901: The Sidecar server calls the Prometheus interface, obtains each monitoring target through service discovery, and reports the basic information of the monitoring target to the coordinator;

[0187] Step S902: the coordinator obtains the data scale of the corresponding monitoring target according to the basic information of each monitoring target;

[0188] Step S903: The coordinator allocates a corresponding monitoring target set to the monitoring node where each Sidecar server is located according to the data scale of each monitoring target and the pre-configured sharding rule, and notifies each Sidecar server;

[0189] Step S904: The Prometheus server initiates a data capture request for the monitoring target to be captured, and proxies the data capture request to the corresponding Sidecar server based on the pre-configured target configuration file;

[0190] Step S905: After receiving the data capture request, the Sidecar server captures data according to the monitoring target set;

[0191] Step S906: The Sidecar server returns the data crawling result to the Prometheus server;

[0192] Step S907: The Prometheus server sends the captured monitoring data to the time series database;

[0193] Step S908: The time series database summarizes the monitoring data captured by each monitoring node;

[0194] Step S909: After capturing the data, the Sidecar server updates the data scale of the monitoring target in real time, and then reports the latest data scale to the coordinator, and then repeats steps S903 to S909.

[0195] It should be noted that Fig. 9 Only one Sidecar server and one Prometheus server are listed in the figure. In fact, there will be multiple Sidecar servers and multiple Prometheus servers in the cluster. The processing process of each Sidecar server and Prometheus server is basically similar. For details, please refer to the above embodiment, and no specific limitation is made here.

[0196] Based on the same inventive concept, the present application also provides a data monitoring control device. Fig.10 As shown, it is a structural schematic diagram of a data monitoring control device 1000 provided in an embodiment of the present application, which is applied to a monitoring node and may include:

[0197] The data reporting unit 1001 is used to obtain monitoring targets through service discovery, and report each obtained monitoring target to the coordinator, so that the coordinator can allocate a corresponding monitoring target set to each monitoring node according to the data scale of each monitoring target obtained and the pre-configured sharding rule;

[0198] The data capture unit 1002 is used to receive the monitoring target set assigned by the coordinator, and capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database so that the time series database summarizes the monitoring data captured by each monitoring node.

[0199] Optionally, the monitoring node includes a monitoring server and a container process server; the data capture unit 1002 is specifically used for:

[0200] Initiate a data capture request for the monitoring target to be captured based on the monitoring server, and proxy the data capture request to the corresponding container process server based on the pre-configured target configuration file;

[0201] After receiving the data capture request, the container process server captures data according to the monitoring target set, obtains the data capture result for the data capture request, and returns the data capture result to the monitoring server;

[0202] The monitoring server receives the data capture results returned by the container process server.

[0203] Optionally, the data capture unit 1002 is specifically used for:

[0204] According to the data capture request of the monitoring server for the monitoring target to be captured, when the monitoring target to be captured is in the monitoring target set, sending a data capture request to the monitoring target to be captured; capturing monitoring data from the monitoring target to be captured, and returning the captured monitoring data to the monitoring server; or

[0205] When the monitoring target to be captured is not in the monitoring target set, a null value is returned to the monitoring server.

[0206] Optionally, the target configuration file includes proxy instruction information and network address information of the container process server; the data capture unit 1002 is specifically used to:

[0207] According to the proxy instruction information in the target configuration file and the network address information of the container process server, the data crawling request is proxied to the container process server via the http protocol, wherein the target configuration file is obtained and updated by the container process server.

[0208] Based on the same inventive concept, the present application also provides a data monitoring control device. Fig.11 As shown, it is a structural diagram of a data monitoring control device 1100 provided in an embodiment of the present application, which is applied to a coordinator and may include:

[0209] The data acquisition unit 1101 is used to acquire the data scale of each monitoring target reported by each monitoring node;

[0210] The target allocation unit 1102 is used to allocate a corresponding monitoring target set to each monitoring node according to the data scale of each monitoring target and the pre-configured sharding rules, and notify each monitoring node so that after each monitoring node receives the monitoring target set allocated by the coordinator, it captures data for each monitoring target in the monitoring target set, and sends the captured monitoring data to the time series database, which summarizes the monitoring data captured by each monitoring node.

[0211] Optionally, the data acquisition unit 1101 is specifically used for:

[0212] Receive each monitoring target reported by each monitoring node, detect the data scale of each monitoring target; after allocating a corresponding monitoring target set to each monitoring node, send the data scale of each monitoring target in the monitoring target set to the corresponding monitoring node; or

[0213] Receive the updated data scale of each monitoring target reported by each monitoring node.

[0214] Optionally, the device further comprises:

[0215] The node control unit 1103 is used to control the number of monitoring nodes according to the total data size of all acquired monitoring targets and the data capture capability of each monitoring node.

[0216] Optionally, the target allocation unit 1102 is specifically configured to:

[0217] Get the total data size of each monitoring target currently captured by each monitoring node;

[0218] For the first-class monitoring nodes whose total data scale exceeds the preset threshold, after selecting at least one candidate monitoring target from the monitoring targets currently captured by the first-class monitoring nodes, the set consisting of all the remaining monitoring targets is used as the monitoring target set of the first-class monitoring nodes, wherein the sharding rule includes that the total data scale of each monitoring target in the monitoring target set of the first-class monitoring nodes is not less than the preset threshold;

[0219] For the second-category monitoring nodes whose total data scale does not exceed the preset threshold, the set consisting of the various monitoring targets currently captured by the second-category monitoring nodes is used as the monitoring target set of the second-category monitoring nodes, wherein the sharding rule includes that the total data scale of each monitoring target in the monitoring target set of the second-category monitoring nodes does not exceed the preset threshold; or the set consisting of each monitoring target currently captured by the second-category monitoring node and at least one candidate monitoring target is used as the monitoring target set of the second-category monitoring nodes, wherein the sharding rule includes that the total data scale of each monitoring target and the candidate monitoring target in the monitoring target set of the second-category monitoring nodes does not exceed the preset threshold.

[0220] Optionally, the target allocation unit 1102 is used to:

[0221] When assigning candidate monitoring targets to second-class monitoring nodes, if the sum of the data sizes currently corresponding to all second-class monitoring nodes exceeds the preset threshold after adding the candidate monitoring targets, the candidate monitoring targets are added to the monitoring target set corresponding to the original first-class monitoring nodes.

[0222] Optionally, the target allocation unit 1102 is used to:

[0223] For the monitoring targets that are not captured by any monitoring nodes at present, check whether the data scale of the monitoring targets has been detected;

[0224] If yes, the monitoring target is assigned to the monitoring node with the smallest total data scale of each monitoring target in the current corresponding monitoring target set, wherein the monitoring target set corresponding to the monitoring node with the smallest total data scale does not exceed the preset threshold after the monitoring target is added;

[0225] Otherwise, no monitoring target is assigned in this cycle.

[0226] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0227] After introducing the data monitoring control method and device according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0228] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0229] See also Fig.12 As shown, it is a schematic diagram of the composition structure of an electronic device in an embodiment of the present application. The electronic device can be used for data monitoring and control. In one embodiment, the electronic device can be a server, such as Figure 1 The server 120 shown may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.12 As shown, it includes a memory 1201 , a communication module 1203 and one or more processors 1202 .

[0230] The memory 1201 is used to store computer programs executed by the processor 1202. The memory 1201 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0231] The memory 1201 may be a volatile memory, such as a random-access memory (RAM); the memory 1201 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1201 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1201 may be a combination of the above memories.

[0232] The processor 1202 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1202 is used to implement the above data monitoring control method when calling the computer program stored in the memory 1201 .

[0233] The communication module 1203 is used to communicate with terminal devices and other servers.

[0234] The specific connection medium between the memory 1201, the communication module 1203 and the processor 1202 is not limited in the embodiment of the present application. Fig.12 In the embodiment, the memory 1201 and the processor 1202 are connected via a bus 1204. The bus 1204 is connected to the processor 1202 via a bus 1204. Fig.12 The connections between other components are shown in bold lines, and are not intended to be limiting. Bus 1204 can be divided into address bus, data bus, control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0235] The memory 1201 stores a computer storage medium, which stores computer executable instructions, and the computer executable instructions are used to implement the data monitoring control method of the embodiment of the present application. The processor 1202 is used to execute the above data monitoring control method.

[0236] An embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. The computer executable instructions are used to implement the data monitoring and control method recorded in any embodiment of the present application.

[0237] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0238] Alternatively, if the above-mentioned integrated unit of the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks, or optical disks.

[0239] In some optional implementations, the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the steps of the above-mentioned data monitoring control method. For example, the computer device may execute the following steps: Figure 7 or Figure 8 Follow the steps shown in .

[0240] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0241] The program product of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, device or device.

[0242] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0243] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0244] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user equipment, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0245] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0246] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data monitoring and control system, characterized in that: include: Multiple monitoring nodes, time series databases, and coordinators; Each of the multiple monitoring nodes is used to obtain a monitoring target through service discovery, and report each obtained monitoring target to the coordinator; receive a monitoring target set assigned by the coordinator, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database; The time series database is used to aggregate the monitoring data captured by each monitoring node; The coordinator is used to obtain the data scale of each monitoring target reported by each monitoring node, allocate a corresponding monitoring target set to each monitoring node according to the data scale of each monitoring target and a pre-configured sharding rule, and notify each monitoring node; wherein the monitoring target set allocated by the coordinator to each monitoring node will be dynamically adjusted during operation, and the coordinator is specifically used to divide each monitoring node into a first category of monitoring nodes whose total data scale exceeds the preset threshold and a second category of monitoring nodes whose total data scale does not exceed the preset threshold based on the size relationship between the preset threshold and the total data scale of each monitoring node, and adopt different sharding rules to allocate the first category of monitoring nodes and the second category of monitoring nodes respectively.

2. The system according to claim 1, characterized in that The coordinator is specifically used for: Receive each monitoring target reported by each monitoring node, detect the data scale of each monitoring target; after allocating a corresponding monitoring target set to each monitoring node, send the data scale of each monitoring target in the monitoring target set to the corresponding monitoring node; or Receive the updated data scale of each monitoring target reported by each monitoring node.

3. The system according to claim 1, characterized in that The coordinator is also used to: The number of the plurality of monitoring nodes is controlled according to the total data size of all acquired monitoring targets and the data capture capability of each monitoring node.

4. The system according to claim 1, characterized in that Each monitoring node includes a monitoring server and a container process server; The monitoring server is used to initiate a data capture request for a monitoring target to be captured, and proxy the data capture request to a corresponding container process server based on a pre-configured target configuration file; and receive a data capture result for the data capture request returned by the container process server; The container process server is used to, after receiving the data capture request, capture data according to the monitoring target set, obtain a data capture result for the data capture request, and return the data capture result to the monitoring server.

5. The system according to claim 4, characterized in that The container process server is specifically used for: According to the data capture request of the monitoring server for the monitoring target to be captured, when the monitoring target to be captured is in the monitoring target set, sending a data capture request to the monitoring target to be captured; capturing monitoring data from the monitoring target to be captured, and returning the captured monitoring data to the monitoring server; or When the monitoring target to be captured is not in the monitoring target set, a null value is returned to the monitoring server.

6. The system according to claim 4, characterized in that The target configuration file includes proxy instruction information and network address information of the container process server; the monitoring server is specifically used to: According to the proxy indication information in the target configuration file and the network address information of the container process server, the data crawling request is proxied to the container process server via the hypertext transfer protocol, wherein the target configuration file is obtained and updated by the container process server.

7. The system according to any one of claims 1 to 6, characterized in that: The coordinator is specifically used for: Get the total data size of each monitoring target currently captured by each monitoring node; For the first-type monitoring node whose total data scale exceeds the preset threshold, after selecting at least one candidate monitoring target from each monitoring target currently captured by the first-type monitoring node, a set consisting of all remaining monitoring targets is used as the monitoring target set of the first-type monitoring node, wherein the sharding rule includes that the total data scale of each monitoring target in the monitoring target set of the first-type monitoring node is not less than the preset threshold; For the second-category monitoring node whose total data scale does not exceed the preset threshold, the set consisting of the various monitoring targets currently captured by the second-category monitoring node is used as the monitoring target set of the second-category monitoring node, wherein the sharding rule includes that the total data scale of the various monitoring targets in the monitoring target set of the second-category monitoring node does not exceed the preset threshold; or the set consisting of the various monitoring targets currently captured by the second-category monitoring node and at least one candidate monitoring target is used as the monitoring target set of the second-category monitoring node, wherein the sharding rule includes that the total data scale of the various monitoring targets and the candidate monitoring targets in the monitoring target set of the second-category monitoring node does not exceed the preset threshold.

8. The system according to claim 7, characterized in that The coordinator is specifically used for: When the candidate monitoring target is assigned to the second-category monitoring node, if the total data size currently corresponding to all second-category monitoring nodes exceeds the preset threshold after adding the candidate monitoring target, the candidate monitoring target is added to the monitoring target set corresponding to the original first-category monitoring node.

9. The system according to claim 8, characterized in that The coordinator is also used to: For a monitoring target that is not currently captured by any monitoring node, detecting whether the data scale of the monitoring target has been detected; If yes, the monitoring target is assigned to the monitoring node with the smallest total data scale of each monitoring target in the currently corresponding monitoring target set, wherein after the monitoring target set corresponding to the monitoring node with the smallest total data scale is added, the total data scale of each monitoring target does not exceed the preset threshold; Otherwise, the monitoring target is not allocated in this cycle.

10. A data monitoring control method, characterized in that: Applied to the monitoring node according to any one of claims 1 to 9, the method comprises: The monitoring target is obtained through service discovery, and each obtained monitoring target is reported to the coordinator, so that the coordinator allocates a corresponding monitoring target set to each monitoring node according to the acquired data scale of each monitoring target and the pre-configured sharding rules; wherein the monitoring target set allocated by the coordinator to each monitoring node will be dynamically adjusted during operation, and the monitoring target set allocated by the coordinator to each monitoring node will be dynamically adjusted during operation, and the coordinator is specifically used to divide each monitoring node into a first category of monitoring nodes whose total data scale exceeds the preset threshold and a second category of monitoring nodes whose total data scale does not exceed the preset threshold based on the relationship between the preset threshold and the total data scale of each monitoring node, and adopt different sharding rules to allocate the first category of monitoring nodes and the second category of monitoring nodes respectively; Receive the monitoring target set assigned by the coordinator, capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database, so that the time series database summarizes the monitoring data captured by each monitoring node.

11. A data monitoring control method, characterized in that: Applied to the coordinator of any one of claims 1 to 9, the method comprising: Obtain the data scale of each monitoring target reported by each monitoring node; According to the data scale of each monitoring target and the pre-configured sharding rules, a corresponding monitoring target set is assigned to each monitoring node, and each monitoring node is notified so that after each monitoring node receives the monitoring target set assigned by the coordinator, it captures data for each monitoring target in the monitoring target set, and sends the captured monitoring data to the time series database, which summarizes the monitoring data captured by each monitoring node; wherein the monitoring target set assigned by the coordinator to each monitoring node will be dynamically adjusted during operation, and the coordinator is specifically used to divide the various monitoring nodes into a first category of monitoring nodes whose total data scale exceeds the preset threshold and a second category of monitoring nodes whose total data scale does not exceed the preset threshold based on the relationship between the preset threshold and the total data scale of each monitoring node, and adopts different sharding rules to allocate the first category of monitoring nodes and the second category of monitoring nodes respectively.

12. A data monitoring control device, characterized in that: The monitoring node applied to any one of claims 1 to 9 comprises: A data reporting unit, used to obtain monitoring targets through service discovery, and report each obtained monitoring target to the coordinator, so that the coordinator can allocate a corresponding monitoring target set to each monitoring node according to the data scale of each acquired monitoring target and the pre-configured sharding rules; wherein the monitoring target set allocated by the coordinator to each monitoring node will be dynamically adjusted during operation, and the coordinator is specifically used to divide each monitoring node into a first category of monitoring nodes whose total data scale exceeds the preset threshold and a second category of monitoring nodes whose total data scale does not exceed the preset threshold based on the size relationship between the preset threshold and the total data scale of each monitoring node, and adopt different sharding rules to allocate the first category of monitoring nodes and the second category of monitoring nodes respectively; The data capture unit is used to receive the monitoring target set assigned by the coordinator, and capture data for each monitoring target in the monitoring target set, and send the captured monitoring data to the time series database so that the time series database can summarize the monitoring data captured by each monitoring node.

13. A data monitoring control device, characterized in that: The coordinator applied to any one of claims 1 to 9 comprises: A data acquisition unit, used to acquire the data scale of each monitoring target reported by each monitoring node; A target allocation unit is used to allocate a corresponding monitoring target set to each monitoring node according to the data scale of each monitoring target and a pre-configured sharding rule, and to notify each monitoring node so that after each monitoring node receives the monitoring target set allocated by the coordinator, it captures data for each monitoring target in the monitoring target set, and sends the captured monitoring data to a time series database, which summarizes the monitoring data captured by each monitoring node; wherein the monitoring target set allocated by the coordinator to each monitoring node will be dynamically adjusted during operation, and the coordinator is specifically used to divide each monitoring node into a first type of monitoring node whose total data scale exceeds the preset threshold and a second type of monitoring node whose total data scale does not exceed the preset threshold based on the relationship between the preset threshold and the total data scale of each monitoring node, and adopts different sharding rules to allocate the first type of monitoring nodes and the second type of monitoring nodes respectively.

14. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to claim 10 or claim 11.

15. A computer-readable storage medium, characterized in that: It includes program code, and when the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to claim 10 or claim 11.

Images