Health examination system and method

Abstract

The embodiment of the invention provides a health examination system and method, relates to the technical field of computing networks, and aims to provide the expansion capability of the health examination system so as to support larger-scale service support on the basis of reducing the maintenance cost. The system at least comprises a console, a server and an agent, wherein the console is used for sending a creation request to the server; the server side is used for determining a target detector deployed in the agent based on the creation request; the agent is used for sending a pulling request to the server side and obtaining detection configuration corresponding to a health examination task in the server side; the agent is used for detecting the detection object through the target detector according to the detection configuration to obtain a detection result, and reporting the detection result to the server; and the server is used for querying the original detection result of the detection object, comparing the original detection result with the detection result, and determining that the state of the detection object changes under the condition that the original detection result is different from the detection result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computing network technology, and in particular to a health check system and method. Background Art

[0002] In a computing network environment, it is often necessary to implement load balancing, high availability, and QoS to ensure concurrent and efficient access by clients. These functions require health check or detection functions as a basic support.

[0003] However, when performing health checks, related technologies mostly process detection within a single cluster or a single scenario, which cannot support larger-scale business support. In addition, the detection methods are scattered across various systems, and the diagnosis, troubleshooting and maintenance methods are diverse, requiring higher maintenance costs. Summary of the invention

[0004] The embodiments of the present invention provide a health check system and method to provide the expansion capability of the health check system, thereby supporting larger-scale business support on the basis of reducing maintenance costs.

[0005] A first aspect of an embodiment of the present invention provides a health check system, the system comprising at least: a console, a server and an agent;

[0006] The console is used to send a creation request to the server, wherein the creation request is used to create a health check task;

[0007] The server is used to determine the target detector deployed in the agent based on the creation request;

[0008] The agent is used to send a pull request to the server to obtain the detection configuration corresponding to the health check task in the server;

[0009] The agent is used to detect the detection object through the target detection machine according to the detection configuration, obtain the detection result, and report the detection result to the server;

[0010] The server is used to query the original detection result of the detection object, compare the original detection result with the detection result, and determine that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

[0011] A second aspect of an embodiment of the present invention provides a health check method, which is applied to a server in the health check system of the first aspect of an embodiment of the present invention, and the method includes:

[0012] Receive a creation request sent by the console, where the creation request is used to create a health check task;

[0013] Determine the target detector deployed in the agent based on the creation request, and send detector confirmation information to the agent corresponding to the target detector;

[0014] Receive and, based on the pull request sent by the agent corresponding to the target detector, obtain the detection configuration corresponding to the health check task and return it to the agent, so that the agent detects the detection object through the target detector according to the detection configuration, obtains the detection result, and reports the detection result to the server;

[0015] An original detection result of the detection object is queried, and the original detection result is compared with the detection result. If there is a difference between the original detection result and the detection result, it is determined that the state of the detection object has changed.

[0016] A third aspect of an embodiment of the present invention provides a health check method, which is applied to an agent in the health check system described in the first aspect of an embodiment of the present invention. The method includes:

[0017] Receive the detector confirmation information sent by the server; the detector confirmation message is generated by the server after receiving the creation request sent by the console and determining the target detector deployed in the agent based on the creation request, and the creation request is used to create a health check task;

[0018] Send a pull request to the server to obtain the detection configuration corresponding to the health check task returned by the server;

[0019] According to the detection configuration, the detection object is detected by the target detection machine to obtain a detection result, and the detection result is reported to the server, so that the server queries the original detection result of the detection object, compares the original detection result with the detection result, and determines that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

[0020] The health check system provided in the embodiment of the present invention at least includes: a console, a server and an agent; wherein the control sends a creation request for creating a health check task to the server, and the server determines the target detector deployed in the agent based on the creation request; the agent sends a pull request to the server, obtains the detection configuration corresponding to the health check task in the server, and detects the detection object through the target detector according to the detection configuration to obtain the detection result, and reports the detection result to the server; the server queries the original detection result of the detection object, compares the original detection result with the detection result, and determines that the state of the detection object has changed when there is a difference between the original detection result and the detection result, so as to realize the health check of the detection object. In this way, the present embodiment realizes a distributed health check system through the agent-server architecture, provides the expansion capability of the health check system, and for new services and scenarios, there is no need to modify the health check system or only expand a small number of modules, so as to realize the support of larger-scale business support on the basis of reducing maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 is a structural block diagram of a health check system shown in one embodiment of the present invention;

[0023] Figure 2 It is a framework diagram of a distributed health check system proposed in one embodiment of the present invention;

[0024] Figure 3 is an agent deployment architecture diagram shown in one embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of a mapping relationship shown in an embodiment of the present invention;

[0026] Figure 5 is a module division diagram of an agent and a server shown in an embodiment of the present invention;

[0027] Figure 6 is a flow chart of a health examination method provided by an embodiment of the present invention;

[0028] Figure 7 is a flow chart of a health examination method provided by an embodiment of the present invention;

[0029] Figure 8 This is an overall flow chart of a health examination method proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In related technologies, common health check functions include:

[0032] 1. Detection scheduling: The number of detectors and target nodes increases with the increase of business. If all the detectors and target nodes are detected, a large number of detection requests will be generated and the performance of the target node will be affected. The load balancing of detection can be achieved through scheduling. The scheduling method can use a graph algorithm to select the detector.

[0033] 2. Detection method: Common detection methods include UDP, TCP, ICMP and HTTP, as well as simulation combined with business requests.

[0034] 3. Detection scenario support: result replacement in DDNS, service switching in service discovery, fault removal and recovery in ECMP (Equal-cost multi-path routing) and load balancing.

[0035] 4. Detection performance: From the data plane perspective, use partial link detection, and then use shared memory to synchronize the results to other business devices.

[0036] 5. Detection cluster management: You can use business-related devices, such as load balancing and ECMP, to group; you can also use independent detection devices to group.

[0037] 6. Monitoring of the detection system: In order to ensure the quality of detection, the detection system itself needs to be monitored in different dimensions to ensure the stability and reliability of the detection results; the methods adopted include component monitoring, detection machine operating system monitoring (such as CPU, memory, IO, etc.), and log monitoring.

[0038] 7. Detection result data analysis: The detection results include packet loss, latency, and inspection time, combined with other system indicators (such as CPU, memory, IO, etc.) to perform weighted analysis quality.

[0039] However, most of the related technologies process detection within a single cluster or a single scenario and cannot support larger-scale business support. In addition, the detection methods are scattered across various systems, and the diagnosis, troubleshooting and maintenance methods are diverse, requiring higher maintenance costs.

[0040] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, an embodiment of the present invention proposes a health check system, which mainly provides the expansion capability of the health check system, including horizontal and business expansion. Among them, horizontal expansion mainly includes stateless design of components, consistent hashing and balancing requests, etc. Business expansion mainly means that the system can adapt to business systems that comply with the IP protocol and provide the ability to switch callback states. The embodiment of the present invention implements a distributed health check system through an agent-server architecture. For new businesses and scenarios, there is no need to modify the health check system or expand only a small number of modules, thereby achieving support for larger-scale business support while reducing maintenance costs.

[0041] refer to Figure 1 , Figure 1 FIG. 1 is a structural block diagram of a health check system according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, since the health check data model is small, the health check configuration does not change much, and the main processing flow of the detection is actually similar, the model can be abstracted and processed uniformly. Therefore, the health check system is designed as an agent-server architecture, that is, a distributed system architecture. Among them, the health check system of this embodiment includes at least: a console, a server, and an agent.

[0042] The console is used to send a creation request to the server, where the creation request is used to create a health check task.

[0043] In this embodiment, the console is used to operate resources. For health checks, users can initiate a creation request through the console, and the console can send the creation request to the server through HTTP. The creation request is used to create a health check task, and the health check task is used to perform a health check on the business service.

[0044] The server is used to determine the target detector deployed in the agent based on the creation request.

[0045] In this embodiment, after receiving the creation request, the server can select a target detector from the detectors, and the target detector is the detector that performs the health check task. The detector in this embodiment is deployed in the agent.

[0046] In a preferred embodiment, the detector is deployed in the agent in the form of a plug-in. After the model abstraction is unified and the detection method in the agent is plug-in, this embodiment can be applied to most scenarios, such as load balancing, ECMP, website survival status, etc., and can support centralized management.

[0047] In an optional embodiment, the detector deployed in the agent can be added through the console or by starting the automatic registration of the agent. When adding, the region information (region) and label information (label) of the detector are set according to the actual situation to facilitate the scheduling of the server. The load information (load) of the created node (i.e., the created detector) defaults to 0. In this embodiment, the steps of adding the detector are repeated according to business needs until there are enough detectors.

[0048] In one embodiment, after determining the target detector deployed in the agent, the server can generate detector confirmation information based on the target detector. The server can send the detector confirmation information to the agent corresponding to the target detector through GRPC, or the agent can periodically pull the corresponding detector confirmation information from the server through GRPC to inform the agent of business changes. The business changes may include: changes in the detection configuration, including at least any of the following: addition or deletion of detection targets, increase or decrease in detection frequency.

[0049] The agent is used to send a pull request to the server to obtain the detection configuration corresponding to the health check task in the server.

[0050] In this embodiment, the agent corresponding to the target detector can send a pull request to the server, and the server obtains the detection configuration corresponding to the detection object in the health check task based on the pull request, and returns the detection configuration to the agent, so that the agent can obtain the detection configuration corresponding to the health check task in the server. The server can save the detection configuration for multiple detection objects.

[0051] The agent is used to detect the detection object through the target detection machine according to the detection configuration, obtain the detection result, and report the detection result to the server.

[0052] In this embodiment, the agent can detect the detection object through the deployed target detection machine based on the acquired detection configuration, obtain the detection result, and report the obtained detection result to the server. In this embodiment, the detection object can refer to the business service to be health checked in the health check task. The detection result includes at least: detection status.

[0053] In an optional embodiment, after the agent obtains the detection configuration, it can detect the detection object using the coroutine or process of the current process, or use an additional program to detect the detection object according to the detection configuration, and then collect the detection results.

[0054] The server is used to query the original detection result of the detection object, compare the original detection result with the detection result, and determine that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

[0055] In this embodiment, after the server receives the detection result for the detection object reported by the agent, it can query and obtain the original detection result of the detection object, which can refer to the last detection result for the detection object. Then the original detection result is compared with the detection result to determine whether there is a difference between the original detection result and the detection result. When the server determines that there is a difference between the original detection result and the detection result, it can determine that the state of the detection object has changed and obtain the health check result of the detection object. For example, if the crude oil detection state in the original detection result is active, and the detection state in the current detection result is abnormal (such as packet loss or no keyword found), it can be determined that there is a difference between the original detection result and the detection result, and it is determined that the state of the detection object has changed, and the health check result of the detection object is obtained.

[0056] In this embodiment, a distributed health check system is implemented through the agent-server architecture. The health check of the detection object can be achieved through the mutual interaction of the console, the server and the agent. For new businesses and scenarios, there is no need to modify the health check system or only to expand a small number of modules. This achieves support for larger-scale business support while reducing maintenance costs.

[0057] In addition, in combination with the above embodiments, in one implementation, there are multiple target detectors, and the above “detecting the detection object by the target detector to obtain the detection result” may specifically include:

[0058] Determine the detection result obtained by each target detection machine when detecting the detection object;

[0059] Determine the number of target detectors with normal detection results / the total number of valid detectors, and determine the number of target detectors with abnormal detection results / the total number of valid detectors;

[0060] When the detection result shows that the number of target detectors / the total number of valid detectors is greater than the first threshold, the agent can determine that the detection state corresponding to the detection object is normal and obtain the detection result.

[0061] When the number of target detectors with abnormal detection results / the total number of valid detectors is greater than the second threshold, the agent can determine that the detection state corresponding to the detection object is abnormal and obtain the detection result.

[0062] In combination with the above embodiments, in one implementation, an embodiment of the present invention further provides a health check system. Specifically, in this embodiment, the server is further used to perform a callback according to the callback processing method in the detection configuration when the state of the detection object changes, and notify the corresponding business system.

[0063] In this embodiment, the server can also perform a callback according to the callback processing method in the detection configuration when determining that the state of the detection object has changed, and notify the corresponding business system of the health check result, for example, by email alert, traffic switching, etc. This embodiment supports custom callbacks, and can be connected to various business or notification systems. If the health check system does not change, there will be no stability problems. That is, this embodiment can achieve state switching by adding a new business system through custom callbacks without changing the health system.

[0064] In combination with the above embodiments, in one implementation, an embodiment of the present invention further provides a health check system. Specifically, in this embodiment, the above “sending a pull request to the server to obtain the detection configuration corresponding to the health check task in the server” may include the following steps:

[0065] Step S11: Perform consistent hashing on the hostname to determine the first server among multiple servers.

[0066] In this embodiment, there are multiple servers, that is, the health check system agent includes multiple servers, and the agent can perform consistent hashing calculation on the hostname to determine the first server from the multiple servers. The number of servers can be obtained through additional service discovery or DNS system. Among them, the first server is a server that provides detection configuration for the health check task.

[0067] Step S12: Send the pull request to the first server based on the address of the first server to obtain the detection configuration in the first server.

[0068] In this embodiment, after determining the first server, the agent may send a pull request to the first server based on the address of the first server to obtain the detection configuration in the first server. Specifically, a pull request may be sent to the first server, and the first server obtains the detection configuration corresponding to the health check task based on the pull request and returns it to the agent.

[0069] In this embodiment, the hostname is consistently hashed to obtain the address of the specific first server and then request a detection configuration, which can ensure that the requests are evenly distributed, achieve load balancing, and reduce the amount of data migration after a server exception.

[0070] In combination with the above embodiments, in another implementation, the embodiment of the present invention further provides a health check system. Specifically, in this embodiment, the above “reporting the detection result to the server” may specifically include the following steps:

[0071] Step S21: Perform consistent hashing on the health check ID of the health check task to determine a second server among multiple servers.

[0072] In this embodiment, there are multiple servers, that is, the health check system agent includes multiple servers. The agent can detect the detection object based on the detection frequency (such as once every 1-60 minutes) to obtain multiple detection results, thereby determining the final detection result from the multiple detection results. In addition, the agent can use the health check ID of the health check task to perform a consistent hash calculation to determine the second server from the multiple servers, and the second server is the server that reports the detection results for the health check task. It should be noted that the first server and the second server of the present invention can be the same server or different servers, and there is no limitation on this.

[0073] Step S22: reporting the detection result to the second server.

[0074] In this embodiment, after the second server is determined, the final detection result may be reported to the second server.

[0075] In a specific example, the agent can obtain a server list, that is, determine multiple servers of the health check system. For example, the server list can be obtained through an additional service discovery system or DNS system. For example, connect to the DNS system and obtain the server list from the DNS system. The health check ID can be used for consistent hashing, and after calculating the mapping relationship hc_id->server_ip, after aggregating server_ip->[]{hc_id}, the second server can be selected from the server category, and the detection results can be reported to the second server in batches. Among them, hc_id is the health check ID, and server_ip is the address of the server.

[0076] The analysis of detection results on the server side often consumes more CPU, and horizontal expansion needs to be considered. Therefore, this embodiment performs consistent hashing on the health check IDs and reports them in batches to the second server, thereby achieving horizontal expansion and supporting larger-scale business support.

[0077] In combination with the above embodiment, in another implementation, the embodiment of the present invention further provides a health check system. In this embodiment, the server is also used to manage the information of the detector, and the information of the detector includes at least: load information (load), label information (label) and region information (region).

[0078] In this embodiment, the above-mentioned "determining the target detector deployed in the agent" may specifically include: selecting the target detector for executing the health check task according to the actual packet sending amount and CPU consumption based on the minimum load information, label information and area information.

[0079] In this embodiment, the server can select the target detector that performs the health check task from multiple detectors based on the information of the detector, the minimum load information, the label information and the area information, according to the actual packet sending volume and the CPU consumption, and write the determined target detector into the persistent database. In this embodiment, the load value calculation of the detector load changes, compared with the scheduling based on the detection object, can make the traffic and load more balanced.

[0080] For example, the scheduling strategy for screening the detectors may include: based on the regional information, disperse between regions as much as possible, and select one detector for each region. Filter according to the business type and tenant information of the health check task, and try to ensure that there are more than two target detectors. In addition, the scheduling strategy of this embodiment can be adjusted and configured as needed.

[0081] The server is further configured to determine the agent on which the target detector is deployed as the agent for obtaining the detection configuration, and send detector confirmation information to the agent corresponding to the target detector.

[0082] In this embodiment, after determining the target detector, the server may generate detector confirmation information, determine the agent deployed with the target detector as the agent for obtaining detection configuration, and send the detector confirmation information to the agent corresponding to the target detector.

[0083] The agent is specifically used to send a pull request to the server based on the detection machine confirmation information.

[0084] In this embodiment, after receiving the confirmation information from the detector, the agent can send a pull request to the server based on the confirmation information from the detector.

[0085] In combination with the above embodiments, in one implementation, the embodiment of the present invention further provides a health check system. Specifically, in this embodiment, the server is also used to:

[0086] Connect to multiple caches or time series databases, and store the detection results and / or the detection configuration fragments in a distributed cache.

[0087] In this embodiment, the server is connected to a variety of caches or time series databases, and the server can store the detection results or / detection configuration shards in the distributed cache. In this way, the server stores the detection results or / detection configuration shards in the distributed cache, which can ensure that the components are stateless and horizontal expansion can be achieved. That is, this embodiment can achieve horizontal expansion through data sharding, and can then support larger-scale business support.

[0088] like Figure 2 As shown, Figure 2 This is a framework diagram of a distributed health check system proposed in one embodiment of the present invention. Figure 2 In the embodiment, the distributed health check system at least includes: multiple agents (such as agent1 and agent2), multiple servers (such as server 1 (server1), server 2 (server2) and server 3 (server3)) and a database. The database includes MariaDB and a distributed cache cluster time series database (distributed cache cluterTime series database). For example, the server can store the detection results or / detection configuration fragments in the database to store them in the distributed cache. And in this embodiment, in order to improve the communication performance between the agent and the server, the agent and the server can communicate through grpc to send the health check ID (i.e., hc id) to the server, where hc is health check.

[0089] In combination with the above embodiments, in one implementation, an embodiment of the present invention further provides a health check system. Specifically, in this embodiment, there are multiple agents, there are multiple servers, the multiple servers are deployed in different areas, and the multiple agents in each area are deployed in physical machines, virtual machines and / or containers in the area.

[0090] Among them, the health check system of this embodiment also includes: an external server, which at least includes: an intelligent operation and maintenance system, which is used to receive the mirrored detection results sent by the server, and based on the mirrored detection results, perform performance and quality analysis on the detection object to obtain analysis results.

[0091] In this embodiment, after receiving the detection results, the server can mirror the detection results and send them to the intelligent operation and maintenance system. After receiving the mirrored detection results sent by the server, the intelligent operation and maintenance system can analyze the performance and quality of the detection object based on the mirrored detection results to obtain the analysis results.

[0092] like Figure 3 As shown, Figure 3 FIG. 1 is a diagram of an agent deployment architecture shown in an embodiment of the present invention. Figure 3 In Region A, there is a server deployed in Region A, and there are three agents deployed in Region A. The three agents are deployed in host A1, virtual machine A1, and container A1 in Region A. There is a server deployed in Region B, and there are three agents deployed in host B1, virtual machine B1, and container B1 in Region B. And, Figure 3 The top external server (server) is the external server of the health check system, such as the intelligent operation and maintenance system.

[0093] In combination with the above embodiments, in one implementation, the embodiment of the present invention further provides a health check system. Specifically, in this embodiment, the server is also used to:

[0094] Receive the pull request, and determine the agentID according to the hostname in the pull request;

[0095] Based on the agentID and the health check association table, a health check ID is obtained, wherein the health check association table includes a mapping relationship between the agentID and the health check ID;

[0096] Based on the health check ID and the associated health check table, query the detection configuration corresponding to the health check ID, and return the detection configuration to the agent; the health check table includes a mapping relationship between the health check ID and the detection configuration;

[0097] The detection configuration includes at least any one of the following: detection frequency, health threshold, abnormal threshold and detection type.

[0098] In this embodiment, the server can determine the hostname in the pull request based on the received pull request and query its corresponding detection configuration. Specifically, the agent ID (node) is determined based on the hostname, and then the health check association table is associated with the agent ID to obtain the health check ID, and then the health check table is associated with the health check ID to query and obtain the detection configuration corresponding to the health check ID, and the detection configuration is returned to the agent.

[0099] In this embodiment, the server side saves the health check association table (TableNodeHcAssociation) and the health check table (Table healthcheck) in advance. The health check association table includes the mapping relationship between the agentID and the health check ID, and the health check table includes the mapping relationship between the health check ID and the detection configuration. Figure 4 As shown, Figure 4 is a schematic diagram of a mapping relationship shown in an embodiment of the present invention. Figure 4 The node table (Table Node) includes the node id (ie, agentID), and then based on the node id and the health check association table (TableNodeHcAssociation), the hc id (ie, health check ID) is determined. Finally, based on the hc id and the health check table (Tablehealthcheck), the corresponding detection configuration is determined.

[0100] In this embodiment, the detection configuration includes at least one of the following for all possible health check scenarios in the same abstraction: detection frequency, health threshold, abnormal threshold and detection type. The detection frequency is interval, which indicates how many seconds the detection is performed; the health threshold is healthy threshold, which indicates how many consecutive normal checks are considered normal; the abnormal threshold is unhealthy threshold, which indicates how many consecutive normal checks are considered abnormal; the detection type is type, which includes UDP, TCP, ICMP and HTTP detection methods, etc.

[0101] In addition, in another embodiment, the detection configuration may also include at least one of the following: source IP (src ip), destination ip (dest ip), source port (src port), destination port (dest port), custom detection parameters (options, such as http specified uri detection and custom return status code), callback processing method (callback, can support http, kafka, etc.), encapsulation method (encapsulate type, such as flat no encapsulation, vxlan encapsulation, geneve encapsulation, etc.), encapsulation id (encapsulate id, such as vni, etc.) and encapsulation options (encapsulate options).

[0102] like Figure 5 As shown, Figure 5 FIG. 1 is a module division diagram of an agent and a server shown in an embodiment of the present invention. Figure 5 In the server, the server includes the following five modules:

[0103] 1. Cluster manager: used to manage the information and scheduling of the detector. Node.load represents the load of the node. The value Σ(m*n) is calculated by the detection method m (icmp:1, udp:1, tcp:2, http:3, ssl:5, http / 2:6, the specific value can be estimated based on the CPU consumption and the actual protocol packet volume) and the detection frequency n (qps, request per second).

[0104] 2. Health check configuration management (health check manager): used to manage health check configuration. From the perspective of unified abstraction of all possible health check scenarios, the common fields include source IP (src ip), destination IP (dest ip), source port (src port), destination port (dest port), detection frequency (interval, detection every few seconds), healthy threshold (healthy threshold, several consecutive checks are considered normal), abnormal threshold (unhealthy threshold, several consecutive checks are considered abnormal), type (detection type, icmp, udp, tcp, http, etc.), options (custom detection parameters, such as http specified uri detection and custom return status code), callback (callback processing method, can support http, kafka, etc.), encapsulation method (encapsulate type, flat no encapsulation, vxlan encapsulation, geneve encapsulation, etc.), encapsulation id (encapsulate id, such as vni, etc.), encapsulation options (encapsulate options).

[0105] In this way, through the unified abstract design of detection parameters, the agent or independent detection module performs detection processing according to different detection types and different environments, ensuring adaptation to various environments and improving resource utilization.

[0106] 3. Callback manager: used to process callbacks when the state changes. It supports modular design and processing, mainly supporting (http, kafka, etc.).

[0107] 4. Result analyzer: used for analysis and judgment of fault switching and recovery.

[0108] 5. Detection result cache management (cache manager): connect to various caches or time series databases, save the reported results in shards (can be used as independent components), and the agent pulls the cache of the detection configuration.

[0109] Among them, callback manager, result analyzer and cache manager can be integrated into separate components. The contents of server components are independent of each other, so they can be expanded horizontally.

[0110] The agent consists of the following four modules:

[0111] 1. Probe manager: Initiates probe requests based on the probe configuration.

[0112] 2. Result reporting (reporter): Analyzing the detection results on the server side often consumes more CPU, and horizontal expansion needs to be considered. Therefore, for the detection results generated or collected by the agent, the health check ID can be consistently hashed and reported to the server in batches for unified processing.

[0113] 3. Detection adapter (driver): Provide modular detection plug-ins according to the deployment environment, such as OVS packet in.

[0114] 4. Configuration puller: Get the server's detection configuration. Perform consistent hashing on the hostname, obtain the address of a specific server, and then request configuration to achieve load balancing and reduce the amount of data migration after a server exception.

[0115] Among them, the probe manager can be considered to be an independent additional component, and for the sake of performance, grpc can be considered to be used for communication between the agent and the server.

[0116] Based on the same inventive concept, an embodiment of the present invention provides a health examination method. Figure 6 , Figure 6 is a flow chart of a health check method provided by an embodiment of the present invention. The health check method of this embodiment is applied to the server end of the health check system described in any of the above embodiments, such as Figure 6 As shown, the health check method includes at least the following steps:

[0117] Step S31: receiving a creation request sent by the console, where the creation request is used to create a health check task;

[0118] Step S32: determining the target detector deployed in the agent based on the creation request, and sending detector confirmation information to the agent corresponding to the target detector;

[0119] Step S33: receiving and based on the pull request sent by the agent corresponding to the target detector, obtaining the detection configuration corresponding to the health check task and returning it to the agent, so that the agent detects the detection object through the target detector according to the detection configuration, obtains the detection result, and reports the detection result to the server;

[0120] Step S34: querying an original detection result of the detection object, comparing the original detection result with the detection result, and determining that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

[0121] Optionally, the method further comprises:

[0122] When the state of the detection object changes, a callback is performed according to the callback processing method in the detection configuration to notify the corresponding business system.

[0123] Optionally, the method further comprises:

[0124] It is also used to manage the information of the detector, and the information of the detector includes at least: load information, label information and area information;

[0125] The “determining the target detector deployed in the agent” in the above step S32 includes:

[0126] According to the minimum load information, label information and area information, the target detector for performing the health check task is selected based on the actual packet sending volume and CPU consumption;

[0127] The method further comprises:

[0128] An agent deployed with the target detector is determined as an agent for obtaining detection configuration, and detector confirmation information is sent to an agent corresponding to the target detector, so that the agent sends a pull request to the server based on the detector confirmation information.

[0129] Optionally, the method further comprises:

[0130] Connect to multiple caches or time series databases, and store the detection results and / or the detection configuration fragments in a distributed cache.

[0131] Optionally, there are multiple agents, there are multiple servers, the multiple servers are deployed in different areas respectively, and the multiple agents in each area are deployed in physical machines, virtual machines and / or containers in the area;

[0132] The system further comprises: an external service end, wherein the external service end comprises at least: an intelligent operation and maintenance system;

[0133] The method further comprises:

[0134] The detection result is mirrored, and the mirrored detection result is sent to the intelligent operation and maintenance system, so that the intelligent operation and maintenance system receives the mirrored detection result sent by the server, and based on the mirrored detection result, analyzes the performance and quality of the detection object to obtain an analysis result.

[0135] Optionally, the method further comprises:

[0136] Receive the pull request, and determine the agentID according to the hostname in the pull request;

[0137] Based on the agentID and the health check association table, a health check ID is obtained, wherein the health check association table includes a mapping relationship between the agentID and the health check ID;

[0138] Based on the health check ID and the associated health check table, query the detection configuration corresponding to the health check ID, and return the detection configuration to the agent; the health check table includes a mapping relationship between the health check ID and the detection configuration;

[0139] The detection configuration includes at least any one of the following: detection frequency, health threshold, abnormal threshold and detection type.

[0140] Based on the same inventive concept, an embodiment of the present invention provides a health examination method. Figure 7 , Figure 7 is a flow chart of a health check method provided by an embodiment of the present invention. The health check method of this embodiment is applied to the agent in the health check system described in any of the above embodiments, such as Figure 7 As shown, the health check method includes at least the following steps:

[0141] Step S41: receiving the detector confirmation message sent by the server; the detector confirmation message is generated by the server after receiving the creation request sent by the console and determining the target detector deployed in the agent based on the creation request, and the creation request is used to create a health check task;

[0142] Step S42: Send a pull request to the server to obtain the detection configuration corresponding to the health check task returned by the server;

[0143] Step S43: According to the detection configuration, the detection object is detected by the target detection machine to obtain a detection result, and the detection result is reported to the server, so that the server queries the original detection result of the detection object, compares the original detection result with the detection result, and determines that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

[0144] Optionally, there are multiple servers, and the above step S42 may include:

[0145] Perform consistent hashing on the hostname to determine the first server among multiple servers;

[0146] The pull request is sent to the first server based on the address of the first server to obtain the detection configuration in the first server.

[0147] Optionally, there are multiple servers, and "reporting the detection result to the server" in the above step S43 may include:

[0148] Performing consistent hashing on the health check ID of the health check task to determine a second server among the multiple servers;

[0149] Report the detection result to the second server.

[0150] like Figure 8 As shown, Figure 8 This is an overall flow chart of a health check method proposed in one embodiment of the present invention. Figure 8 In the example, the health check is performed using the website monitoring scenario. The processing flow of other business scenarios, such as load balancing, ECMP, dynamic DNS, uptime monitoring service, etc., is similar. The health check method is applied to a health check system, which at least includes: a console, a server, an agent, and an external server. The health check includes the following steps:

[0151] Step 100, add a detector and register a detector (create node and register node): The node here can be a detector. The deployed detector can be added through the console or by starting the agent to automatically register. When adding, set the region and label according to the actual situation to facilitate scheduling. The load of the created node defaults to 0. Repeat this step according to business needs until there are enough detectors.

[0152] Step 101, create health check: the console requests the server through HTTP to create a health check.

[0153] Step 102, select the detector by load (schedule node by load): The server selects the appropriate detector according to the minimum load and label, and the region. The selected strategy is to be as dispersed as possible among the regions, and one can be selected for each region. Filter according to the type of health check and tenant information, and try to ensure that there are more than two detectors. The scheduling algorithm can be adjusted as needed. After writing to the persistent database, return.

[0154] Step 103, pull health check configuration by hostname: The server can notify the agent through grpc, allowing the agent to quickly inform the business changes. You can also wait until the agent's scheduled task is pulled through grpc. The agent selects the server using a consistent hashing algorithm to ensure uniform requests. The number of servers can be obtained through additional service discovery or DNS systems. After receiving the pull request, the server queries the node according to the parameter hostname, associates NodeHcAssociation, and then associates healthcheck to query the health check configuration that needs to be returned.

[0155] Step 104, perform consistent hashing based on the health check id to obtain the detection result (report result by consistent hashing hc id): After the agent obtains the configuration, it can use the coroutine or process of the current process for detection, or use an additional program for detection, and then collect detailed detection results. The detection frequency is generally once every 1-60 minutes. The collected detection results are calculated by consistent hashing using the health check id, and the server list is obtained through an additional service discovery system or DNS as in step 103. After calculating the mapping relationship hc_id->server_ip, after aggregating server_ip->[]{hc_id}, it is reported to the server in batches.

[0156] Step 105, analyze the result by checking the status and result: the server queries the status of the existing detection object, and if there is a difference compared with the detection result, it is necessary to trigger the status judgment. For example, if the original detection status is active, and the current detection result is abnormal (packet loss or keyword not found), the status judgment method can be abnormal or normal detection machine number / effective total detection machine number.

[0157] Step 106, callback if the status is changed: If the status of the detection target changes, a callback is performed according to the content of the callback to notify the corresponding business system (i.e. Figure 8 For example, email alerts, traffic switching, etc. In addition, the health check system can also add observable monitoring in other dimensions for troubleshooting.

[0158] The health check method provided in this embodiment has at least the following advantages:

[0159] 1. After the model abstraction is unified and the agent detection method is plug-in, it can be applied to most scenarios, such as load balancing, ecmp, website survival status, etc. It also supports centralized management.

[0160] 2. After supporting custom callbacks, you can connect to various business or notification systems. If the health check system does not change, there will be no stability issues.

[0161] 3. Horizontal expansion can be achieved through data sharding, consistent hashing, and more detailed traffic scheduling calculations, thereby supporting larger-scale business support.

[0162] This will at least achieve the following: 1. Unified management to reduce maintenance costs; 2. Productization to provide standard health check capabilities; 3. Undertaking large-scale business.

[0163] That is to say, this embodiment can improve the horizontal expansion and business expansion capabilities of the system through data sharding, unified model abstraction, and support for custom callbacks; the detection machine is scheduled according to the actual packet sending volume and CPU consumption to make the detection traffic and load more balanced; the detection results are reported after consistency hashing, which can make the server's COU load more balanced; the agent supports modular design, which is convenient for expanding multiple standard detection methods; the detection model is uniformly abstracted for easy maintenance; and custom callbacks are supported for convenience of various business systems.

[0164] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0165] As for the method embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.

[0166] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0167] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0168] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0173] The above is a detailed introduction to a health check system and method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A health check system, characterized in that: The system at least includes: a console, a server and an agent; The console is used to send a creation request to the server, wherein the creation request is used to create a health check task; The server is used to determine the target detector deployed in the agent based on the creation request; The agent is used to send a pull request to the server to obtain the detection configuration corresponding to the health check task in the server; The agent is used to detect the detection object through the target detection machine according to the detection configuration, obtain the detection result, and report the detection result to the server; The server is used to query the original detection result of the detection object, compare the original detection result with the detection result, and determine that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

2. The health check system according to claim 1, characterized in that: The server is also used for: When the state of the detection object changes, a callback is performed according to the callback processing method in the detection configuration to notify the corresponding business system.

3. The health check system according to claim 1, characterized in that: There are multiple servers, and a pull request is sent to the server to obtain the detection configuration corresponding to the health check task in the server, including: Perform consistent hashing on the hostname to determine the first server among multiple servers; The pull request is sent to the first server based on the address of the first server to obtain the detection configuration in the first server.

4. The health check system according to claim 1, characterized in that: There are multiple servers, and reporting the detection results to the servers includes: Performing consistent hashing on the health check ID of the health check task to determine a second server among the multiple servers; Report the detection result to the second server.

5. The health check system according to claim 1, characterized in that: The server is also used to manage the information of the detector, and the information of the detector includes at least: load information, label information and area information; Determining the target detector deployed in the agent includes: According to the minimum load information, label information and area information, the target detector for performing the health check task is screened out according to the actual packet sending volume and CPU consumption; The server is further configured to determine the agent where the target detector is deployed as the agent for obtaining the detection configuration, and send the detector confirmation information to the agent corresponding to the target detector; The agent is specifically used to send a pull request to the server based on the detection machine confirmation information.

6. The health check system according to claim 1, characterized in that: The server is also used for: Connect to multiple caches or time series databases, and store the detection results and / or the detection configuration fragments in a distributed cache.

7. The health check system according to claim 1, characterized in that: There are multiple agents, there are multiple servers, the multiple servers are deployed in different areas respectively, and the multiple agents in each area are deployed in physical machines, virtual machines and / or containers in the area; The system further includes: an external server, which at least includes: The intelligent operation and maintenance system is used to receive the mirrored detection result sent by the server, and based on the mirrored detection result, analyze the performance and quality of the detection object to obtain an analysis result.

8. The health check system according to any one of claims 1 to 7, characterized in that: The server is also used for: Receive the pull request, and determine the agentID according to the hostname in the pull request; Based on the agentID and the health check association table, a health check ID is obtained, wherein the health check association table includes a mapping relationship between the agentID and the health check ID; Based on the health check ID and the associated health check table, query the detection configuration corresponding to the health check ID, and return the detection configuration to the agent; the health check table includes a mapping relationship between the health check ID and the detection configuration; The detection configuration includes at least any one of the following: detection frequency, health threshold, abnormal threshold and detection type.

9. A health examination method, characterized in that: Applied to the server of the health check system according to any one of claims 1 to 8, the method comprising: Receive a creation request sent by the console, where the creation request is used to create a health check task; Determine the target detector deployed in the agent based on the creation request, and send detector confirmation information to the agent corresponding to the target detector; Receive and, based on the pull request sent by the agent corresponding to the target detector, obtain the detection configuration corresponding to the health check task and return it to the agent, so that the agent detects the detection object through the target detector according to the detection configuration, obtains the detection result, and reports the detection result to the server; An original detection result of the detection object is queried, and the original detection result is compared with the detection result. If there is a difference between the original detection result and the detection result, it is determined that the state of the detection object has changed.

10. A health examination method, characterized in that: An agent used in a health check system according to any one of claims 1 to 8, the method comprising: Receive the detector confirmation information sent by the server; the detector confirmation message is generated by the server after receiving the creation request sent by the console and determining the target detector deployed in the agent based on the creation request, and the creation request is used to create a health check task; Send a pull request to the server to obtain the detection configuration corresponding to the health check task returned by the server; According to the detection configuration, the detection object is detected by the target detection machine to obtain a detection result, and the detection result is reported to the server, so that the server queries the original detection result of the detection object, compares the original detection result with the detection result, and determines that the state of the detection object has changed when there is a difference between the original detection result and the detection result.

Images