Monitoring method, computer equipment and storage medium of distributed system

By deploying monitoring components in a distributed system, the coordination service components and data processing components of exceptions are monitored and automatically restarted in real time, the problems of low manual processing efficiency and low recovery success rate are solved, and the automated exception handling capabilities of distributed systems are improved.

CN114138551BActive Publication Date: 2024-11-15RUIJIE NETWORKS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111290386.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-02
Publication Date
2024-11-15
Estimated Expiration
2041-11-02

AI Technical Summary

Technical Problem

Existing distributed systems often experience abnormalities during the operation of data processing components, resulting in low manual processing efficiency and low recovery success rate.

Method used

Deploy monitoring components in a distributed system to monitor the running status of coordination service components and data processing components in real time. Use an automatic restart mechanism to restart the components when an exception occurs until the component resumes normal operation, limiting the number of restarts to avoid frequent restarts.

Benefits of technology

The automatic discovery and automatic recovery of abnormalities in distributed systems is realized, which improves processing efficiency and recovery success rate, and reduces the risk of manual intervention and abnormal aggravation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114138551B_ABST
    Figure CN114138551B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a monitoring method, computer equipment and storage medium for a distributed system. In the embodiment of the present application, any monitoring component in the distributed monitoring service monitors the operating status of the coordination service component and the data processing component deployed on the same service node, and when the operating status of the corresponding coordination service component is monitored to be abnormal, an automatic restart mechanism is used to restart the corresponding coordination service component and data processing component. In addition, if only the operating status of the data processing component is monitored to be abnormal, a restart mechanism is also used to restart the corresponding data processing component. Thereby, the automatic discovery and automatic recovery of the anomalies of the distributed system are realized, and the efficiency of the automated exception handling from automatic discovery of the anomaly to automatic recovery of the anomaly is higher, and the success rate of the abnormal recovery is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a monitoring method, computer equipment and storage medium for a distributed system. Background Art

[0002] At present, distributed systems provide data processing components to the outside world through distributed data processing components deployed on them. At the same time, in order to facilitate the solution of some data management problems often encountered by distributed data processing components, distributed systems usually also deploy distributed application coordination services. Among them, distributed application coordination services can provide unified naming services, state synchronization services, cluster management, management of distributed data processing component configuration items, etc. Distributed application coordination services are, for example, zookeeper software or etcd software, among which zookeeper is software that provides consistency services for distributed data processing components, and the functions provided include: configuration maintenance, domain name services, distributed synchronization, group services, etc. etcd software is implemented based on the Go language, which helps to achieve the availability and consistency of distributed system data.

[0003] In actual applications, the coordination service component in the distributed application coordination service or the data processing component in the distributed data processing component may have an exception during operation. Usually, in this case, it is necessary to notify the administrator immediately to investigate the cause of the exception and perform targeted exception recovery. After the exception is recovered, the coordination service or data processing component is restarted to continue to provide the corresponding service. However, the efficiency of manual exception handling is low, and the success rate of exception recovery is low. Summary of the invention

[0004] Multiple aspects of the present application provide a distributed system monitoring method, a computer device, and a storage medium to implement automated exception handling for the distributed system, thereby improving exception handling efficiency and the success rate of exception recovery.

[0005] The embodiment of the present application provides a monitoring method for a distributed system, wherein the distributed system includes multiple service nodes, each of which is deployed with a coordination service component, a data processing component, and a monitoring component, and the operation of the data processing component depends on the normal operation of the coordination service component; the method includes:

[0006] The first monitoring component monitors the running status of the first coordination service component and the first data processing component deployed on the same service node; the first monitoring component is any monitoring component;

[0007] If it is monitored that the running state of the first coordination service component is abnormal, the first coordination service component and the first data processing component are controlled to terminate their running, and the first coordination service component is controlled to restart until the first coordination service component enters a normal running state before the number of restarts in this round reaches a first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and in the case where the first coordination service component enters a normal running state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters a normal running state before the number of restarts in this round reaches a second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts;

[0008] If the operating status of the first data processing component is monitored to be abnormal, only the first data processing component is controlled to terminate operation, and the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0009] The embodiment of the present application also provides a computer device, including: a memory and a processor;

[0010] Memory for storing computer programs;

[0011] The processor is coupled to the memory and is used to execute the computer program for executing the steps in the monitoring method of a distributed system.

[0012] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps in the monitoring method of a distributed system.

[0013] In an embodiment of the present application, any monitoring component in the distributed monitoring service monitors the operating status of the coordination service component and the data processing component deployed on the same service node, and when the operating status of the corresponding coordination service component is monitored to be abnormal, the automatic restart mechanism is used to restart the corresponding coordination service component and the data processing component. In addition, if only the operating status of the data processing component is monitored to be abnormal, the restart mechanism is also used to restart the corresponding data processing component. This realizes the automatic discovery and recovery of abnormalities in the distributed system, and the efficiency of the automated abnormality handling from automatic discovery of abnormalities to automatic recovery of abnormalities is higher, and the success rate of abnormal recovery is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 A schematic diagram of the structure of a distributed system provided for an exemplary embodiment of the present application;

[0016] Figure 2 A flow chart of a monitoring method for a distributed system provided by an exemplary embodiment of the present application;

[0017] Figure 3 A flowchart of another distributed system monitoring method provided by an exemplary embodiment of the present application;

[0018] Figure 4 A schematic diagram of the structure of a computer device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0020] In view of the low efficiency of the existing manual exception handling method and the low success rate of exception recovery, the embodiment of the present application provides a monitoring method, computer equipment and storage medium for a distributed system. In the embodiment of the present application, any monitoring component in the distributed monitoring service monitors the operating status of the coordination service component and the data processing component deployed on the same service node, and when the operating status of the corresponding coordination service component is monitored to be abnormal, an automatic restart mechanism is used to restart the corresponding coordination service component and data processing component. In addition, if only the operating status of the data processing component is monitored to be abnormal, a restart mechanism is also used to restart the corresponding data processing component. Thereby, the automatic discovery and automatic recovery of the abnormalities of the distributed system are realized, and the efficiency of the automated exception handling from automatic discovery of the abnormalities to automatic recovery of the abnormalities is higher, and the success rate of abnormal recovery is higher.

[0021] Figure 1 The following is a schematic diagram of a distributed system structure provided by an exemplary embodiment of the present application. Figure 1As shown, a distributed application coordination service, a distributed data processing service and a distributed monitoring service are deployed on the system. Among them, the distributed system includes multiple service nodes 101, and each service node 101 can be a server. A coordination service component 104 in the distributed application coordination service, a data processing component 102 in the distributed data processing service and a monitoring component 103 in the distributed monitoring service are deployed on any service node 101, and the operation of the data processing component 102 depends on the normal operation of the coordination service component 104, that is, the normal operation of the coordination service component 104 deployed on the same service node 101 is a prerequisite for the normal operation of the associated data processing component 102. Among them, the coordination service component 104 deployed on the same service node 101 is respectively connected to the associated monitoring component 103 and data processing component 102. In addition, the monitoring component 103 and the data processing component 102 deployed on the same service node 101 interact with data through the coordination service component 104.

[0022] Among them, the distributed application coordination service is a software that provides consistency services for distributed applications. It is mainly used to solve some data management problems often encountered in distributed applications, such as: unified naming service, state synchronization service, cluster management, management of distributed application configuration items, etc. Among them, the distributed application coordination service includes, but is not limited to: zookeeper, etcd. Zookeeper is a reliable coordination system for distributed systems. The functions it provides include: configuration maintenance, domain name service, distributed synchronization, group service, etc. The goal of etcd is to build a highly available distributed key-value database. etcd uses the raft protocol as the consistency algorithm internally, and etcd is implemented based on the Go language.

[0023] In an embodiment of the present application, a distributed application coordination service is formed by coordination service components distributed on various service nodes in a distributed system, and various coordination service components belonging to the distributed application coordination service coordinate their work with each other.

[0024] The distributed data processing service is a system composed of multiple data processing components 102 deployed on a distributed system. The multiple data processing components 102 communicate through a network and jointly provide data processing services to the outside. In specific applications, the distributed data processing service can be any system that provides various services, such as but not limited to a distributed computing system, a distributed storage system, or a distributed file system.

[0025] The distributed monitoring service is a system composed of multiple monitoring components 103 deployed on a distributed system, which jointly provide monitoring services to the outside world.

[0026] In the embodiment of the present application, any monitoring component 103 in the distributed monitoring service monitors the running status of the coordination service component 104 and the data processing component 102 deployed on the same service node 101, and when the running status of the corresponding coordination service component 104 is monitored to be abnormal, the automatic restart mechanism is used to restart the corresponding coordination service component 104 and the data processing component 102. In addition, if only the running status of the data processing component 102 is monitored to be abnormal, the restart mechanism will also be used to restart the corresponding data processing component 102. In this way, the automatic discovery and recovery of the abnormalities of the distributed system are realized, and the efficiency of the automated abnormality handling from the automatic discovery of the abnormalities to the automatic recovery of the abnormalities is higher, and the success rate of the abnormality recovery is higher.

[0027] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0028] Figure 2 A flow chart of a monitoring method for a distributed system provided by an exemplary embodiment of the present application. Figure 2 , the method comprises the following steps:

[0029] 201. A first monitoring component monitors the running status of a first coordination service component and a first data processing component deployed on the same service node.

[0030] 202. If it is monitored that the operating state of the first coordination service component is abnormal, the first coordination service component and the first data processing component are controlled to terminate operation, and the first coordination service component is controlled to restart until the first coordination service component enters a normal operating state before the number of restarts in this round reaches a first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and in the case that the first coordination service component enters a normal operating state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters a normal operating state before the number of restarts in this round reaches a second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0031] 203. If it is monitored that the operating status of the first data processing component is abnormal, only the first data processing component is controlled to terminate the operation, and the first data processing component is controlled to restart until the first data processing component enters a normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0032] It is worth noting that the execution order of step 202 can be before step 203 or after step 203, and the embodiment of the present application does not limit this.

[0033] In the embodiment of the present application, the first monitoring component refers to any monitoring component in the distributed monitoring service. The first coordination service component is any coordination service component in the distributed application coordination service deployed on the same service node as the first monitoring component, and the first data processing component is any data processing component in the distributed monitoring service deployed on the same service node as the first monitoring component.

[0034] The first monitoring component first starts the first coordination service component, and after monitoring that the running status of the first coordination service component is normal, starts the first data processing component and monitors the running status of the first data processing component. If the first monitoring component monitors that the running status of the first data processing component is normal, the first monitoring component confirms that the startup tasks of the first coordination service component and the first data processing component are completed, and continues to monitor the running status of the first coordination service component and the first data processing component.

[0035] In specific applications, the first monitoring component may monitor the abnormal operation state of the first coordination service component. In this case, since the operation of the first data processing component depends on the normal operation of the first coordination service component, the abnormal operation state of the first coordination service component will also cause the abnormal operation state of the first data processing component to a certain extent. Therefore, in order to ensure the reliability of the distributed system, when the abnormal operation state of the first coordination service component is monitored, the first coordination service component and the first data processing component are controlled to terminate operation, and the automatic restart mechanism is used to restart the first coordination service component and the first data processing component.

[0036] In specific applications, the first monitoring component may also monitor that the running state of the first coordination service component is normal, but the running state of the first data processing component is abnormal. In this case, it is only necessary to restart the first data processing component.

[0037] The embodiment of the present application does not limit the monitoring method of the running status of the first coordination service component. For example, the running status of the first coordination service component can be judged to be normal or abnormal by checking the process status of the first coordination service component. If the process status of the first coordination service component is normal, the running status of the first coordination service component is normal; if the process status of the first coordination service component is abnormal, the running status of the first coordination service component is abnormal. As an example, the process status of the first coordination service component can be checked by a system daemon process management tool. For example, the system daemon process management tool is systemd, and the systemd can use the "systemctl status+service name" command to query the process status, and judge by the keywords returned by the command. When the return value of the status field is "active (running)", it is confirmed that the process of the object identified by the service name is in a normal state. When the return value of the status field is not "active (running)", it is confirmed that the process of the object identified by the service name is in an abnormal state. As another example, the system is queried by the process name or the path of the executable file to see if there is a corresponding process. If it exists, the process status of the first coordination service component is normal, and if it does not exist, the process status of the first coordination service component is abnormal. For another example, the running state of the first coordination service component may be queried through a mechanism provided by the distributed application coordination service, wherein the distributed application coordination service manages and maintains the running state of each coordination service component.

[0038] The embodiment of the present application does not limit the monitoring method of the running status of the first data processing component. For example, the running status of the first data processing component can be checked to determine whether it is normal or abnormal. If the process status of the first data processing component is abnormal, the running status of the first data processing component is abnormal; if the process status of the first data processing component is normal, the communication connection status between the first data processing component and the first coordination service component is combined to jointly determine whether the running status of the first data processing component is normal or abnormal. Regarding the method for checking the process status of the first data processing component, please refer to the method for checking the process status of the first coordination service component described in the aforementioned content, which will not be repeated here.

[0039] Further optionally, an optional implementation method for jointly judging whether the running status of the first data processing component is normal or abnormal in combination with the communication connection status between the first data processing component and the first coordination service component is: query whether there is a coordination service component including a temporary node corresponding to the first data processing component among multiple coordination service components, and the temporary node is created after the first data processing component and the first coordination service component establish a communication connection; if there is a coordination service component including a temporary node corresponding to the first data processing component among multiple coordination service components, the running status of the first data processing component is normal; if there is no coordination service component that does not include the temporary node corresponding to the first data processing component among multiple coordination service components, the running status of the first data processing component is abnormal.

[0040] It is worth noting that the first monitoring component can send a query request to the first coordination service component, and the first coordination service component responds to the query request to locally query the temporary node corresponding to the first data processing component; and forward the query request to other coordination service components, and other coordination service components respond to the query request and return the query results to the first coordination service component; the first coordination service component collects the query results of other coordination service components and its own query results, and feeds back all the query results to the first monitoring component so that the first monitoring component can confirm whether there is a corresponding temporary node in the coordination service component.

[0041] It should be noted that after the first data processing component establishes a communication connection with the first coordination service component, the first data processing component sends a temporary node creation request to any coordination service component in the distributed application coordination service through the coordination service client API (Application Programming Interface) to request any coordination service component to create a temporary node that can instruct the first data processing component to establish a communication connection with the first coordination service component.

[0042] For example, a distributed application coordination service usually provides a keep-alive mechanism for the server and the client. Taking Zookeeper as an example, Zookeeper saves the temporary node created when the coordination service client API in any data processing component establishes a communication connection with the corresponding coordination service component. When the coordination service client API is disconnected from the corresponding coordination service component, the associated temporary node is deleted from Zookeeper. Therefore, the first data processing component can query whether there is a coordination service component on the distributed application coordination service that includes the temporary node corresponding to the first data processing component to further confirm whether the operating status of the first data processing component is abnormal or normal.

[0043] Further optionally, if no coordination service component including the temporary node corresponding to the first data processing component is found, the first monitoring component sends a first modification request to at least one of the multiple coordination service components, so that at least one of the multiple coordination service components changes the operating status of the first data processing component stored in the status management form from an online status to an offline status.

[0044] It is worth noting that when one or more coordination service components in the distributed application coordination service receive the first modification request sent by the first monitoring component, the multiple coordination service components cooperate with each other to decide one or more coordination service components to modify the state management form. When it is decided that multiple coordination service components modify the state management form, each coordination service component can be controlled to modify the content of different parts of the state management form respectively. In addition, when only one coordination service component receives the first modification request sent by the first monitoring component, in addition to requesting multiple coordination service components to cooperate with each other and decide one or more coordination service components to modify the state management form, the coordination service component can also modify the state management form in response to the first modification request.

[0045] Among them, a globally unique state management form is maintained and managed in the distributed application coordination service. The state management form records the running state of each data processing component in the distributed data processing service. If the running state recorded in the state management form of the data processing component is an online state, it means that the data processing component has not been separated from the distributed data processing service, that is, the data processing component belongs to a node of the distributed data processing service. If the running state recorded in the state management form of the data processing component is an offline state, it means that the data processing component is separated from the distributed data processing service, that is, the data processing component does not belong to a node of the distributed data processing service.

[0046] In the embodiment of the present application, after the running state of the first coordination service component changes from normal to abnormal, a new round of restarting the first coordination service component is performed. For each round of restarting the first coordination service component, the number of restarts in this round is less than or equal to the preset first maximum number of restarts, wherein the first maximum number of restarts can be flexibly set according to actual application requirements.

[0047] Similarly, if the running state of the first data processing component changes from normal to abnormal, a new round of restarting the first data processing component is performed. Alternatively, if the first coordination service component needs to be restarted, a new round of restarting the first data processing component is performed. For each round of restarting the first coordination service component, the number of restarts in this round is less than or equal to the preset second maximum number of restarts, where the second maximum number of restarts can be flexibly set according to actual application requirements.

[0048] It is worth noting that the embodiment of the present application can automatically recover some self-healing anomalies by controlling the number of restarts of the first coordination service component and the first data processing component, without the need for manual intervention in abnormal fault troubleshooting. If the operating state is still abnormal when the number of restarts of the first coordination service component in this round reaches the first maximum number of restarts, or the operating state is still abnormal when the number of restarts of the first data processing component in this round reaches the second maximum number of restarts, it means that there are currently abnormalities in the distributed system that cannot be self-healed, and manual intervention is required for abnormal fault troubleshooting. At this time, the first monitoring component can output an alarm message, and the alarm message is used to prompt the use of manual exception handling methods for processing. For example, the alarm message can be output to the terminal device of the operation and maintenance personnel, and the operation and maintenance personnel view the alarm message on the terminal device and start the abnormal fault troubleshooting work. Thereby, the entire monitoring system for the distributed system is made more complete, and the monitoring of various abnormalities and closed-loop processing after the abnormality can be realized.

[0049] In addition, the restart time interval of the first data processing component and the restart time interval of the first coordination service component are set according to actual application requirements. It should be understood that the time interval between two adjacent restarts of the first data processing component or the first coordination service component needs to be greater than or equal to the corresponding restart time interval.

[0050] It is worth noting that the distributed monitoring service can fully automatically realize the recovery of abnormal data processing components and introduce restart restrictions during the recovery process, which improves the success rate of abnormal recovery of distributed data processing services and reduces the risk of frequent restarts causing aggravation of data processing component abnormalities.

[0051] The monitoring method of the distributed system provided by the embodiment of the present application, any monitoring component in the distributed monitoring service monitors the running status of the coordination service component and the data processing component deployed on the same service node, and when the running status of the corresponding coordination service component is monitored to be abnormal, the automatic restart mechanism is used to restart the corresponding coordination service component and the data processing component. In addition, if only the running status of the data processing component is monitored to be abnormal, the restart mechanism will also be used to restart the corresponding data processing component. Then the automatic discovery and automatic recovery of the abnormality of the distributed system are realized, and the efficiency of the automated abnormal processing from the automatic discovery of the abnormality to the automatic recovery of the abnormality is higher, and the success rate of abnormal recovery is higher. In addition, the monitoring method of the distributed system provided by the embodiment of the present application can find the data processing component with accurate abnormality in seconds, realize the abnormal monitoring and discovery capability close to real-time, shorten the time from abnormal discovery to recovery, reduce the impact of the abnormality, reduce the probability of abnormal misjudgment and reduce the risk of abnormal aggravation of the data processing component. A strict restart number limit scheme is used to prevent frequent restarts from causing the data processing component to fall into more serious abnormalities.

[0052] In the above or below embodiments of the present application, in order to improve the success rate of restarting the first data processing component, the first monitoring component may further reconfigure the operating environment of the first data processing component before controlling the first data processing component to restart, and control the first data processing component to restart after configuring the operating environment of the first data processing component. Reconfiguring the operating environment includes but is not limited to configuring the cluster, network configuration, CPU (central processing unit), memory, hard disk, etc.

[0053] It is noteworthy that the first data processing component is restored only after its operating environment is reconfigured, which fully guarantees the normal restoration of the first data processing component and ensures that the restoration of a data processing component can be fully automatically achieved without administrator intervention.

[0054] In the above or following embodiments of the present application, an implementation process of restarting the first data processing component is: determining whether the cause of the abnormal state of the first data processing component during the restart process comes from inside the first data processing component or outside the first data processing component; if it comes from inside the first data processing component, then the number of restarts of the first data processing component in this round is increased by one; if it comes from outside the first data processing component, then the number of restarts of the first data processing component in this round is kept unchanged; and when the number of restarts of the first data processing component in this round is less than the second maximum number of restarts, controlling the first data processing component to restart; if the restart fails, returning to determine whether the cause of the abnormal state of the first data processing component during the restart process comes from inside the first data processing component or outside the first data processing component.

[0055] In an embodiment of the present application, when the first data processing component is abnormal due to external reasons such as the network, the number of restarts in this round is not increased. When the first data processing component program causes an abnormality due to an internal error, the number of restarts in this round is increased. In actual application, the system daemon management tool can be used to check whether the abnormal cause of the first data processing component is internal or external. For example, the system daemon management tool is systemd, and the systemd can use the "systemctl status+service name" command to query the process status and judge by the keywords returned by the command. When the return value of the status field is not "active (running)", it is confirmed that the process of the first data processing component is in an abnormal state. When the returned signal field is "SEGV" or "ABRT", and the external environment is checked to be normal (for example, whether the network can be connected to other nodes in the cluster), it is considered that an internal abnormality has occurred; otherwise, it is considered that an external abnormality has occurred. Checking that the external environment is normal, for example, means that the first data processing component can be connected to other nodes in the cluster, and other nodes in the cluster include the coordination service component in the distributed application coordination service, the monitoring component in the distributed monitoring service, or the data processing component in the distributed data processing service.

[0056] Generally, external anomalies do not have a fatal impact on the first data processing component, and the first monitoring component in the distributed monitoring service can automatically restore the first data processing component. In contrast, internal anomalies are more serious, and the distributed monitoring service needs to limit the number of restarts of the first data processing component. It is worth noting that distinguishing whether the cause of the abnormal state of the first data processing component comes from inside the first data processing component or outside the first data processing component can further improve the automation and accuracy of the distributed monitoring service.

[0057] In the above or following embodiments of the present application, the first monitoring component starts the first coordination service component and monitors that the running status of the first coordination service component is normal. Before starting the first data processing component for the first time, the first monitoring component sends election information to at least one coordination service component among the multiple coordination service components, so that at least one coordination service component requests the multiple coordination service components to negotiate together to elect a main monitoring component from the multiple monitoring components; and sends a registration request to the first coordination service component to receive various exception notification information sent by the first coordination service component in the case of successful registration. Among them, the registration request is used to request the distributed application coordination service to register the first monitoring component as a monitoring component that receives various exception notification information. Among them, the various exception notification information sent by the first coordination service component includes various exception notification information sent by other coordination service components to the first coordination service component.

[0058] Further optionally, the first monitoring component locally registers first abnormality notification information of abnormal connection between the first monitoring component and the first coordination service component. It is worth noting that when the connection between the first monitoring component and the first coordination service component is abnormal, the first monitoring component can detect the first abnormality notification information locally; when the connection between the first monitoring component and the first coordination service component is normal, the first monitoring component will not detect the first abnormality notification information locally.

[0059] It is worth noting that by utilizing the notification mechanism provided by the distributed application coordination service, faster anomaly detection capabilities can be obtained with less system overhead.

[0060] In the above or following embodiments of the present application, when the first coordination service component and the first data processing component are both in normal operating state, if the first monitoring component locally detects the first abnormal notification information indicating that the connection between the first monitoring component and the first coordination service component is abnormal, the first monitoring component controls the first coordination service component and the first data processing component to terminate operation, and controls the first coordination service component to restart until the first coordination service component enters normal operating state before the number of restarts in this round reaches the first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and when the first coordination service component enters normal operating state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0061] It is worth noting that the abnormal connection between the first monitoring component and the first coordination service component may refer to the disconnection of the communication connection between the coordination service client API in the first monitoring component and the connection with the first coordination service component. When the first monitoring component is connected to the first coordination service component normally, there is a coordination service component including a temporary node corresponding to the first monitoring component in the distributed application coordination service. When the first monitoring component is connected to the first coordination service component abnormally, there is no coordination service component including a temporary node corresponding to the first monitoring component in the distributed application coordination service.

[0062] In the above or following embodiments of the present application, when the first coordination service component and the first data processing component are both in normal operating state, if a second abnormal notification information indicating that a data processing component in the distributed system is offline is received from any coordination service component among multiple coordination service components, it is determined whether the offline data processing component is the first data processing component; if so, only the first data processing component is controlled to terminate its operation, and the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0063] Further optionally, if the offline data processing component is not the first data processing component, the first monitoring component determines whether it is the main monitoring component; if the first monitoring component is the main monitoring component, it determines whether the second monitoring component deployed on the same service node as the offline data processing component is online; if the second monitoring component is not online, a second modification request is sent to at least one of the multiple coordination service components, so that at least one of the multiple coordination service components modifies the running state of the offline data processing component stored in the status management form from the online state to the offline state. In addition, if the second monitoring component is online, the second monitoring component itself sends a modification request to at least one of the multiple coordination service components, so that at least one of the multiple coordination service components modifies the running state of the offline data processing component stored in the status management form from the online state to the offline state.

[0064] It is worth noting that when one or more coordination service components in the distributed application coordination service receive the second modification request sent by the first monitoring component, the multiple coordination service components cooperate with each other to decide one or more coordination service components to modify the state management form. When it is decided that one coordination service component modifies the state management form, the decided coordination service component can be controlled to modify the state management form. When it is decided that multiple coordination service components modify the state management form, each coordination service component can be controlled to modify the content of different parts of the state management form respectively. In addition, when only one coordination service component receives the second modification request sent by the first monitoring component, in addition to requesting multiple coordination service components to cooperate with each other and decide one or more coordination service components to modify the state management form, the coordination service component can also modify the state management form in response to the second modification request.

[0065] The offline state of a data processing component can be understood as the data processing component being disconnected from a distributed data processing service, that is, the data processing component does not belong to a node of a distributed data processing service. The online state of a data processing component can be understood as the data processing component not being disconnected from a distributed data processing service, that is, the data processing component belongs to a node of a distributed data processing service.

[0066] It is worth noting that when a data processing component is abnormal, the monitoring component deployed on the same service node can set the offline status of the data processing component. Since the monitoring components constitute a distributed monitoring service, even if the monitoring components deployed on the same service node fail, the main monitoring component can still be used to set the offline status of a data processing component, thereby achieving highly reliable offline status management of data processing components. In addition, the distributed monitoring service runs in the form of a cluster, and the reliability of offline status management of data processing components is improved by the collaboration between local monitoring components and the main monitoring group.

[0067] In order to better understand the solution provided by the embodiment of the present application, Figure 3 This paper introduces a distributed system monitoring method in practical application. Figure 3 , the distributed system monitoring method comprises the following steps:

[0068] 11. The first monitoring component starts the first coordination service component.

[0069] 12. The first monitoring component monitors the running state of the first coordination service component. If the running state of the first coordination service component is abnormal, execute step 13; if the running state of the first coordination service component is normal, execute step 15;

[0070] 13. Terminate the operation of the first coordination service component and the first data processing component (if the first data processing component is started), determine whether the number of restarts of the first coordination service component in this round has reached the first maximum number of restarts, if not, when the interval between the last restart and the last restart reaches the preset restart interval, return to step 11; if the first maximum number of restarts has been reached, execute step 14;

[0071] 14. Output prompt information indicating that the operating status of the first coordination service component is abnormal to the management personnel.

[0072] 15. The first monitoring component establishes a communication connection between the monitoring coordination service (eg, Zookeeper) client API and the first coordination service component, so that the first monitoring component can access the distributed application coordination service. If the connection is established successfully, execute step 16.

[0073] 16. The first monitoring component performs the specified tasks: (1) The first monitoring component participates in the distributed monitoring service master election; (2) Registers the abnormal communication connection notification between the coordination service client API and the first coordination service component in the first monitoring component; (3) Registers the abnormal notification of the data processing component going offline from the distributed data processing service. If (1)(2)(3) tasks are all completed, the first monitoring component completes the task of starting the first coordination service component. After the task of starting the first coordination service component is completed, execute step 17.

[0074] For an introduction to the first monitoring component executing tasks (1), (2), and (3), please refer to the aforementioned record of the first monitoring component sending election information and registration requests to the distributed application coordination service.

[0075] 17. The first monitoring component starts the first data processing component.

[0076] 18. The first monitoring component monitors the operating status of the first data processing component.

[0077] 19. If the running state of the first data processing component is abnormal, terminate the running of the first data processing component, set the state of the first data processing component to offline state, and execute step 20. If the running state of the first data processing component is normal, execute step 22.

[0078] Among them, the implementation method of setting the status of the first data processing component to the offline state can be: the first monitoring component modifies the request to the distributed application coordination service so that the distributed application coordination service changes the running status of the first data processing component stored in the status management form from the online state to the offline state.

[0079] 20. Determine whether the number of restarts of the first data processing component in this round reaches the second maximum number of restarts. If so, execute step 23. If not, execute step 21.

[0080] It is worth noting that if the number of restarts of the first data processing component in this round reaches the second maximum number of restarts, a prompt message indicating that the operating status of the first data processing component is abnormal is output to the administrator to prompt the administrator to perform abnormal recovery manually.

[0081] 21. If the second maximum restart times have not been reached, when the interval between the last restart and the last restart reaches the preset restart interval, reconfigure the operating environment of the first data processing component. When the configuration of the operating environment of the first data processing component is completed, return to step 17.

[0082] 22. After the first coordination service component and the first data processing component are successfully started, the first coordination service component and the first data processing component are continuously monitored.

[0083] 23. Output prompt information indicating that the operating status of the first data processing component is abnormal to the management personnel.

[0084] If the first monitoring component detects locally a notification of abnormal communication connection between the coordination service client API in the first monitoring component and the first coordination service component, the process returns to step 11 .

[0085] If an abnormal notification of a data processing component going offline is received from a distributed application coordination service, the first monitoring component determines whether the offline data processing component is the first data processing component. If so, the first monitoring component sets the state of the first data processing component to an offline state. If not, the first monitoring component determines whether it is the main monitoring component. If not, return to execute step 22. If the first monitoring component is the main monitoring component, then if the monitoring component where the offline data processing component is located is separated from the monitoring cluster (that is, the distributed data processing service), the first monitoring component sets the state of the first data processing component to an offline state. If the first monitoring component is not the main monitoring component, return to execute 22.

[0086] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 201 to 203 can be device A; for another example, the execution subject of steps 201 and 202 can be device A, and the execution subject of step 203 can be device B; and so on.

[0087] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 201, 202, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0088] Figure 4 A schematic diagram of the structure of a computer device provided by another exemplary embodiment of the present application. Figure 4 As shown, the computer device includes: a memory 41 and a processor 42.

[0089] The memory 41 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0090] The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0091] The processor 42 is coupled to the memory 41 and is configured to execute the computer program in the memory 41 to:

[0092] Monitor the running status of the first coordination service component and the first data processing component deployed on the same service node; the first monitoring component is any monitoring component in multiple service nodes of the distributed system, each service node is deployed with a coordination service component, a data processing component and a monitoring component, and the operation of the data processing component depends on the normal operation of the coordination service component;

[0093] If it is monitored that the running state of the first coordination service component is abnormal, the first coordination service component and the first data processing component are controlled to terminate their running, and the first coordination service component is controlled to restart until the first coordination service component enters a normal running state before the number of restarts in this round reaches a first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and in the case where the first coordination service component enters a normal running state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters a normal running state before the number of restarts in this round reaches a second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts;

[0094] If the operating status of the first data processing component is monitored to be abnormal, only the first data processing component is controlled to terminate operation, and the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0095] Further optionally, before controlling the first data processing component to restart, the processor 42 is further configured to:

[0096] Reconfigure the operating environment of the first data processing component, and after configuring the operating environment of the first data processing component, control the first data processing component to restart.

[0097] Further optionally, when the processor 42 monitors the running state of the first data processing component, it is specifically used to:

[0098] Monitoring a process status corresponding to the first data processing component;

[0099] If the process state corresponding to the first data processing component is abnormal, determining that the running state of the first data processing component is abnormal;

[0100] If the process state corresponding to the first data processing component is normal, query whether there is a coordination service component including a temporary node corresponding to the first data processing component among the multiple coordination service components, where the temporary node is created after the first data processing component establishes a communication connection with the first coordination service component;

[0101] If there is a coordination service component including a temporary node corresponding to the first data processing component among the multiple coordination service components, the operation status of the first data processing component is normal;

[0102] If there is no coordination service component including the temporary node corresponding to the first data processing component among the multiple coordination service components, the running state of the first data processing component is abnormal.

[0103] Further optionally, the processor 42 is further configured to:

[0104] If the coordination service component including the temporary node corresponding to the first data processing component cannot be found, the first monitoring component sends a first modification request to at least one of the multiple coordination service components, so that at least one of the multiple coordination service components changes the operating status of the first data processing component stored in the status management form from an online status to an offline status.

[0105] Further optionally, when both the first coordination service component and the first data processing component are in normal operation, the processor 42 is further configured to:

[0106] If the first monitoring component locally detects first abnormal notification information indicating an abnormal connection between the first monitoring component and the first coordination service component, the first coordination service component and the first data processing component are controlled to terminate operation, and the first coordination service component is controlled to restart until the first coordination service component enters normal operating state before the number of restarts in this round reaches the first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and when the first coordination service component enters normal operating state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0107] Further optionally, when both the first coordination service component and the first data processing component are in normal operation, the processor 42 is further configured to:

[0108] If any coordination service component among the multiple coordination service components sends second abnormal notification information indicating that a data processing component in the distributed system is offline, determining whether the offline data processing component is the first data processing component;

[0109] If so, only the first data processing component is controlled to terminate operation, and the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

[0110] Further optionally, the processor 42 is further configured to:

[0111] If the offline data processing component is not the first data processing component, the first monitoring component determines whether it is the main monitoring component;

[0112] If the first monitoring component is the main monitoring component, determining whether a second monitoring component deployed on the same service node as the offline data processing component is online;

[0113] If the second monitoring component is not online, a second modification request is sent to at least one of the multiple coordination service components so that at least one of the multiple coordination service components changes the operating status of the offline data processing component stored in the status management form from an online state to an offline state.

[0114] Further optionally, before the first data processing component is started for the first time, the processor 42 is further configured to:

[0115] The first monitoring component sends election information to at least one coordination service component among the plurality of coordination service components, so that the at least one coordination service component requests the plurality of coordination service components to jointly negotiate to elect a master monitoring component from the plurality of monitoring components; and

[0116] Sending a registration request to the first coordination service component to receive various abnormal notification information sent by the first coordination service component in the case of successful registration, wherein the various abnormal notification information sent by the first coordination service component includes various abnormal notification information sent by other coordination service components to the first coordination service component;

[0117] And the first monitoring component locally registers first abnormal notification information of abnormal connection between the first monitoring component and the first coordination service component. When the first monitoring component and the first coordination service component are abnormally connected, the first monitoring component can detect the first abnormal notification information locally.

[0118] Further optionally, when the processor 42 restarts the first data processing component, it is specifically used to:

[0119] Determine whether the cause of the abnormal state of the first data processing component during the restart process is from inside the first data processing component or outside the first data processing component;

[0120] If it comes from inside the first data processing component, the number of restarts of the first data processing component in this round is increased by one; if it comes from outside the first data processing component, the number of restarts of the first data processing component in this round remains unchanged; and

[0121] When the number of restarts of the first data processing component in this round is less than the second maximum number of restarts, the first data processing component is controlled to restart; if the restart fails, the process returns to determine whether the cause of the abnormal state of the first data processing component during the restart process comes from inside the first data processing component or outside the first data processing component.

[0122] Further, if Figure 4 As shown, the computer device also includes: a communication component 43, a display 44, a power component 45, an audio component 46 and other components. Figure 4 Only some components are shown schematically, and it does not mean that the computer equipment only includes Figure 4 In addition, Figure 4 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the computer device. The computer device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the computer device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 4 If the computer device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 4 Components within the dashed box.

[0123] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by a computer device in the above method embodiment.

[0124] Above Figure 4The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 4G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0125] Above Figure 4 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0126] Above Figure 4 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0127] Above Figure 4 The audio component in can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0128] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0133] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0134] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0135] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0136] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A monitoring method for a distributed system, It is characterized in that The distributed system includes a plurality of service nodes, each of which is deployed with a coordination service component, a data processing component and a monitoring component, and the operation of the data processing component depends on the normal operation of the coordination service component; the method includes: The first monitoring component monitors the running status of the first coordination service component and the first data processing component deployed on the same service node; the first monitoring component is any monitoring component; If it is monitored that the running state of the first coordination service component is abnormal, the first coordination service component and the first data processing component are controlled to terminate running, and the first coordination service component is controlled to restart until the first coordination service component enters a normal running state before the number of restarts in this round reaches a first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and in the case that the first coordination service component enters a normal running state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters a normal running state before the number of restarts in this round reaches a second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts; If it is monitored that the operating status of the first data processing component is abnormal, only the first data processing component is controlled to terminate operation, and the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

2. The method according to claim 1, It is characterized in that Before controlling the first data processing component to restart, the method further includes: Reconfigure the operating environment of the first data processing component, and after configuring the operating environment of the first data processing component, control the first data processing component to restart.

3. The method according to claim 1, It is characterized in that Monitoring the operating status of the first data processing component includes: Monitoring the process status corresponding to the first data processing component; If the process state corresponding to the first data processing component is abnormal, determining that the running state of the first data processing component is abnormal; If the process state corresponding to the first data processing component is normal, query whether there is a coordination service component including a temporary node corresponding to the first data processing component among multiple coordination service components, where the temporary node is created after the first data processing component establishes a communication connection with the first coordination service component; If there is a coordination service component including a temporary node corresponding to the first data processing component among the multiple coordination service components, the operation status of the first data processing component is normal; If there is no coordination service component including the temporary node corresponding to the first data processing component among the multiple coordination service components, the running state of the first data processing component is abnormal.

4. The method according to claim 3, It is characterized in that Also includes: If the coordination service component including the temporary node corresponding to the first data processing component cannot be found, the first monitoring component sends a first modification request to at least one of the multiple coordination service components, so that at least one of the multiple coordination service components changes the operating status of the first data processing component stored in the status management form from an online status to an offline status.

5. The method according to claim 1, It is characterized in that When both the first coordination service component and the first data processing component are in normal operation, the method further includes: If the first monitoring component locally detects first abnormal notification information indicating an abnormal connection between the first monitoring component and the first coordination service component, the first coordination service component and the first data processing component are controlled to terminate operation, and the first coordination service component is controlled to restart until the first coordination service component enters a normal operating state before the number of restarts in this round reaches a first maximum number of restarts or the number of restarts in this round reaches the first maximum number of restarts; and in the case that the first coordination service component enters a normal operating state before the number of restarts in this round reaches the first maximum number of restarts, the first data processing component is controlled to restart until the first data processing component enters a normal operating state before the number of restarts in this round reaches a second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

6. The method according to claim 5, It is characterized in that When both the first coordination service component and the first data processing component are in normal operation, the method further includes: If a second abnormal notification message indicating that a data processing component in the distributed system is offline is received from any coordination service component among the multiple coordination service components, determining whether the offline data processing component is the first data processing component; If so, only the first data processing component is controlled to terminate operation, and the first data processing component is controlled to restart until the first data processing component enters normal operating state before the number of restarts in this round reaches the second maximum number of restarts or the number of restarts in this round reaches the second maximum number of restarts.

7. The method according to claim 6, It is characterized in that Also includes: If the offline data processing component is not the first data processing component, the first monitoring component determines whether it is the main monitoring component; If the first monitoring component is a main monitoring component, determining whether a second monitoring component deployed on the same service node as the offline data processing component is online; If the second monitoring component is not online, a second modification request is sent to at least one of the multiple coordination service components, so that at least one of the multiple coordination service components changes the running status of the offline data processing component stored in the status management form from an online state to an offline state.

8. The method according to claim 7, It is characterized in that Before the first data processing component is started for the first time, the method further includes: The first monitoring component sends election information to at least one coordination service component among the multiple coordination service components, so that the at least one coordination service component requests the multiple coordination service components to negotiate together to elect a master monitoring component from the multiple monitoring components; and Sending a registration request to the first coordination service component to receive various abnormal notification information sent by the first coordination service component in the case of successful registration; wherein the various abnormal notification information sent by the first coordination service component includes various abnormal notification information sent by other coordination service components to the first coordination service component; And the first monitoring component locally registers first abnormality notification information of abnormal connection between the first monitoring component and the first coordination service component. When the first monitoring component and the first coordination service component are abnormally connected, the first monitoring component can detect the first abnormality notification information locally.

9. The method according to any one of claims 1 to 8, It is characterized in that Restarting the first data processing component includes: Determine whether the cause of the abnormal state of the first data processing component during the restart process is from inside the first data processing component or outside the first data processing component; If it comes from inside the first data processing component, the number of restarts of the first data processing component in this round is increased by one; if it comes from outside the first data processing component, the number of restarts of the first data processing component in this round remains unchanged; and When the number of restarts of the first data processing component in this round is less than the second maximum number of restarts, the first data processing component is controlled to restart; if the restart fails, the method returns to determine whether the cause of the abnormal state of the first data processing component during the restart process comes from inside the first data processing component or outside the first data processing component.

10. A computer device, It is characterized in that include: Memory and processor; The memory is used to store computer programs; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 9.