A data center cluster monitoring method, device and system

Abstract

The application relates to the technical field of data processing, in particular to a data center cluster monitoring method, device and system. The method comprises the following steps: S1, each master node receives a synchronization signal sent by a plurality of corresponding redundant nodes, and sends a response signal to the redundant nodes; S2, when the master node cannot receive the synchronization signal of the i-th redundant node or the i-th redundant node cannot receive the response signal, an abnormal command is sent to a monitoring unit, the monitoring unit sends a detection command, and a fault type is determined; S3, when the master node and the i-th redundant node communicate normally, whether the data is synchronized is judged based on the update list in the master node, the data version number of each data type of the i-th redundant node in the master node and the synchronization signal; and S4, a target source node and a target time period are acquired, and target data is sent to the target node for data synchronization in the target time period. The application solves the problem that the cluster node monitoring precision is insufficient, and improves the monitoring precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, more particularly to a data center cluster monitoring method, device and system. BACKGROUND

[0002] Cluster node failure monitoring technology is one of the key technologies to ensure the stable operation of large-scale cluster systems. With the wide application of cloud computing, big data and distributed systems, cluster node failure monitoring technology is also constantly developing and improving. Similar prior art has a Chinese patent with publication number CN115102887A, which discloses a cluster node monitoring method, device, electronic equipment and computer readable storage medium, applied to each node in the cluster system, the method comprises: initiating a heartbeat signal by using a heartbeat monitoring thread; sending the heartbeat signal to each node in the cluster system by using a transmission thread, and receiving the reply signal fed back by each node; determining the offline node in the cluster system according to each reply signal by using the heartbeat monitoring thread, and uploading the corresponding offline node information to the main thread; executing the node state update operation according to the offline node information by using the main thread. The application of the technical solution provided in the present application can effectively avoid the impact of the operation content of the main service process in the cluster node being too much and the load pressure being too large on the cluster business, thereby ensuring the normal operation of the cluster business. In addition, similar prior art has a US patent with publication number US10152382B2, which discloses a method and system for monitoring a virtual machine cluster, comprising: sending a state parameter query instruction to a virtual machine in the virtual machine cluster by a physical machine at a first preset time interval; sending response information to the physical machine in response to receiving the query; the physical machine determines that the virtual machine is faulty in response to the response information exceeding a second preset time, judges whether the faulty machine meets the restart condition, and sends a restart instruction to a second computer running the faulty machine; the faulty machine meets the restart condition through the virtual machine; and restarts the second physical machine, i.e. the faulty virtual machine, according to the restart instruction. The above two patent documents solve the problem of cluster node failure monitoring, but do not further determine the fault location and data damage problem, so the fault monitoring precision is low. SUMMARY

[0003] In order to better solve the above problems, the present application provides a data center cluster monitoring method, which comprises the following steps:

[0004] Step S1: each master node in the cluster receives the synchronization signal sent by the plurality of redundant nodes corresponding to the master node, and sends an acknowledgement signal to each redundant node, wherein the master node and any of the redundant nodes are not in the same subnetwork.

[0005] Step S2: when the communication between the master node and the i-th redundant node is abnormal, sending an abnormal command to a monitoring unit by the master node or the i-th redundant node, the monitoring unit sending a detection command to the i-th redundant node or the master node based on the abnormal command, and determining a fault type according to a detection response signal of the detection command;

[0006] Step S3: when the communication between the master node and the i-th redundant node is normal, determining whether the master node and the i-th redundant node are data-synchronized and whether data needs to be updated based on an update list in the master node, data version numbers of each data type of the i-th redundant node in the master node and the synchronization signal;

[0007] Step S4: taking the master node or the i-th redundant node which is not data-synchronized and needs to be updated as a target node, obtaining a data source node corresponding to a target data type which needs to be updated from the update list, and obtaining a target source node and a target time period based on a sum of access amounts of each data source node and the target node in each time period, and sending target data corresponding to the target data type to the target node by the target source node in the target time period to perform data synchronization.

[0008] As a preferred technical scheme of the present application, the step S1 comprises the following steps:

[0009] Step S11: each master node in the cluster receives the synchronization signal periodically sent by the redundant node corresponding to the master node, wherein the synchronization signal comprises data update records of the redundant node, and the data update records comprise data version and data update time of each data type;

[0010] Step S12: the master node sends a response signal to the i-th redundant node when successfully receiving the synchronization signal of the i-th redundant node.

[0011] As a preferred technical scheme of the present application, the step S2 comprises the following steps:

[0012] Step S21: when the master node cannot receive the synchronization signal of the i-th redundant node for T1 times in succession, a first abnormal command is sent to the monitoring unit, wherein the first abnormal command comprises address information and an abnormal type of the i-th redundant node, and the i-th redundant node is one of the redundant nodes, and T1 is a positive integer greater than or equal to 3;

[0013] Step S22: when the response signal cannot be received by at least T2 of the ith redundant nodes, a second abnormal command is sent to the monitoring unit by any of the ith redundant nodes, wherein the second abnormal command comprises address information of the master node, and T2 is a positive integer greater than or equal to 2;

[0014] Step S23: the monitoring unit receives the first abnormal signal and the second abnormal signal, and sends a first detection signal and a second detection signal to the ith redundant node and the master node based on the first abnormal signal and the second abnormal signal, respectively. When the monitoring unit can receive the first detection response signal, the fault exists in the ith redundant node. When the monitoring unit cannot receive the first detection response signal, the fault exists in the subnetwork where the ith redundant node is located. Similarly, the fault location of the master node is obtained.

[0015] Wherein, the abnormal command comprises the first abnormal command and the second abnormal command, the detection command comprises the first detection command and the second detection command, and the detection response signal comprises the first detection response signal and the second detection response signal.

[0016] As a preferred technical solution of the present application, the step S2 further comprises: when the subnetwork where the master node is located fails or the subnetwork where the ith redundant node is located fails, the monitoring unit sends a query instruction to other redundant nodes except the ith redundant node at a preset time interval, and obtains data update records and load information of all other redundant nodes. An update list is created according to the data update records of all other redundant nodes, and a jth redundant node with the latest version of each data type in the update list is obtained. When there are multiple jth redundant nodes, in the case of failure of the subnetwork where the master node is located, the best-performing kth redundant node among the multiple jth redundant nodes is selected as a temporary master node. In the case of failure of the subnetwork where the ith redundant node is located, the load of the ith redundant node is distributed to multiple jth redundant nodes. When the master node or the ith redundant node fails, the monitoring unit queries the fault data corresponding to the fault location of the master node or the ith redundant node, and assigns an access task of accessing the fault data to N other redundant nodes corresponding to the fault location, which have the latest version of the corresponding data and the least access task amount.

[0017] As a preferred technical solution of the present application, the step S3 comprises:

[0018] Step S31: When the main node and the i-th redundant node communicate normally, the current version of each data type in the update record of the i-th redundant node is obtained through the synchronization signal, and the current version of each data type is compared with the update version of the i-th redundant node stored in the update list in the main node to obtain a first comparison result.

[0019] Step S32: When the first comparison result is that the current version of the i-th redundant node corresponding to the data type is consistent with the update version, the current version of the i-th redundant node corresponding to the data type is also compared with the update version of the main node and other redundant nodes corresponding to the data type to obtain a second comparison result. When the second comparison result is that the current version of the i-th redundant node corresponding to the data type is lower than the update version of the main node or the other redundant nodes, the i-th redundant node corresponding to the data type is out of synchronization and needs to be updated; otherwise, it does not need to be updated.

[0020] Step S33: When the first comparison result is that the current version of the i-th redundant node corresponding to the data type is inconsistent with the update version, when the current version of the data type is higher than the update version, the data corresponding to the data type in the i-th redundant node has been updated, and when the current version of the data type is lower than the update version, the data corresponding to the data type in the i-th redundant node is out of synchronization and needs to be updated.

[0021] Step S34: The current version of the main node and the update version in the update list are also compared, and the method of steps S32 and S33 is repeated to determine whether each data type in the main node is synchronized and updated.

[0022] As a preferred technical solution of the present application, the step S4 comprises:

[0023] When the data of the master node or the i-th redundant node is not synchronized and needs to be updated, the master node or the i-th redundant node is taken as a target node, the target node address, the target data type needing synchronization, and the data source node address having the latest version of the target data type are sent to the monitoring unit by the master node, the monitoring unit calculates the sum of the access amounts of the target node and each data source node in each time period by the historical access amounts of the target node and each data source node in different time periods, obtains a target time period closest to the current time and having a sum of access amounts less than a set threshold, takes the data source node corresponding to the target time period as a target source node, sends the target data from the target source node to the target node in the target time period, and the target node backs up the target data to the storage unit after receiving the target data.

[0024] As a preferred technical scheme of the present application, the step S4 further comprises:

[0025] After the target node completes the data update, the version number of the target data type corresponding to the target node is updated and sent to the master node to replace the update version number of the target data type corresponding to the target node in the update list.

[0026] As a preferred technical scheme of the present application, the values of i, j, and k are positive integers greater than or equal to 1, and the value of N is a positive integer less than or equal to 5.

[0027] The present application also provides a data center cluster monitoring system, which is used to realize the above method, and comprises:

[0028] The master node is used to receive the synchronization signals sent by the plurality of redundant nodes corresponding to the master node, and send a response signal to each redundant node, wherein the master node and any redundant node are not in the same subnetwork.

[0029] The redundant node is used to send a synchronization signal to the corresponding master node, and receive the response signal sent by the corresponding master node.

[0030] The monitoring unit is configured to: when communication between the master node and the i-th redundant node is abnormal, receive an abnormal command sent by the master node or the i-th redundant node, send a detection command to the i-th redundant node or the master node based on the abnormal command, and determine a fault type according to a detection response signal of the detection command; when communication between the master node and the i-th redundant node is normal, determine whether the master node and the i-th redundant node are data-synchronized and whether data needs to be updated based on an update list in the master node, data version numbers of each data type of the master node and the i-th redundant node in the synchronization signal; take the master node or the i-th redundant node which is not data-synchronized and needs to be updated as a target node, obtain a data source node corresponding to a target data type which needs to be updated and having a latest data version from the update list, further obtain a target source node and a target time period based on a sum of access amounts of each data source node and the target node in each time period, and send target data corresponding to the target data type to the target node through the target source node in the target time period to perform data synchronization.

[0031] The application further provides a data center cluster monitoring device, a storage medium of the device storing program instructions, wherein the program instructions control a device where the storage medium is located to perform the method in any one of claims 1-8 when the program instructions are executed.

[0032] Compared with the prior art, the application has at least the following beneficial effects:

[0033] The application detects the communication condition between each redundancy node and the master node by sending a synchronization signal from the redundancy node to the corresponding master node, and sending a reply signal to the redundancy node when the master node receives the synchronization signal sent by the redundancy node, and sends the first abnormal command to the monitoring unit through the master node when the master node fails to receive the synchronization signal of the ith redundancy node for N times in succession, and judges the fault position of the ith redundancy node according to the first detection reply signal, and judges the fault position of the master node by the same method, compares the current version of each data type in the ith redundancy node with the updated version stored in the update list when the master node and the ith redundancy node can normally communicate, and obtains the first comparison result to judge whether the data is updated, and compares the current version of the data type corresponding to the ith redundancy node with the updated version of the data type corresponding to other nodes in the update list to judge whether the data of the data type corresponding to the ith redundancy node is synchronized and needs to be updated, and judges whether the data of each data type in the master node is synchronized and needs to be updated, and obtains the target time period closest to the current time as the data synchronization time period when the target node data is not synchronized and needs to be updated, and takes the data source node corresponding to the target time period as the target source node, sends the target data to the target node through the target source node in the target time period, and backs up the target data after the target node receives the target data, completes data synchronization, and through the mutual cooperation of the technical solutions, not only can the fault and the corresponding fault position be accurately detected, but also when the data is damaged and the version is restored to the previous version or the data is updated and upgraded, the data synchronization can be updated in time. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A data center cluster monitoring method flow chart is provided in the application.

[0035] Figure 2 A data center cluster monitoring system structure diagram is provided in the application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0037] The application provides a data center cluster monitoring method, as shown in Figure 1 The method comprises the following steps:

[0038] Step S1: each master node in the cluster receives the synchronization signal sent by the plurality of redundant nodes corresponding to the master node, and sends a response signal to the corresponding redundant node, wherein the master node and any of the corresponding redundant node are not in the same subnetwork;

[0039] Specifically, the cluster includes a plurality of node groups, the access data stored by the nodes of different node groups are different, the access data stored by the nodes in the same node group are the same, the nodes in the same node group are divided into a master node and a plurality of redundant nodes, each of the above-mentioned i-th redundant node communicates with the master node by periodically sending the above-mentioned synchronization signal, when the master node receives the above-mentioned synchronization signal sent by the above-mentioned i-th redundant node, a response signal is sent to the above-mentioned redundant node, since the above-mentioned synchronization signal includes the above-mentioned data update record corresponding to the node, through the above-mentioned technical solution, not only can the data update of the above-mentioned redundant node be checked by receiving the above-mentioned synchronization signal, but also the communication condition between each of the above-mentioned redundant node and the above-mentioned master node can be detected, thereby laying a foundation for obtaining the fault information of the above-mentioned master node and each of the above-mentioned redundant node.

[0040] Step S2: when the communication between the master node and the i-th redundant node is abnormal, an abnormal command is sent to the monitoring unit by the master node or the i-th redundant node, the monitoring unit sends a detection command to the i-th redundant node or the master node based on the abnormal command, and determines the fault type according to the detection response signal of the detection command;

[0041] Specifically, when the master node cannot receive the above-mentioned synchronization signal of the above-mentioned i-th redundant node for N times in succession, it is considered that the i-th redundant node or the subnetwork where the i-th redundant node is located has a fault, therefore, the first abnormal command is sent to the monitoring unit by the master node, the address information of the i-th redundant node is sent to the monitoring unit by the first abnormal command, a first detection signal is sent according to the address information of the i-th redundant node, and the fault position of the i-th redundant node is judged by the first detection response signal sent by the i-th redundant node, when the monitoring unit can receive the first detection response signal, it is indicated that the subnetwork where the i-th redundant node is located has no fault, and it is possible that the i-th redundant node has a fault, when the monitoring unit cannot receive the first detection response signal, it is possible that the subnetwork where the i-th redundant node is located has a fault, the fault position of the master node is determined based on the second abnormal signal sent by the monitoring unit and the second detection response signal by the same method, through the above-mentioned technical solution, the fault position of the master node and the corresponding i-th redundant node can be accurately monitored and determined.

[0042] Step S3: when the master node and the ith redundant node can communicate normally, judging whether the master node and the ith redundant node are data-synchronized and whether the data need to be updated based on the update list in the master node, the data version number of each data type in the master node and the data version number of each data type of the ith redundant node in the synchronization signal.

[0043] Specifically, when the master node and the ith redundant node can communicate normally, the master node compares the current version of each data type in the update record of the synchronization signal sent by the ith redundant node with the update version stored in the update list, and obtains the first comparison result to judge whether the data is updated. However, it cannot be explained that the data of the ith redundant node corresponding to the data type is synchronized with the data of other nodes corresponding to the data type. Therefore, the current version of the ith redundant node corresponding to the data type is compared with the update version of other nodes corresponding to the data type in the update list, a plurality of second comparison results are obtained, and whether the data of the ith redundant node corresponding to the data type is synchronized and whether the data need to be updated is judged through the plurality of second comparison results. The data of each data type in the master node is judged whether it is synchronized and whether it need to be updated through the same method. Through the technical solution, when the master node and the ith redundant node communicate, whether the data of the master node and the ith redundant node is synchronized and whether the data need to be updated can be accurately judged, and the foundation for data synchronization is laid.

[0044] Step S4: taking the master node or the ith redundant node whose data is not synchronized and need to be updated as a target node, obtaining the data source node corresponding to the target data type which need to be updated from the update list, and obtaining the target source node and the target time period based on the sum of the access amount of each data source node and the target node in each time period, and sending the target data corresponding to the target data type to the target node through the target source node in the target time period to synchronize the data.

[0045] Specifically, when the target node data is not synchronized and needs to be updated, the address of the target node, the type of data that needs to be synchronized, and the addresses of the multiple data source nodes that have the latest version of the corresponding data of the target data type obtained through the update list are sent to the monitoring unit by the master node, the monitoring unit collects and calculates the sum of the access amounts of the target node and each data source node in each time period according to the historical access amounts of the target node and each data source node in different time periods. Since the corresponding target data of the target data type needs to occupy a certain communication resource when it is sent from the data source node to the target node, and the more timely the update is, the better, the target time period closest to the current time and having an access amount sum less than the set threshold is taken as the data synchronization time period, and the data source node corresponding to the target time period is taken as the target source node. The target data is sent to the target node by the target source node in the target time period, and the target data is backed up after being received by the target node, completing data synchronization. Through the technical solution, when the target node data is not synchronized and needs to be updated, it can be timely and not affect normal access.

[0046] Further, the step S1 comprises the following steps:

[0047] Step S11: Each master node in the cluster receives the synchronization signal periodically sent by the redundant node corresponding to the master node, wherein the synchronization signal comprises the data update record of the redundant node, and the data update record comprises the data version and data update time of each data type.

[0048] Step S12: When the master node successfully receives the synchronization signal of each redundant node, it sends an acknowledgement signal to the corresponding redundant node.

[0049] Specifically, each i-th redundant node communicates with the master node by periodically sending the synchronization signal. When the master node receives the synchronization signal sent by the i-th redundant node, it sends an acknowledgement signal to the redundant node. Since the synchronization signal comprises the data update record of the corresponding node, through the technical solution, not only can the data update of the corresponding redundant node be checked by receiving the synchronization signal and the acknowledgement, but also the communication condition between each redundant node and the master node can be detected, thereby laying a foundation for obtaining the fault information of the master node and each redundant node.

[0050] Further, the step S2 comprises the following steps:

[0051] Step S21: when the master node fails to receive the synchronization signal of the ith redundant node for T1 times, a first abnormal command is sent to the monitoring unit, wherein the first abnormal command comprises address information of the ith redundant node and an abnormal type, and T1 is a positive integer greater than or equal to 3;

[0052] Step S22: when at least T2 of the ith redundant nodes fail to receive the response signal, a second abnormal command is sent to the monitoring unit by any of the ith redundant nodes, wherein the second abnormal command comprises address information of the master node, and T2 is a positive integer greater than or equal to 2;

[0053] Step S23: the monitoring unit receives the first abnormal signal and the second abnormal signal, and sends a first detection signal and a second detection signal to the ith redundant node and the master node based on the first abnormal signal and the second abnormal signal, respectively. When the monitoring unit can receive the first detection response signal, a fault exists in the ith redundant node. When the monitoring unit cannot receive the first detection response signal, a fault exists in the subnetwork where the ith redundant node is located. Similarly, the fault location of the master node is obtained.

[0054] Wherein, the abnormal command comprises the first abnormal command and the second abnormal command, the detection command comprises the first detection command and the second detection command, and the detection response signal comprises the first detection response signal and the second detection response signal.

[0055] Specifically, when the master node fails to receive the synchronization signal of the ith redundant node for T1 times, it is considered that the ith redundant node or the subnetwork where the ith redundant node is located has a fault. Therefore, the first abnormal command is sent to the monitoring unit by the master node, the address information of the ith redundant node is sent to the monitoring unit by the first abnormal command, the first detection signal is sent according to the address information of the ith redundant node, and the fault location of the ith redundant node is determined by the first detection response signal sent by the ith redundant node. When the monitoring unit can receive the first detection response signal, it indicates that the subnetwork where the ith redundant node is located has no fault, and the ith redundant node may have a fault. When the monitoring unit cannot receive the first detection response signal, the subnetwork where the ith redundant node is located may have a fault. The fault location of the master node is determined based on the second abnormal signal sent by the monitoring unit and the second detection response signal. By the technical solution, the fault locations of the master node and the corresponding ith redundant node can be accurately monitored and determined.

[0056] Further, the step S2 further comprises:

[0057] When the sub-network where the master node is located fails or the sub-network where the ith redundant node is located fails, the monitoring unit sends a query instruction to each ith redundant node at a preset time interval, and acquires the data update record and the load information of all ith redundant nodes, creates an update list according to the data update record of all ith redundant nodes, and acquires the jth redundant node corresponding to each data update event in the update list, when there are multiple jth redundant nodes, when the sub-network where the master node is located fails, the kth redundant node with the smallest total access amount among the multiple jth redundant nodes is taken as a temporary master node, when the sub-network where the ith redundant node is located fails, the load of the ith redundant node is distributed to the multiple jth redundant nodes; when the master node or the ith redundant node fails, the monitoring unit queries the failure data corresponding to the failure position of the master node or the ith redundant node, and distributes an access task for accessing the failure data to N other mth redundant nodes corresponding to the latest corresponding data at the failure position and the least access task.

[0058] Specifically, when the sub-network where the master node or the i-th redundant node is located fails, the master node and the i-th redundant node lose the node function, and the access task of the master node or the i-th redundant node needs to be allocated to other redundant nodes. Therefore, the monitoring unit sends the query instruction to other redundant nodes except the i-th redundant node at a preset time interval, obtains the update record of the other redundant nodes through the query instruction, creates an update list, obtains the latest update record of each data type based on the update list, and filters out the j-th redundant node of each data type which is the latest version based on the latest update record of each data type. When there are multiple j-th redundant nodes, the k-th redundant node with the best performance among the multiple j-th redundant nodes is selected as a temporary master node in the case of failure of the sub-network where the master node is located, and the load of the i-th redundant node is allocated to the multiple j-th redundant nodes according to the load of each j-th redundant node in the case of failure of the sub-network where the i-th redundant node is located. The greater the load, the greater the load of the i-th redundant node allocated, and vice versa. When the master node or the i-th redundant node fails, the master node and the i-th redundant node are regarded as a failure node. Since the failure node can communicate with the monitoring node, it is possible that the synchronization signal and the corresponding response signal cannot be normally received due to data update or partial data damage. The monitoring unit queries the update record of the failure node and obtains the failure position, i.e., the data version is restored to the data position of the previous old version due to data damage or data anomaly, and the access task of the failure position is allocated to N m-th redundant nodes corresponding to the latest version of the data at the failure position and having the least access task. Through the technical solution, the temporary master node can be dynamically obtained when the master node and the i-th redundant node fail or when the corresponding sub-network fails, so as to ensure the normal execution of the access task of the master node and the i-th redundant node.

[0059] Further, the step S3 comprises:

[0060] Step S31: When the master node and the i-th redundant node communicate normally, the current version of each data type in the update record of the i-th redundant node is obtained through the synchronization signal, and the current version of each data type is compared with the update version of the i-th redundant node stored in the update list in the master node to obtain a first comparison result.

[0061] Step S32: when the first comparison result is that the current version of the data type corresponding to the i-th redundant node is consistent with the updated version corresponding to the master node and other redundant nodes, the current version of the data type corresponding to the i-th redundant node is further compared with the updated version of the data type corresponding to the master node and other redundant nodes, and a plurality of second comparison results are obtained. When at least one of the plurality of second comparison results is that the current version of the data type corresponding to the i-th redundant node is lower than the updated version of the master node or other redundant nodes, the data type corresponding to the i-th redundant node is not synchronized and needs to be updated; otherwise, it does not need to be updated.

[0062] Step S33: when the first comparison result is that the current version of the data type corresponding to the i-th redundant node is inconsistent with the updated version, when the current version of the data type is higher than the updated version, the data corresponding to the data type in the i-th redundant node has been updated, and when the current version of the data type is lower than the updated version, the data corresponding to the data type in the i-th redundant node is not synchronized and needs to be updated.

[0063] Step S34: whether each data type in the master node is synchronized and updated is further determined by comparing the current version of the master node with the updated version in the update list, and repeating the method of steps S32 and S33.

[0064] Specifically, when the main node and the i-th redundant node can communicate normally, i.e. the main node can receive the synchronization signal sent by the i-th redundant node and the i-th redundant node can receive the response signal returned by the main node, including the case that the sub-node failure is replaced by a temporary main node, the main node also records the current version of each data type in the update record of the synchronization signal sent by the i-th redundant node, compares the current version of each data type in the i-th redundant node with the update version stored in the update list, and obtains the first comparison result. When the comparison result is consistent, it means that the data of the data type in the i-th redundant node has not been updated, but it cannot be concluded that the data of the data type in the i-th redundant node is synchronized with the data of the data type in other nodes, i.e. whether the data version is consistent. Therefore, the current version of the data type in the i-th redundant node is compared with the update version of the data type in other nodes in the update list, the other nodes include the main node and other redundant nodes except the i-th redundant node, a plurality of second comparison results are obtained, and when the current version of the data type in the i-th redundant node is lower than the update version of the data type in at least one of the other nodes, i.e. the data of the data type in the i-th redundant node is not the latest version, it means that the data of the data type in the i-th redundant node is not synchronized and needs to be updated. Conversely, when the current version of the data type in the i-th redundant node is higher than or equal to the update version of the data type in at least one of the other nodes, it means that the data of the data type in the i-th redundant node is synchronized with the latest version and does not need to be updated. The same method is used to determine whether the data of each data type in the main node is synchronized and needs to be updated. Through the technical solution, when the main node and the i-th redundant node communicate, it can be determined whether the data of the main node and the i-th redundant node is synchronized and needs to be updated, and lays a foundation for data synchronization.

[0065] Further, the step S4 comprises:

[0066] When the data of the master node or the i-th redundant node is not synchronized and needs to be updated, the master node sends the address of the target node, the target data type that needs to be synchronized, and the address of the data source node with the latest version of the target data type to the monitoring unit, the monitoring unit calculates the sum of the access amounts of the target node and each data source node in each time period according to the historical access amounts of the target node and each data source node in different time periods, obtains a target time period closest to the current time and having a sum of access amounts less than a set threshold, takes the data source node corresponding to the target time period as a target source node, sends the target data from the target source node to the target node in the target time period, and the target node backs up the target data to the storage unit after receiving the target data.

[0067] Specifically, when the data of the target node is not synchronized and needs to be updated, the master node sends the address of the target node, the data type that needs to be synchronized, and the address of the data source node with the latest version of the corresponding data of the target data type obtained from the update list to the monitoring unit, the monitoring unit collects and calculates the sum of the access amounts of the target node and each data source node in each time period according to the historical access amounts of the target node and each data source node in different time periods, takes a target time period closest to the current time and having a sum of access amounts less than a set threshold as a data synchronization time period, takes the data source node corresponding to the target time period as the target source node, sends the target data from the target source node to the target node in the target time period, and backs up the target data after the target node receives the target data, thereby completing data synchronization. According to the technical solution, when the data of the target node is not synchronized and needs to be updated, the data can be synchronized in time and without affecting normal access.

[0068] Further, the step S4 further includes:

[0069] After the target node completes data updating, the data version number of the target data type corresponding to the target node is updated, and the updated data version number is sent to the master node and replaces the update version number of the target data type corresponding to the target node in the update list.

[0070] Specifically, after the target node completes the data update, the data version number of the target node corresponding to the target data type is updated, and the updated data version number is replaced with the update version number in the update list. Through the technical scheme, data synchronization detection is facilitated in the subsequent process.

[0071] Further, the values of i, j, and k are positive integers greater than or equal to 1, and the value of N is a positive integer less than or equal to 5.

[0072] The application also provides a data center cluster monitoring system, which is used to implement the above method. Figure 2 As shown in the figure, the system comprises:

[0073] The master node is configured to receive synchronization signals sent by a plurality of redundant nodes corresponding to the master node, and send a response signal to each of the redundant nodes, wherein the master node and any of the redundant nodes are not in the same subnetwork.

[0074] The redundant node is configured to send a synchronization signal to the corresponding master node and receive the response signal sent by the corresponding master node.

[0075] The monitoring unit is configured to: when the communication between the master node and the i-th redundant node is abnormal, receive an abnormal command sent by the master node or the i-th redundant node, send a detection command to the i-th redundant node or the master node based on the abnormal command, and determine the fault type according to the detection response signal of the detection command; when the communication between the master node and the i-th redundant node is normal, determine whether the master node and the i-th redundant node are data synchronized and whether the data needs to be updated based on the update list in the master node, the data version number of each data type of the i-th redundant node in the master node and the synchronization signal; take the master node or the i-th redundant node as a target node when the data is not synchronized and needs to be updated, obtain a data source node corresponding to the latest data version of a target data type that needs to be updated from the update list, further obtain a target source node and a target time period based on the sum of the access amounts of each data source node and the target node in each time period, and send target data corresponding to the target data type to the target node through the target source node in the target time period for data synchronization.

[0076] The application also provides a data center cluster monitoring device, wherein the storage medium of the device stores program instructions, and the program instructions control the device to execute the above method when running.

[0077] In summary, the application detects the communication condition between each of the redundant nodes and the master node by sending a synchronization signal from the redundant node to the corresponding master node, and sending a reply signal to the redundant node when the master node receives the synchronization signal sent by the redundant node, and sends the first abnormal command to the monitoring unit through the master node when the master node fails to receive the synchronization signal of the ith redundant node for N consecutive times, and judges the fault position of the ith redundant node according to the first detection reply signal, and judges the fault position of the master node by the same method, and compares the current version of each data type in the ith redundant node with the updated version stored in the update list when the master node and the ith redundant node can normally communicate, and obtains the first comparison result to judge whether the data is updated, and compares the current version of the ith redundant node corresponding to the data type with the updated version of the other nodes corresponding to the data type in the update list, judges whether the data of the ith redundant node corresponding to the data type is synchronized and needs to be updated, and judges whether the data of each data type in the master node is synchronized and needs to be updated, and obtains the target time period closest to the current time as the data synchronization time period when the target node data is not synchronized and needs to be updated, and takes the data source node corresponding to the target time period as the target source node, and sends the target data to the target node through the target source node in the target time period, and backs up after the target node receives the target data, completes data synchronization, and through the mutual cooperation of the above technical solutions, not only can the fault and the corresponding fault position be accurately detected, but also when the data is damaged and the version is restored to the previous version or the data is updated and upgraded, the data synchronization can be updated in time.

[0078] The technical features of the above embodiments can be combined in any way, and to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0079] The above embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as the limitation of the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

[0080] The above merely describes the preferred embodiments of the present application, and is not used to limit the present application, and any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of monitoring a cluster of data centers, the method comprising: The method comprises the following steps: Step S1: each master node in the cluster receives the synchronization signals sent by the multiple redundant nodes corresponding to the master node, and sends a response signal to each of the redundant nodes, the master node and any of the corresponding redundant nodes are not in the same subnetwork; Step S2: when the communication between the master node and the ith redundant node is abnormal, the master node or the ith redundant node sends an exception command to a monitoring unit, the monitoring unit sends a detection command to the ith redundant node or the master node based on the exception command, and determines the fault type according to the detection response signal of the detection command, wherein i is a positive integer greater than or equal to 1; Step S3: when the communication between the master node and the ith redundant node is normal, it is judged whether the master node and the ith redundant node are data synchronized and whether the data need to be updated based on the update list in the master node, the data version number of each data type of the ith redundant node in the master node and the synchronization signal; Step S4: taking the master node or the ith redundant node as a target node when the data is not synchronized and needs to be updated, obtaining the data source node with the latest data version corresponding to the target data type that needs to be updated from the update list, and obtaining the target source node and the target time period based on the sum of the access amounts of each data source node and the target node in each time period, and sending the target data corresponding to the target data type to the target node through the target source node in the target time period to synchronize the data; Wherein, step S4, comprising: taking the master node or the ith redundant node as a target node when the data is not synchronized and needs to be updated, sending the target node address, the target data type that needs to be synchronized and the multiple data source node addresses with the latest version of the target data type to the monitoring unit through the master node, calculating the access amount of the target node and each data source node in each time period through the historical access amounts of the target node and each data source node in different time periods, obtaining the target time period closest to the current time and with the access amount sum less than a set threshold, and taking the data source node corresponding to the target time period as a target source node, sending the target data from the target source node to the target node in the target time, and backing up the target data to a storage unit after the target node receives the target data.

2. The method of claim 1, wherein, The step S1 comprises the following steps: Step S11: each master node in the cluster receives the synchronization signals periodically sent by the redundant nodes corresponding to the master node, wherein the synchronization signals comprise data update records of the redundant nodes, the data update records comprise data version and data update time of each data type; Step S12: the master node sends a response signal to the corresponding ith redundant node when successfully receiving the synchronization signal of the ith redundant node.

3. The method of claim 1, wherein, The step S2 comprises the following steps: Step S21: when the master node fails to receive the synchronization signal from the ith redundant node for continuous T1 times, a first abnormal command is sent to the monitoring unit, wherein the first abnormal command comprises address information of the ith redundant node and an abnormal type, and the ith redundant node is one of the redundant nodes, and T1 is a positive integer greater than or equal to 3; Step S22: when at least T2 of the ith redundant nodes fail to receive the response signal, a second abnormal command is sent to the monitoring unit by any of the ith redundant nodes, wherein the second abnormal command comprises address information of the master node, and T2 is a positive integer greater than or equal to 2; Step S23: the monitoring unit receives the first abnormal signal and the second abnormal signal, and sends a first detection signal and a second detection signal to the ith redundant node and the master node based on the first abnormal signal and the second abnormal signal, respectively. When the monitoring unit can receive the first detection response signal, the fault exists in the ith redundant node. When the monitoring unit cannot receive the first detection response signal, the fault exists in the subnetwork where the ith redundant node is located. Similarly, the fault location of the master node is obtained. Wherein, the abnormal command comprises the first abnormal command and the second abnormal command, the detection command comprises the first detection command and the second detection command, and the detection response signal comprises the first detection response signal and the second detection response signal.

4. The method of claim 1, wherein, The step S2 further comprises: when the subnetwork where the master node is located fails or the subnetwork where the ith redundant node is located fails, the monitoring unit sends a query instruction to other redundant nodes except the ith redundant node at a preset time interval, and obtains data update records and load information of all other redundant nodes. An update list is created according to the data update records of all other redundant nodes, and a jth redundant node with each data type being the latest version in the update list is obtained. When the jth redundant node is multiple, in the case that the subnetwork where the master node is located fails, the best-performing kth redundant node among the multiple jth redundant nodes is selected as a temporary master node. In the case that the subnetwork where the ith redundant node is located fails, the load of the ith redundant node is distributed to the multiple jth redundant nodes. When the master node or the ith redundant node fails, the monitoring unit queries the fault data corresponding to the fault location of the master node or the ith redundant node, and assigns an access task of accessing the fault data to N other redundant nodes corresponding to the fault location, which have the latest version of corresponding data and the least access task amount.

5. The method of claim 1, wherein, The step S3 comprises: Step S31: When the main node and the i-th redundant node communicate normally, the current version of each data type in the update record of the i-th redundant node is obtained through the synchronization signal, and the current version of each data type is compared with the update version of the i-th redundant node stored in the update list of the main node to obtain a first comparison result. Step S32: When the first comparison result is that the current version of the i-th redundant node corresponding to the data type is consistent with the update version, the current version of the i-th redundant node corresponding to the data type is further compared with the update version of the main node and other redundant nodes corresponding to the data type to obtain a second comparison result. When the second comparison result is that the current version of the i-th redundant node corresponding to the data type is lower than the update version of the main node or the other redundant nodes, the i-th redundant node corresponding to the data type is out of synchronization and needs to be updated; otherwise, it does not need to be updated. Step S33: When the first comparison result is that the current version of the i-th redundant node corresponding to the data type is inconsistent with the update version, when the current version of the data type is higher than the update version, the data corresponding to the data type in the i-th redundant node has been updated, and when the current version of the data type is lower than the update version, the data corresponding to the data type in the i-th redundant node is out of synchronization and needs to be updated. Step S34: The current version of the main node and the update version in the update list are further compared, and the method of steps S32 and S33 is repeated to determine whether each data type in the main node is synchronized and updated.

6. The method of claim 1, wherein, The step S4 further comprises: After the target node completes data update, the version number of the target node corresponding to the target data type is updated and sent to the main node to replace the update version number of the target node corresponding to the target data type in the update list.

7. The method of claim 4, wherein, The values of j and k are positive integers greater than or equal to 1, and the value of N is a positive integer less than or equal to 5.

8. A data center cluster monitoring system, the system being configured to implement the method of any one of claims 1-7, characterized in that, The system comprises: The main node is configured to receive synchronization signals sent by a plurality of redundant nodes corresponding to the main node, and send a response signal to each of the redundant nodes, wherein the main node and any of the redundant nodes are not in the same subnetwork. The redundant node is configured to send a synchronization signal to the corresponding main node and receive the response signal sent by the corresponding main node. The monitoring unit is configured to: receive an abnormal command sent by the master node or the i-th redundant node when communication between the master node and the i-th redundant node is abnormal, send a detection command to the i-th redundant node or the master node based on the abnormal command, and determine a fault type according to a detection response signal of the detection command; when communication between the master node and the i-th redundant node is normal, determine whether the master node and the i-th redundant node are data-synchronized and whether data needs to be updated based on an update list in the master node, data version numbers of each data type of the master node and the i-th redundant node in the synchronization signal; take the master node or the i-th redundant node which is not data-synchronized and needs to be updated as a target node, obtain a data source node corresponding to a target data type which needs to be updated and having a latest data version from the update list, further obtain a target source node and a target time period based on a sum of access amounts of each data source node and the target node in each time period, and send target data corresponding to the target data type to the target node through the target source node in the target time period to perform data synchronization.

9. A data center cluster monitoring apparatus, characterized by, The storage medium of the device stores program instructions, wherein the program instructions control the device where the storage medium is located to execute the method of any one of claims 1-7 when the program instructions are executed.

Images