Time delay monitoring method and device of cluster system, electronic equipment and storage medium

Abstract

The invention discloses a time delay monitoring method and device of a cluster system, electronic equipment and a storage medium, the cluster system comprises a management service platform and a plurality of cluster nodes, each cluster node is in communication connection with the management service platform, the method is applied to the management service platform, and the method comprises the following steps: for each cluster node in the cluster system, determining the time delay of each cluster node; performing time delay monitoring on the cluster node according to the time delay monitoring instruction to obtain time delay information of the cluster node; when the time delay information of the cluster node meets a first intervention condition, controlling a data reconstruction service of a target disk in the cluster node to stop running; performing time delay monitoring on the cluster nodes in a preset observation time period to obtain new time delay information of the cluster nodes; and when the new time delay information of the cluster node meets a second intervention condition, controlling data reconstruction services of all disks in the cluster node to stop running. By applying the technical scheme, more real-time and efficient time delay monitoring can be realized in the cluster system, and the stability of the cluster system is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of cluster system performance monitoring, and in particular to a cluster system delay monitoring method, and also to a cluster system delay monitoring device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the continuous advancement of science and technology, the continuous improvement of the degree of electronicization and informatization, the data generated by society, enterprises, groups and even individuals is growing exponentially, and the requirements for data processing capabilities are also constantly increasing. Building large-scale clusters for data collection, storage, calculation and distribution has become an industry consensus.

[0003] However, the larger the cluster size, the more complex it is to manage and maintain. When the cluster system is upgraded online, there is a trade-off between version upgrade efficiency and business stability. In order to solve the problem of increased latency caused by unexpected situations during the online upgrade of the cluster, traditional technologies mainly configure latency monitoring. When the latency exceeds the configured threshold, an alarm is generated. The alarm receiver then contacts the technician for analysis. Finally, an optimization or processing plan is formulated based on the analysis results, and the solution is imported for on-site processing through a patch. However, this implementation method has the following limitations: one is that the timeliness of solving the problem is insufficient. From triggering the alarm to the technician analyzing and providing the patch, the information transmission, cause analysis, and patch production in the middle take too long, which cannot meet the requirements of projects with high business continuity requirements; the other is that the business impact has already occurred. Even if there are good solutions and patches to optimize the latency problem, they can only be imported in the next upgrade.

[0004] Therefore, how to achieve more real-time and efficient delay monitoring in the cluster system and ensure the stability of the cluster system is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] The purpose of the present application is to provide a delay monitoring method for a cluster system, which can realize more real-time and efficient delay monitoring in the cluster system and ensure the stability of the cluster system; another purpose of the present application is to provide a delay monitoring device, electronic device, computer-readable storage medium and computer program product for a cluster system, all of which have the above-mentioned beneficial effects.

[0006] In a first aspect, the present application provides a method for monitoring latency of a cluster system, wherein the cluster system includes a management service platform and a plurality of cluster nodes, each of the cluster nodes is communicatively connected to the management service platform, and the method is applied to the management service platform, including:

[0007] For each cluster node in the cluster system, performing latency monitoring on the cluster node according to the latency monitoring instruction to obtain latency information of the cluster node;

[0008] When the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running;

[0009] During a preset observation period, monitoring the latency of the cluster nodes to obtain new latency information of the cluster nodes;

[0010] When the new delay information of the cluster node meets the second intervention condition, the data reconstruction service of all disks in the cluster node is controlled to stop running.

[0011] Optionally, when the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running includes:

[0012] When the delay information of the cluster node reaches a preset delay threshold, delay monitoring is performed on the cluster node according to a preset monitoring period to obtain delay growth information within a preset number of the preset monitoring periods;

[0013] If all the delay growth information within the preset monitoring period reaches the preset delay growth threshold, controlling the data reconstruction service of the target disk in the cluster node to stop running;

[0014] If the delay growth information in all the preset monitoring periods does not reach the preset delay growth threshold, then the total delay growth information in all the preset monitoring periods is counted;

[0015] If the total delay growth information within all the preset monitoring periods exceeds the preset total delay growth threshold, controlling the data reconstruction service of the target disk in the cluster node to stop running;

[0016] If the total delay growth information within all the preset monitoring periods does not exceed the preset total delay growth threshold, return to the step of performing delay monitoring on the cluster nodes according to the delay monitoring instruction to obtain the delay information of the cluster nodes.

[0017] Optionally, the cluster system is specifically a storage cluster system, and the cluster node is specifically a storage node;

[0018] Controlling the data reconstruction service of the target disk in the cluster node to stop running includes:

[0019] In the cluster node, determining a disk used to provide storage services as the target disk;

[0020] Control the data reconstruction services of all the target disks in the cluster nodes to stop running.

[0021] Optionally, when the new latency information of the cluster node meets the second intervention condition, controlling the data reconstruction service of all disks in the cluster node to stop running includes:

[0022] Determine the delay state of the cluster node within the preset observation time period according to the new delay information of the cluster node; wherein the delay state includes an increasing state and a non-increasing state;

[0023] When the delay state of the cluster node is the increasing state, determining that new delay information of the cluster node satisfies the second intervention condition;

[0024] Control the data reconstruction service of all disks in the cluster node to stop running.

[0025] Optionally, when the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running includes:

[0026] When the latency information of the cluster node meets the first intervention condition, determining the target storage pool to which the cluster node belongs;

[0027] If the target storage pool is set with a delay intervention mark, then stopping the delay monitoring of the cluster node;

[0028] If the target storage pool is not set with the delay intervention mark, the data reconstruction service of the target disk in the cluster node is controlled to stop running, and the delay intervention mark is added to the target storage pool.

[0029] Optionally, when the new latency information of the cluster node meets the second intervention condition, after controlling the data reconstruction service of all disks in the cluster node to stop running, the method further includes:

[0030] Performing delay monitoring on the cluster nodes to obtain post-intervention delay information on the cluster nodes;

[0031] If the post-intervention delay information of the cluster node meets the preset recovery condition, the data reconstruction service of all disks in the cluster node is controlled to resume operation, and the delay intervention mark of the target storage pool is cancelled;

[0032] If the post-intervention delay information of the cluster node does not meet the preset recovery condition, an alarm prompt is output.

[0033] Optionally, the cluster system delay monitoring method further includes:

[0034] Determine the working state of the cluster system; wherein the working state includes an online upgrade state and a service processing state;

[0035] When the working state is the online upgrade state, the delay monitoring instruction is triggered.

[0036] In a second aspect, the present application further discloses a time delay monitoring device for a cluster system, wherein the cluster system includes a management service platform and a plurality of cluster nodes, each of the cluster nodes is communicatively connected to the management service platform, and the device is applied to the management service platform, including:

[0037] A first monitoring module, configured to perform latency monitoring on each cluster node in the cluster system according to a latency monitoring instruction to obtain latency information of the cluster node;

[0038] A first control module, configured to control the data reconstruction service of the target disk in the cluster node to stop running when the latency information of the cluster node meets a first intervention condition;

[0039] A second monitoring module is used to monitor the latency of the cluster node within a preset observation time period and obtain new latency information of the cluster node;

[0040] The second control module is used to control the data reconstruction services of all disks in the cluster node to stop running when the new delay information of the cluster node meets the second intervention condition.

[0041] In a third aspect, the present application further discloses an electronic device, comprising:

[0042] Memory for storing computer programs;

[0043] A processor is used to implement the steps of any one of the above-mentioned methods for monitoring the time delay of a cluster system when executing the computer program.

[0044] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for monitoring the delay of a cluster system are implemented.

[0045] In a fifth aspect, the present invention further discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned methods for monitoring the delay of a cluster system.

[0046] The present application provides a latency monitoring method for a cluster system, wherein the cluster system includes a management service platform and multiple cluster nodes, each of the cluster nodes is communicatively connected to the management service platform, and the method is applied to the management service platform, including: for each cluster node in the cluster system, performing latency monitoring on the cluster node according to a latency monitoring instruction to obtain latency information of the cluster node; when the latency information of the cluster node meets a first intervention condition, controlling a data reconstruction service of a target disk in the cluster node to stop running; within a preset observation time period, performing latency monitoring on the cluster node to obtain new latency information of the cluster node; when the new latency information of the cluster node meets a second intervention condition, controlling the data reconstruction services of all disks in the cluster node to stop running.

[0047] By applying the technical solution provided by this application, each cluster node in the cluster system can be monitored for latency, and when the latency information of the cluster node meets different intervention conditions, different intervention means are triggered, namely, disk-level intervention means (controlling the data reconstruction service of the target disk in the cluster node to stop running) and node-level intervention means (controlling the data reconstruction service of all disks in the cluster node to stop running), realizing real-time latency monitoring and real-time over-latency intervention, which can effectively avoid the business impact caused by over-latency problems. It can be seen that this technical solution realizes more real-time and efficient latency monitoring in the cluster system, further ensuring the stability of the cluster system.

[0048] In one embodiment of the present application, in view of the situation where a storage cluster system is divided into multiple storage pools, in order to further ensure the stability of system services within the cluster system, it is set that only a small number (such as one) of cluster nodes in a storage pool are subject to ultra-delay intervention, thereby effectively avoiding too many cluster nodes being intervened at the same time, resulting in increased complexity and adverse effects on system services, thereby further ensuring the stability of the cluster system.

[0049] The cluster system delay monitoring device, electronic device, computer-readable storage medium and computer program product provided in this application also have the above-mentioned technical effects, which will not be described in detail in this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0051] Figure 1 A schematic diagram of the structure of a cluster system provided in this application;

[0052] Figure 2 A schematic diagram of a process flow of a cluster system delay monitoring method provided in this application;

[0053] Figure 3 A flowchart of another method for monitoring the time delay of a cluster system provided in the present application;

[0054] Figure 4 A flowchart of a preliminary delay monitoring method provided in this application;

[0055] Figure 5 A flowchart of a time-delay intervention method provided in this application;

[0056] Figure 6 A schematic diagram of the structure of a delay monitoring device for a cluster system provided in this application;

[0057] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0058] The core of the present application is to provide a delay monitoring method for a cluster system, which can realize more real-time and efficient delay monitoring in the cluster system and ensure the stability of the cluster system; another core of the present application is to provide a delay monitoring device, electronic device, computer-readable storage medium and computer program product for a cluster system, all of which have the above-mentioned beneficial effects.

[0059] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0060] An embodiment of the present application provides a method for monitoring latency of a cluster system.

[0061] First, please refer to Figure 1 , Figure 1 This is a structural diagram of a cluster system provided in the present application. The cluster system includes a management service platform and multiple cluster nodes, and each cluster node can be communicated with the management service platform. The delay monitoring method of the cluster system provided in an embodiment of the present application is implemented based on the cluster system.

[0062] For further information, please refer to Figure 2 , Figure 2 A schematic diagram of a time delay monitoring method for a cluster system provided in the present application. The time delay monitoring method for the cluster system is applied to Figure 1 The management service platform shown may include the following S101 to S104.

[0063] S101: For each cluster node in the cluster system, perform latency monitoring on the cluster node according to a latency monitoring instruction to obtain latency information of the cluster node.

[0064] This step aims to implement latency monitoring for each cluster node in the cluster system. When a latency monitoring instruction is received, latency monitoring can be performed on each cluster node in the cluster system so as to obtain latency information of each cluster node in real time, thereby determining whether a corresponding cluster node has an over-latency problem based on the latency information, so as to perform a corresponding over-latency intervention operation on the cluster node when an over-latency problem is determined.

[0065] In one embodiment of the present application, the delay monitoring method of the cluster system may further include:

[0066] Determine the working status of the cluster system; wherein the working status includes online upgrade status and business processing status;

[0067] When the working status is online upgrade status, the delay monitoring instruction is triggered.

[0068] The embodiment of the present application provides a method for triggering delay monitoring, that is, a delay monitoring triggering scheme based on the online upgrade of the cluster system. It can be understood that the delay problem of the cluster system mainly comes from the online upgrade of the cluster system (that is, it means upgrading the version of the relevant software without affecting the business carried by the cluster system to realize the launch of new functions and new requirements or the repair of known problems). Based on this, during the operation of the cluster system, its working status can be monitored in real time to determine whether the cluster system is undergoing an online upgrade. If the cluster system enters the online upgrade state, the delay monitoring instruction can be automatically triggered to execute S101.

[0069] S102: When the latency information of the cluster node meets the first intervention condition, the data reconstruction service of the target disk in the cluster node is controlled to stop running.

[0070] This step is intended to implement a delay intervention operation in the case of over-delay. Specifically, an intervention condition can be pre-set, namely the above-mentioned first intervention condition, and the delay information of each cluster node can be judged and analyzed using the first intervention condition. If the delay information of a cluster node meets the first preset condition, it can be considered that it has an over-delay problem. At this time, a disk-level intervention operation can be triggered, that is, the data reconstruction service of the target disk in the cluster node is controlled to stop running, that is, only some disks in the cluster node are controlled to stop the data reconstruction service. Of course, if the delay information of a cluster node does not meet the first preset condition, it can be considered that it does not have an over-delay problem. At this time, return to S101 to continue delay monitoring.

[0071] In one embodiment of the present application, when the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running may include:

[0072] When the delay information of the cluster node reaches a preset delay threshold, the cluster node is monitored for delay according to a preset monitoring period to obtain delay growth information within a preset number of preset monitoring periods;

[0073] If the latency growth information in all preset monitoring cycles reaches the preset latency growth threshold, the data reconstruction service of the target disk in the control cluster node is stopped;

[0074] If the delay growth information in all preset monitoring periods does not reach the preset delay growth threshold, the total delay growth information in all preset monitoring periods is counted;

[0075] If the total latency growth information within all preset monitoring cycles exceeds the preset total latency growth threshold, the data reconstruction service of the target disk in the control cluster node stops running;

[0076] If the total delay growth information within all preset monitoring periods does not exceed the preset total delay growth threshold, return to the step of performing delay monitoring on the cluster nodes according to the delay monitoring instruction to obtain the delay information of the cluster nodes.

[0077] The embodiment of the present application provides a specific first intervention condition, which realizes the above-mentioned preliminary monitoring based on the preset delay threshold and the key observation based on the cyclic monitoring period, and performs disk-level intervention operations when it is determined that the cluster node has an over-delay problem. It should be noted that the specific content of the above-mentioned preset conditions (such as the preset delay threshold, the preset number, the preset monitoring cycle, the preset delay growth threshold, the preset total delay growth threshold, etc.) does not affect the implementation of this technical solution, and can be set by technical personnel according to actual conditions, and this application does not limit this.

[0078] The cluster system is specifically a storage cluster system, and the cluster node is specifically a storage node; and controlling the data reconstruction service of the target disk in the cluster node to stop running may include:

[0079] In the cluster node, determine the disk used to provide storage services as the target disk;

[0080] The data reconstruction service of all target disks in the control cluster nodes is stopped.

[0081] The embodiment of the present application provides a method for determining the target disk in a disk-level intervention strategy. For a storage cluster system, the disk in the cluster node used to provide storage services (OSD service, Object-Based StorageDevice) can be used as the target disk, and the data reconstruction service therein can be controlled to stop running. Of course, for cluster systems with other functions, the disk in the cluster node used to provide corresponding functional services can also be used as the target disk and the data reconstruction service therein can be stopped. This application will not go into details here.

[0082] S103: During a preset observation period, the latency of the cluster nodes is monitored to obtain new latency information of the cluster nodes.

[0083] This step aims to achieve continued latency monitoring for each cluster node that has completed the disk-level intervention operation in the cluster system, so as to determine whether the disk-level intervention operation that has been executed has produced effective effects. Specifically, for each cluster node that has completed the disk-level intervention operation, it can enter an observation period of a preset duration, that is, the above-mentioned preset observation time period. During this preset observation time period, the latency monitoring of each corresponding cluster node can continue, so as to obtain new latency information of each cluster node in real time, so as to determine whether the corresponding cluster node still has an over-delay problem based on the new latency information, so as to perform another over-delay intervention operation on the cluster node that still has the over-delay problem. Similarly, the specific duration value of the above-mentioned preset observation time period does not affect the implementation of the present technical solution, and can be set by technical personnel according to actual conditions, and this application does not limit this.

[0084] S104: When the new delay information of the cluster node meets the second intervention condition, the data reconstruction service of all disks in the cluster node is controlled to stop running.

[0085] This step is intended to implement another delay intervention operation in the case of over-delay. Specifically, an intervention condition can be pre-set, namely the above-mentioned second intervention condition. It should be noted that the first intervention condition and the second intervention condition may be the same or different, and the present application does not limit this. Further, the new delay information of each cluster node is continuously judged and analyzed using this second intervention condition. If the new delay information of a cluster node meets the second preset condition, it can be considered that it still has an over-delay problem. At this time, a node-level intervention operation can be triggered, namely, the data reconstruction service of all disks in the cluster node is controlled to stop running, that is, all disks in the cluster node are controlled to stop the data reconstruction service. Of course, if the new delay information of a cluster node does not meet the second preset condition, it can be considered that it no longer has an over-delay problem, that is, the disk-level intervention operation in S102 has produced an effective effect. At this time, return to S101 to continue delay monitoring.

[0086] In one embodiment of the present application, when the new latency information of the cluster node meets the second intervention condition, controlling the data reconstruction service of all disks in the cluster node to stop running may include:

[0087] Determine the delay state of the cluster node within a preset observation time period according to the new delay information of the cluster node; wherein the delay state includes an increasing state and a non-increasing state;

[0088] When the delay state of the cluster node is an increasing state, determining that new delay information of the cluster node satisfies a second intervention condition;

[0089] The data reconstruction service of all disks in the control cluster node stops running.

[0090] The embodiment of the present application provides a specific second intervention condition, which realizes the delay monitoring of the above-mentioned delay state, and performs the node-level intervention operation when it is determined that the cluster node still has the problem of super delay. Obviously, if the delay of the cluster node is still increasing, it means that the above-mentioned disk-level intervention operation has not played an effective role, and the node-level intervention operation can be started at this time; on the contrary, if the delay of the cluster node is no longer increasing, it means that the above-mentioned disk-level intervention operation has played an effective role, and there is no need to start the node-level intervention operation.

[0091] It can be seen that the latency monitoring method of the cluster system provided in the embodiment of the present application can monitor the latency of each cluster node in the cluster system, and trigger different intervention means when the latency information of the cluster node meets different intervention conditions, namely, disk-level intervention means (control the data reconstruction service of the target disk in the cluster node to stop running) and node-level intervention means (control the data reconstruction service of all disks in the cluster node to stop running), realizing real-time latency monitoring and real-time over-latency intervention, which can effectively avoid the business impact caused by over-latency problems. It can be seen that the technical solution realizes more real-time and efficient latency monitoring in the cluster system, and further ensures the stability of the cluster system.

[0092] Based on the above embodiments:

[0093] In one embodiment of the present application, when the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running may include:

[0094] When the latency information of the cluster node meets the first intervention condition, determining the target storage pool to which the cluster node belongs;

[0095] If the target storage pool has a latency intervention flag set, latency monitoring of the cluster nodes will be stopped;

[0096] If the target storage pool is not set with a delay intervention mark, the data reconstruction service of the target disk in the control cluster node is stopped, and a delay intervention mark is added to the target storage pool.

[0097] It is understandable that due to different business requirements, the storage cluster system may divide all cluster nodes into multiple storage pools. The cluster nodes between the storage pools are independent of each other, and each storage pool can exclusively use its own corresponding fault domain. On this basis, it can be set that only a small number (such as 1) of cluster nodes in each storage pool in the storage cluster system are subject to super-delay intervention at the same time, so as to effectively ensure the stability of the system business.

[0098] Specifically, for a storage cluster system, before determining that a certain cluster node has a time-delay problem and performing a disk-level intervention operation on it, it is possible to first determine whether the target storage pool to which it belongs has been marked with a time-delay intervention. If a time-delay intervention mark has been added, it can be determined that a cluster node that has been subjected to a time-delay intervention has already existed in the target storage pool, so there is no need to perform a time-delay intervention on this cluster node. If a time-delay intervention mark has not been added, it can be determined that there is no cluster node that has been subjected to a time-delay intervention in the target storage pool, and a disk-level intervention operation can be performed on the cluster node at this time; at the same time, a time-delay intervention mark can be further added to the target storage pool to avoid other cluster nodes in the target storage pool being subjected to a time-delay intervention at the same time when they are determined to have a time-delay problem, thereby affecting the normal operation of the system business.

[0099] Of course, the number of cluster nodes in the storage pool that can simultaneously perform ultra-delay intervention operations can be set according to actual conditions. Normally, in order to ensure the stability of system services, it can be directly set to 1. Of course, it can also be set proportionally according to the number of cluster nodes in the storage pool. For example, it can be set to a ratio of one tenth. This application does not limit this.

[0100] It can be seen that in the case where the storage cluster system is divided into multiple storage pools, in order to further ensure the stability of the system services within the cluster system, it is set that only a small number (such as one) of cluster nodes in a storage pool are subject to ultra-delay intervention, thereby effectively avoiding the increase in complexity caused by too many cluster nodes being intervened at the same time, thereby further ensuring the stability of the cluster system.

[0101] In one embodiment of the present application, when the new latency information of the cluster node meets the second intervention condition, after controlling the data reconstruction service of all disks in the cluster node to stop running, the method may further include:

[0102] Monitor the latency of cluster nodes and obtain the post-intervention latency information on cluster nodes;

[0103] If the delay information after the intervention of the cluster node meets the preset recovery conditions, the data reconstruction service of all disks in the cluster node is controlled to resume operation, and the delay intervention mark of the target storage pool is cancelled;

[0104] If the post-intervention delay information of the cluster node does not meet the preset recovery conditions, an alarm prompt is output.

[0105] The latency monitoring method of the cluster system provided in the embodiment of the present application can further realize service recovery after successful super-delay intervention and alarm prompt after super-delay intervention fails. Specifically, after performing a node-level intervention operation on a cluster node, the real-time latency monitoring of the cluster node can continue, so as to determine whether the aforementioned node-level intervention operation is effective based on the delay information after intervention, that is, to determine whether the current super-delay intervention for the cluster node is successful. In the implementation process, if the delay information after the intervention of the cluster node meets the preset recovery conditions, it means that the previous super-delay intervention for the cluster node has been successful. At this time, the data reconstruction service of all disks in the cluster node can be restored, and the delay intervention mark of the target storage pool to which it belongs can be cancelled; if the delay information after the intervention of the cluster node does not meet the preset recovery conditions, it means that the previous super-delay intervention for the cluster node has failed. At this time, an alarm prompt can be output to promptly remind the technical staff to perform abnormal operation and maintenance.

[0106] In one embodiment of the present application, the post-intervention delay information includes delay change information within a first preset number of preset observation periods and real-time delay information after the first preset number of preset observation periods. Determining whether the post-intervention delay information of the cluster node meets the preset recovery condition may include:

[0107] If all the delay change information within the preset observation period is delay decreasing information, then determine whether the real-time delay information is lower than the preset threshold;

[0108] If the real-time delay information is lower than a preset threshold, it is determined that the post-intervention delay information of the cluster node meets the preset recovery condition;

[0109] If the real-time delay information is not lower than the preset threshold, it is determined that the post-intervention delay information of the cluster node does not meet the preset recovery condition;

[0110] If the delay change information within all preset observation periods is not delay decreasing information, it is determined that the post-intervention delay information of the cluster node does not meet the preset recovery condition.

[0111] The embodiment of the present application provides a specific preset recovery condition, which realizes the above-mentioned delay information monitoring based on the cyclic observation period and the preset threshold, so as to determine whether the over-delay problem of the corresponding cluster node is effectively solved according to the analysis results of the delay information after the intervention. Similarly, the specific content of each of the above-mentioned preset conditions (such as the preset threshold, the first preset number, the preset observation period, etc.) does not affect the implementation of the present technical solution, and can be set by the technical personnel according to the actual situation, and this application does not limit this.

[0112] Based on the above embodiments, the embodiment of the present application takes a storage cluster system as an example to provide another method for monitoring latency of a cluster system.

[0113] Please refer to Figure 3 , Figure 3 This is a flow chart of another method for monitoring the time delay of a cluster system provided by the present application, and its implementation process is as follows:

[0114] 1. After the online upgrade starts, the latency monitoring module is called to record and analyze the latency of all OSD services (storage services) in the cluster system. The implementation process is as follows: Figure 4 As shown, Figure 4 A schematic diagram of a preliminary delay monitoring method provided in this application, the implementation process of which may include:

[0115] (1) The latency monitoring module is started by the MON service (storage management service) and counts the latency (in seconds) of all OSD services every second;

[0116] (2) If the delay of OSD service exceeds 10s, it will enter the key observation period;

[0117] (3) From the moment of entering the observation period, the latency changes of the last five OSD services are recorded in the form of a sliding window: if the latency increase in the last five cycles is greater than or equal to 4s, it means that the latency is increasing rapidly, and the latency intervention module is triggered; otherwise, continue to observe. If the cumulative latency of the OSD service is greater than or equal to 18s, it means that the latency is still increasing steadily, and the latency intervention module is also triggered; if neither condition is met, it means that although the latency is large, it is relatively stable overall and can continue to be observed.

[0118] 2. After the delay monitoring module analyzes that the OSD service delay has reached the intervention threshold, it calls the delay intervention module to actively intervene. The implementation process is as follows: Figure 5 As shown, Figure 5 A flowchart of a time delay intervention method provided in this application, the implementation process of which may include:

[0119] First, enter the OSD service of the latency intervention module, the latency is no less than 14s, record the latency data at this time, and perform the following actions based on the cluster node and storage pool information to which the OSD service belongs:

[0120] (1) Check whether there is a delayed intervention mark in the same storage pool. The same storage pool can have at most one intervention mark at the same time to avoid the complexity of multiple OSD services or cluster nodes intervening at the same time, which makes it difficult to accurately evaluate the intervention effect. If there is an intervention mark, it means that other OSD services have triggered intervention actions, and the current OSD service can also share the intervention effect. Therefore, the intervention action of this OSD service is terminated and enters the intervention observation period; if there is no intervention mark, the storage pool is set with an intervention mark and the intervention action is executed.

[0121] (2) For the OSD service that triggers the intervention flag, the intervention action is performed. First, disk-level intervention is performed, that is, only the disk service corresponding to the OSD service is stopped and reconstructed, and then a 5s observation period is performed. If the latency still increases after the observation period, it is considered that the intervention effect is not obvious and node-level intervention is required. If the latency no longer increases after the observation period, it is considered that the intervention is effective and the intervention observation period begins.

[0122] (3) For those who have entered the node intervention process, all disks on the cluster node where the OSD service is located will be reconstructed and enter a 10-second observation period. If the latency continues to increase after the observation period, the intervention is considered ineffective. At this time, there is no intervention method and the intervention is considered a failure. If the latency no longer increases after the observation period, the intervention is considered effective and the intervention observation period begins.

[0123] (4) Enter the intervention observation period, and perform different judgment actions according to the delay size: If the delay is still >= 30s, it means that the intervention is invalid, report the delay information alarm at this time, and exit the delay intervention module. If the delay is <30s and <10s, it means that the intervention has achieved the expected effect, roll back the intervention action (restore the cluster nodes involved or disk-level reconstruction), cancel the storage pool intervention mark, and exit the delay intervention module; if the delay is still >= 10s, continue to observe. If the delay repeatedly exceeds 30s, the intervention is still invalid.

[0124] 4. Regardless of whether the intervention is successful or ineffective, delay monitoring continues until the delay intervention module is entered again, or the upgrade is paused or completed.

[0125] It can be seen that the latency monitoring method of the cluster system provided in the embodiment of the present application can monitor the latency of each cluster node in the cluster system, and trigger different intervention means when the latency information of the cluster node meets different intervention conditions, namely, disk-level intervention means (control the data reconstruction service of the target disk in the cluster node to stop running) and node-level intervention means (control the data reconstruction service of all disks in the cluster node to stop running), realizing real-time latency monitoring and real-time over-latency intervention, which can effectively avoid the business impact caused by over-latency problems. It can be seen that the technical solution realizes more real-time and efficient latency monitoring in the cluster system, and further ensures the stability of the cluster system.

[0126] An embodiment of the present application provides a delay monitoring device for a cluster system.

[0127] Please refer to Figure 6 , Figure 6 This is a structural diagram of a time delay monitoring device for a cluster system provided by the present application. The cluster system includes a management service platform and multiple cluster nodes. Each cluster node is communicatively connected to the management service platform. The device is applied to the management service platform and may include:

[0128] The first monitoring module 1 is used to monitor the latency of each cluster node in the cluster system according to the latency monitoring instruction to obtain the latency information of the cluster node;

[0129] A first control module 2, configured to control the data reconstruction service of the target disk in the cluster node to stop running when the latency information of the cluster node meets the first intervention condition;

[0130] The second monitoring module 3 is used to monitor the latency of the cluster nodes within a preset observation time period and obtain new latency information of the cluster nodes;

[0131] The second control module 4 is configured to control the data reconstruction services of all disks in the cluster node to stop running when the new delay information of the cluster node meets the second intervention condition.

[0132] It can be seen that the latency monitoring device of the cluster system provided in the embodiment of the present application can monitor the latency of each cluster node in the cluster system, and trigger different intervention means when the latency information of the cluster node meets different intervention conditions, namely, disk-level intervention means (control the data reconstruction service of the target disk in the cluster node to stop running) and node-level intervention means (control the data reconstruction service of all disks in the cluster node to stop running), realizing real-time latency monitoring and real-time over-latency intervention, which can effectively avoid the business impact caused by over-latency problems. It can be seen that the technical solution realizes more real-time and efficient latency monitoring in the cluster system, and further ensures the stability of the cluster system.

[0133] In one embodiment of the present application, the above-mentioned first control module 2 can be specifically used to monitor the latency of the cluster node according to the preset monitoring period when the latency information of the cluster node reaches a preset latency threshold, and obtain the latency growth information within a preset number of preset monitoring periods; if the latency growth information within all preset monitoring periods reaches the preset latency growth threshold, the data reconstruction service of the target disk in the cluster node is controlled to stop running; if the latency growth information within all preset monitoring periods does not reach the preset latency growth threshold, the total latency growth information within all preset monitoring periods is counted; if the total latency growth information within all preset monitoring periods exceeds the preset total latency growth threshold, the data reconstruction service of the target disk in the cluster node is controlled to stop running; if the total latency growth information within all preset monitoring periods does not exceed the preset total latency growth threshold, the step of performing latency monitoring on the cluster node according to the latency monitoring instruction to obtain the latency information of the cluster node is returned.

[0134] In one embodiment of the present application, the cluster system is specifically a storage cluster system, and the cluster node is specifically a storage node; the above-mentioned first control module 2 can be specifically used to determine the disk used to provide storage services as the target disk within the cluster node; and control the data reconstruction service of all target disks in the cluster node to stop running.

[0135] In one embodiment of the present application, the above-mentioned second control module 4 can be specifically used to determine the latency state of the cluster node within a preset observation time period based on the new latency information of the cluster node; wherein the latency state includes a growing state and a non-growing state; when the latency state of the cluster node is a growing state, it is determined that the new latency information of the cluster node meets the second intervention condition; and the data reconstruction service of all disks in the cluster node is controlled to stop running.

[0136] In one embodiment of the present application, the above-mentioned first control module 2 can be specifically used to determine the target storage pool to which the cluster node belongs when the latency information of the cluster node meets the first intervention condition; if the target storage pool is set with a latency intervention mark, then the latency monitoring of the cluster node is stopped; if the target storage pool is not set with a latency intervention mark, then the data reconstruction service of the target disk in the cluster node is controlled to stop running, and a latency intervention mark is added for the target storage pool.

[0137] In one embodiment of the present application, the latency monitoring device of the cluster system may further include a third control module, which is used to monitor the latency of the cluster node after controlling the data reconstruction services of all disks in the cluster node to stop running when the new latency information of the cluster node meets the second intervention condition, and obtain the post-intervention latency information on the cluster node; if the post-intervention latency information of the cluster node meets the preset recovery condition, control the data reconstruction services of all disks in the cluster node to resume running, and cancel the latency intervention mark of the target storage pool; if the post-intervention latency information of the cluster node does not meet the preset recovery condition, output an alarm prompt.

[0138] In one embodiment of the present application, the latency monitoring device of the cluster system may further include a trigger module for determining the working status of the cluster system; wherein the working status includes an online upgrade status and a service processing status; when the working status is an online upgrade status, a latency monitoring instruction is triggered.

[0139] For an introduction to the device provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0140] An embodiment of the present application provides an electronic device.

[0141] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in this application, and the electronic device may include:

[0142] A memory 11, used for storing computer programs;

[0143] The processor 10 is configured to implement the steps of any one of the above-mentioned methods for monitoring the time delay of a cluster system when executing a computer program.

[0144] like Figure 7 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0145] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0146] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the method for monitoring the time delay of a cluster system.

[0147] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0148] For each cluster node in the cluster system, delay monitoring is performed on the cluster node according to the delay monitoring instruction to obtain delay information of the cluster node;

[0149] When the latency information of the cluster node meets the first intervention condition, the data reconstruction service of the target disk in the cluster node is controlled to stop running;

[0150] During the preset observation period, the latency of the cluster nodes is monitored to obtain new latency information of the cluster nodes;

[0151] When the new latency information of the cluster node meets the second intervention condition, the data reconstruction service of all disks in the cluster node is controlled to stop running.

[0152] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0153] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0154] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0155] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 7 More or fewer components than shown, or combinations of certain components.

[0156] An embodiment of the present application provides a computer-readable storage medium.

[0157] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for monitoring the delay of a cluster system can be implemented.

[0158] Among them, the computer-readable storage medium can be any available medium that can be stored in a computer or a data storage device such as a server or data center that includes one or more available media integrated. For example, it can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk) and other media that can store computer program codes.

[0159] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0160] An embodiment of the present application provides a computer program product.

[0161] The computer program product provided in the embodiments of the present application includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of any of the above-mentioned methods for monitoring the delay of a cluster system can be implemented.

[0162] Specifically, in the above embodiments, all or part of them can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of them can be implemented in the form of a computer program product.

[0163] Among them, the computer program product may include one or more computer programs / instructions, and when the computer program / instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application may be generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line, etc.) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0164] For an introduction to the computer program product provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0165] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0166] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0167] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0168] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A method for monitoring the time delay of a cluster system, characterized in that: The cluster system includes a management service platform and a plurality of cluster nodes, each of the cluster nodes is in communication connection with the management service platform, and the method is applied to the management service platform, including: For each cluster node in the cluster system, performing latency monitoring on the cluster node according to the latency monitoring instruction to obtain latency information of the cluster node; When the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running; During a preset observation period, monitoring the latency of the cluster nodes to obtain new latency information of the cluster nodes; When the new delay information of the cluster node meets the second intervention condition, the data reconstruction service of all disks in the cluster node is controlled to stop running.

2. The method for monitoring the time delay of a cluster system according to claim 1, characterized in that: When the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running includes: When the delay information of the cluster node reaches a preset delay threshold, delay monitoring is performed on the cluster node according to a preset monitoring period to obtain delay growth information within a preset number of the preset monitoring periods; If all the delay growth information within the preset monitoring period reaches the preset delay growth threshold, controlling the data reconstruction service of the target disk in the cluster node to stop running; If the delay growth information in all the preset monitoring periods does not reach the preset delay growth threshold, then the total delay growth information in all the preset monitoring periods is counted; If the total delay growth information within all the preset monitoring periods exceeds the preset total delay growth threshold, controlling the data reconstruction service of the target disk in the cluster node to stop running; If the total delay growth information within all the preset monitoring periods does not exceed the preset total delay growth threshold, return to the step of performing delay monitoring on the cluster nodes according to the delay monitoring instruction to obtain the delay information of the cluster nodes.

3. The time delay monitoring method of a cluster system according to claim 1, characterized in that: The cluster system is specifically a storage cluster system, and the cluster nodes are specifically storage nodes; Controlling the data reconstruction service of the target disk in the cluster node to stop running includes: In the cluster node, determining a disk used to provide storage services as the target disk; Control the data reconstruction services of all the target disks in the cluster nodes to stop running.

4. The method for monitoring the time delay of a cluster system according to claim 1, characterized in that: When the new latency information of the cluster node meets the second intervention condition, controlling the data reconstruction service of all disks in the cluster node to stop running includes: Determine the delay state of the cluster node within the preset observation time period according to the new delay information of the cluster node; wherein the delay state includes an increasing state and a non-increasing state; When the delay state of the cluster node is the increasing state, determining that new delay information of the cluster node satisfies the second intervention condition; Control the data reconstruction service of all disks in the cluster node to stop running.

5. The method for monitoring the time delay of a cluster system according to any one of claims 1 to 4, characterized in that: When the latency information of the cluster node meets the first intervention condition, controlling the data reconstruction service of the target disk in the cluster node to stop running includes: When the latency information of the cluster node meets the first intervention condition, determining the target storage pool to which the cluster node belongs; If the target storage pool is set with a delay intervention mark, then stopping the delay monitoring of the cluster node; If the target storage pool is not set with the delay intervention mark, the data reconstruction service of the target disk in the cluster node is controlled to stop running, and the delay intervention mark is added to the target storage pool.

6. The time delay monitoring method of the cluster system according to claim 5, characterized in that: When the new latency information of the cluster node meets the second intervention condition, after controlling the data reconstruction service of all disks in the cluster node to stop running, the method further includes: Performing delay monitoring on the cluster nodes to obtain post-intervention delay information on the cluster nodes; If the post-intervention delay information of the cluster node meets the preset recovery condition, the data reconstruction service of all disks in the cluster node is controlled to resume operation, and the delay intervention mark of the target storage pool is cancelled; If the post-intervention delay information of the cluster node does not meet the preset recovery condition, an alarm prompt is output.

7. The method for monitoring the time delay of a cluster system according to claim 1, characterized in that: Also includes: Determine the working state of the cluster system; wherein the working state includes an online upgrade state and a service processing state; When the working state is the online upgrade state, the delay monitoring instruction is triggered.

8. A time delay monitoring device for a cluster system, characterized in that: The cluster system includes a management service platform and a plurality of cluster nodes, each of the cluster nodes is in communication connection with the management service platform, and the device is applied to the management service platform, including: A first monitoring module, configured to perform latency monitoring on each cluster node in the cluster system according to a latency monitoring instruction to obtain latency information of the cluster node; A first control module, configured to control the data reconstruction service of the target disk in the cluster node to stop running when the latency information of the cluster node meets a first intervention condition; A second monitoring module is used to monitor the latency of the cluster node within a preset observation time period and obtain new latency information of the cluster node; The second control module is used to control the data reconstruction services of all disks in the cluster node to stop running when the new delay information of the cluster node meets the second intervention condition.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the delay monitoring method of the cluster system as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the delay monitoring method of a cluster system according to any one of claims 1 to 7 are implemented.

Images