Metadata compression method, system, device and storage medium for distributed clusters

Abstract

The present application discloses a metadata compression method, system, device and storage medium for a distributed cluster, which is applied to the field of storage technology and to any one monitoring service in a distributed cluster, including: judging whether the disk occupied space of the metadata information managed by the monitoring service itself exceeds the space occupied threshold; if so, judging whether A>(B / 2+1) holds true; if so, judging whether there is a monitoring service in the distributed cluster that is currently compressing metadata information; if not, compressing the metadata information managed by itself; A represents the number of monitoring services currently surviving in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster. By applying the solution of the present application, metadata compression of the distributed cluster can be effectively performed, avoiding the situation where the disk space of the node is exhausted, and at the same time avoiding the situation where the service of the distributed cluster is unavailable.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a metadata compression method, system, device and storage medium for a distributed cluster. Background Art

[0002] Distributed clusters can monitor the health of their clusters using a monitor service. A distributed cluster typically has multiple monitor services, deployed on different storage server nodes. The monitor service maintains cluster metadata, such as the OSDmap (Object-based Storage Device map) and the PGmap (Placement Group map), and persists this metadata in a backend key-value store (kv-store).

[0003] The monitor service can compress the metadata it manages to reduce disk usage. Specifically, the monitor service can compress the kv-store it manages. Typically, kv-store compaction is performed automatically and periodically, but can also be triggered manually. However, in some scenarios, such as large-scale distributed clusters, the space occupied by individual OSDmaps and PGmaps can be large, and the number of these can be large, potentially thousands of OSDmaps or tens of thousands of PGmaps. In such scenarios, if an anomaly such as a monitor service failure occurs, OSDmap and PGmap version updates can occur frequently, leading to a large amount of kv-store space. If automatic compaction is not triggered promptly, disk space exhaustion can occur. In this case, disk space can be reduced by manually triggering compaction, for example, using the CLI commands provided by the distributed cluster. However, disk space exhaustion can still occur if human monitoring and intervention are not timely.

[0004] Therefore, in the current solution, each monitor service regularly collects information about its node's kv-store, including the amount of disk space it occupies, and broadcasts this information to other monitor services. There are currently two solutions. One is for the master monitor service to periodically check the kv-store information of all monitor services. If the kv-store managed by a monitor service occupies too much disk space, a compression instruction is issued to that monitor service, instructing it to compress the kv-store. The other solution is for each monitor service to perform compression when it detects that its own kv-store is too large.

[0005] However, when using the current solution, the services of the distributed cluster often become unavailable.

[0006] In summary, how to effectively compress the metadata of a distributed cluster to avoid the disk space exhaustion of a node and solve the problem of service unavailability of the distributed cluster is a technical problem that currently needs to be solved urgently by those skilled in the art. Summary of the Invention

[0007] The purpose of the present invention is to provide a metadata compression method, system, device and storage medium for a distributed cluster, so as to effectively compress the metadata of the distributed cluster, avoid the situation where the disk space of the node is exhausted, and solve the problem of service unavailability of the distributed cluster.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] A metadata compression method for a distributed cluster, applied to any monitoring service in the distributed cluster, includes:

[0010] Determine whether the disk space occupied by the metadata information managed by the monitoring service itself exceeds a preset space occupation threshold;

[0011] If the space occupation threshold is exceeded, determine whether A>(B / 2+1);

[0012] If so, determining whether there is a monitoring service currently compressing metadata information in the distributed cluster;

[0013] If it does not exist, the metadata information managed by itself is compressed;

[0014] Among them, A represents the number of monitoring services currently surviving in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster.

[0015] Preferably, the determining whether there is a monitoring service currently compressing metadata information in the distributed cluster includes:

[0016] Taking itself as the initiator and each currently surviving monitoring service in the distributed cluster other than itself as each receiver, a compression request is sent to each receiver;

[0017] Wherein, for any one receiving party that receives the compression request, when the protocol variable of the receiving party is in the default state, the receiving party feeds back a first response instruction indicating that it agrees to the compression, and sets the protocol variable of the receiving party to the initiator number value carried in the compression request; when the protocol variable of the receiving party is not in the default state, the receiving party feeds back a second response instruction indicating that it does not agree to the compression;

[0018] When the first response instruction fed back by each recipient is received, determining that no monitoring service currently compressing the metadata information exists; otherwise, determining that a monitoring service currently compressing the metadata information exists;

[0019] Accordingly, the metadata compression method of the distributed cluster further includes:

[0020] After determining that there is no monitoring service currently compressing metadata information and compressing the metadata information managed by itself, a compression cancellation request is sent to each receiving party;

[0021] After determining that there is a monitoring service that is currently compressing the metadata information, a compression cancellation request is sent to each receiving party;

[0022] Wherein, for any one receiving party that receives the compression cancellation request, the receiving party restores its own protocol variables to a default state.

[0023] Preferably, after sending the compression request to each receiving party, the method further includes:

[0024] Set the initiator's own protocol variable to the initiator number value carried in the compression request;

[0025] Accordingly, after sending the compression cancellation request to each recipient, the following steps are also included:

[0026] Restore the initiator's own contract variables to their default state.

[0027] Preferably, it also includes:

[0028] When the protocol variable itself is not in the default state for a first timeout period and no compression cancellation request is received from any monitoring service within the first timeout period, the protocol variable itself is restored to the default state.

[0029] Preferably, it also includes:

[0030] After exiting the distributed cluster due to an exception, rejoin the distributed cluster to restore to a surviving state when the exception is resolved.

[0031] Preferably, after determining that there is a monitoring service currently compressing metadata information, the method further includes:

[0032] Set its own compression flag to its own number value;

[0033] Among them, for any monitoring service, the compression flag of the monitoring service is automatically reduced according to the set rules until it is reduced to the set target value; the number values of each monitoring service are different, and the number value of any monitoring service is not less than the target value; for any monitoring service, when the compression flag of the monitoring service is not the target value, the monitoring service is prohibited from triggering the metadata compression process.

[0034] Preferably, after compressing the metadata information managed by itself, the method further includes:

[0035] Determine whether the disk space occupied by the metadata information managed by itself exceeds the space occupation threshold;

[0036] If so, the compression flag thereof is set to a first value; wherein the first value is higher than the number value of any monitoring service.

[0037] A distributed cluster metadata compression system, applied to any monitoring service in the distributed cluster, including:

[0038] a disk space occupancy determination module, configured to determine whether the disk space occupied by the metadata information managed by the monitoring service itself exceeds a preset space occupancy threshold, and trigger a first condition determination module if the space occupancy threshold is exceeded;

[0039] The first condition judgment module is used to: judge whether A>(B / 2+1) is established, and if so, trigger the second condition judgment module;

[0040] The second condition judgment module is used to: judge whether there is a monitoring service that is currently compressing metadata information in the distributed cluster, and if not, trigger the compression execution module;

[0041] The compression execution module is used to compress the metadata information managed by itself;

[0042] Among them, A represents the number of monitoring services currently surviving in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster.

[0043] A metadata compression device for a distributed cluster, comprising:

[0044] memory for storing computer programs;

[0045] A processor is configured to execute a computer program to implement the steps of the metadata compression method for a distributed cluster as described above.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned distributed cluster metadata compression method.

[0047] Applying the technical solution provided by the embodiment of the present invention, the applicant has taken into consideration that in the traditional solution, whether the master monitor service issues a compression instruction to the slave monitor service to instruct it to compress, or each monitor service performs compression when it detects that its kv-store is too large, the service of the distributed cluster will become unavailable. This is because the monitor service cannot respond to any service request when compressing the kv-store, and the compression time may exceed the heartbeat time of the monitor service, causing the monitor service to be kicked out of the cluster. If more than half of the monitor services are not in the distributed cluster, the master monitor service cannot be elected, which in turn causes the distributed cluster to be unable to provide services to the outside world, that is, the service of the distributed cluster becomes unavailable.

[0048] In this regard, in the solution of this application, for any monitoring service, the monitoring service determines that the disk space occupied by the metadata information it manages exceeds the preset space occupation threshold, indicating that the monitoring service needs to compress the metadata information. However, in order to avoid the situation where the distributed cluster service is unavailable, this application will further determine whether A>(B / 2+1) is established, where A represents the number of currently surviving monitoring services in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster. When A>(B / 2+1) is established, that is, A-1>B / 2, that is, even if the monitoring service compresses the metadata information, the number of remaining surviving nodes is greater than half of the total number of monitoring services, so the distributed cluster service will not be unavailable. In addition, considering that in some situations there may be two or more monitoring services initiating requests at the same time, therefore, in the scheme of the present application, only one monitoring service is allowed to compress the metadata information at the same time, that is, in the scheme of the present application, not only is A>(B / 2+1) required to be established, but it is also necessary to determine whether there is a monitoring service in the distributed cluster that is currently compressing the metadata information. Only when there is no other monitoring service that is compressing the metadata information will the metadata information managed by itself be compressed, thereby further avoiding the unavailability of the distributed cluster service.

[0049] In summary, the solution of the present application can effectively compress the metadata of the distributed cluster, avoid the situation where the disk space of the node is exhausted, and at the same time avoid the situation where the service of the distributed cluster is unavailable. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is a flowchart of an implementation method of metadata compression for a distributed cluster in the present invention;

[0052] Figure 2a A schematic diagram of a scenario in which the monitoring service of the present invention successfully performs compression;

[0053] Figure 2b A schematic diagram of a scenario in which the receiver does not respond when the monitoring service performs compression in the present invention;

[0054] Figure 2c A schematic diagram of a scenario in which a compression conflict occurs when the monitoring service performs compression in the present invention;

[0055] Figure 2d This is a schematic diagram of a scenario in which the monitoring service does not initiate compression in the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of a metadata compression system for a distributed cluster in the present invention;

[0057] Figure 4 This is a structural diagram of a metadata compression device for a distributed cluster in the present invention. DETAILED DESCRIPTION

[0058] The core of the present invention is to provide a metadata compression method for a distributed cluster, which can effectively compress the metadata of the distributed cluster, avoid the situation where the disk space of the node is exhausted, and at the same time avoid the situation where the service of the distributed cluster is unavailable.

[0059] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] Please refer to Figure 1 , Figure 1 The following is a flowchart of a metadata compression method for a distributed cluster according to the present invention. The metadata compression method for a distributed cluster can be applied to any monitoring service in a distributed cluster and may include the following steps:

[0061] Step S101: determine whether the disk space occupied by the metadata information managed by the monitoring service itself exceeds a preset space occupation threshold; if it exceeds the space occupation threshold, execute step S102.

[0062] A distributed cluster can include multiple nodes and can be configured with multiple monitor services. The specific number of nodes and monitor services can be set and adjusted based on actual needs. In typical scenarios, the number of monitor services is much lower than the number of nodes. For example, if a distributed cluster has dozens or more nodes, a few monitor services are sufficient. Different monitor services need to be configured on different storage server nodes.

[0063] The monitor service is a monitoring service. As described above, the monitor service can maintain the cluster's OSDmap, PGmap and other metadata information, and persist these metadata information in the back-end kv-store. Therefore, in such occasions, the monitoring service manages the metadata information, that is, it manages the kv-store. The monitoring service determines whether the disk space occupied by the metadata information it manages exceeds the preset space occupation threshold, that is, it determines whether the disk space occupied by the kv-store it manages exceeds the space occupation threshold. However, it should be pointed out that in other occasions, when other forms other than kv-store are used to store metadata information, the principles of this application can also be applied, and it does not affect the implementation of the present invention.

[0064] Because the solution of this application can be applied to any monitoring service in a distributed cluster, each monitoring service will determine whether the disk space occupied by the metadata information it manages exceeds a preset space usage threshold. This determination can be made in real time or periodically. In other words, each monitoring service can implement the solution of this application to determine whether to allow compression through subsequent steps when the disk space occupied by the metadata information it manages is too large.

[0065] When any monitoring service determines that the disk space occupied by the metadata information it manages exceeds the preset space occupation threshold, the monitoring service can execute the operation of step S102. Of course, if the judgment result is that it does not exceed the preset space occupation threshold, no subsequent operations are required.

[0066] Step S102: Determine whether A>(B / 2+1) holds true; if so, proceed to step S103. A represents the number of currently surviving monitoring services in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster.

[0067] A>(B / 2+1) holds, that is, A-1>B / 2 holds. A-1 represents the number of surviving monitoring services excluding itself. That is to say, a monitoring service needs to compress metadata information. This application requires that during the process of metadata information compression of the monitoring service, the number of remaining surviving monitoring services is still greater than B / 2, that is, greater than half of the total number of monitoring services, to avoid the situation where the distributed cluster service is unavailable due to the inability to elect the main monitoring service.

[0068] Furthermore, if A>(B / 2+1) is not determined, subsequent steps may not be executed until A>(B / 2+1) is determined to be true, at which point subsequent steps may be allowed to proceed. Of course, in some cases, to avoid repeatedly determining whether A>(B / 2+1) is true, a time interval may be set such that whenever A>(B / 2+1) is not determined to be true, the monitoring service will not re-determine until after the time interval.

[0069] Step S103: Determine whether there is a monitoring service that is currently compressing metadata information in the distributed cluster; if not, execute step S104.

[0070] Step S104: compress the metadata information managed by itself.

[0071] Considering that there may be two or more monitoring services initiating compression at the same time, in order to avoid the situation where the distributed cluster service is unavailable, in the solution of this application, only one monitoring service is allowed to compress metadata information at the same time.

[0072] The monitoring service will determine whether there are other monitoring services in the distributed cluster that are currently compressing metadata information. If so, the monitoring service will need to wait until there are no other monitoring services that are currently compressing metadata information before it can execute step S104, that is, the monitoring service will compress the metadata information it manages.

[0073] There are many ways to determine whether there are other monitoring services in the distributed cluster that are currently compressing metadata information. For example, by querying the main monitoring service, or in a specific embodiment of the present invention, by interacting with other monitoring services.

[0074] Specifically, in a specific embodiment of the present invention, step S103 specifically includes:

[0075] Step 1: Use itself as the initiator and each surviving monitoring service in the distributed cluster as the receiver, and send a compression request to each receiver.

[0076] For any receiver that receives a compression request, when the receiver's protocol variable is in the default state, the receiver feeds back a first response instruction indicating that it agrees to the compression and sets the receiver's protocol variable to the initiator number value carried in the compression request; when the receiver's protocol variable is not in the default state, the receiver feeds back a second response instruction indicating that it disagrees with the compression;

[0077] Step 2: upon receiving the first response instruction fed back by each receiving party, determining that no monitoring service currently compresses the metadata information exists; otherwise, determining that a monitoring service currently compresses the metadata information exists;

[0078] Accordingly, when this implementation is adopted, the metadata compression method of the distributed cluster of the present application further includes:

[0079] The first step: after determining that there is no monitoring service currently compressing metadata information and compressing the metadata information managed by itself, a compression cancellation request is sent to each recipient;

[0080] The second step: after determining that there is a monitoring service that is currently compressing the metadata information, a compression cancellation request is sent to each receiving party;

[0081] Wherein, for any receiver that receives the compression cancellation request, the receiver restores its own protocol variables to the default state.

[0082] For easier understanding, see Figure 2a , Figure 2b , Figure 2c as well as Figure 2d , this implementation method is described. Figure 2a A diagram of a scenario where compression is successfully performed for a monitoring service. Figure 2b A diagram showing a scenario where the receiver does not respond when performing compression for the monitoring service. Figure 2c A diagram showing a scenario where compression conflicts occur when performing compression for a monitoring service. Figure 2d Schematic diagram of a scenario where compression is not initiated for monitoring services.

[0083] In this implementation, the interaction between monitoring services ensures that only one monitoring service compresses metadata information at the same time.

[0084] by Figure 2a For example, Figure 2a In the example, monitoring service 4 is the initiator and the remaining surviving monitoring services are the receivers, i.e. monitoring service 0, monitoring service 1, monitoring service 2, and monitoring service 3 are all receivers. Monitoring service 4 as the initiator will send a compression request to each receiver, i.e. send compact_request.

[0085] Each monitoring service is added with a contract variable, namely a compacting variable. The default state of the contract variable can be pre-set uniformly, for example, to empty or 0.

[0086] For example, Figure 2aIn the case where the protocol variables of each receiver are empty, that is, they are all in the default state, then these four receivers will feedback the first response instruction to the initiator to indicate their agreement to compression, and set their respective protocol variables to the initiator number value carried in the compression request. That is, monitoring service 0, monitoring service 1, monitoring service 2, and monitoring service 3 will all set their respective protocol variables to the initiator number value 4, indicating that monitoring service 4 is currently compressing the metadata information.

[0087] Monitoring service 4 sent 4 compression requests, so it also needs to receive 4 first response instructions, that is, it needs to receive the first response instructions sent by monitoring service 0, monitoring service 1, monitoring service 2, and monitoring service 3 respectively, so as to determine that there is currently no monitoring service that is compressing metadata information. Otherwise, it is deemed that there is currently a monitoring service that is compressing metadata information.

[0088] Figure 2a In the example, the monitoring service 4 receives four first response instructions and determines that there is no monitoring service currently compressing metadata information. Then, the monitoring service 4 compresses the metadata information it manages. After the compression is completed, the monitoring service 4 also sends a compression cancellation request to each recipient, so that these recipients restore their respective protocol variables to the default state. Figure 2a In the case of , monitoring service 4 sends a compression cancellation request to monitoring service 0, monitoring service 1, monitoring service 2, and monitoring service 3, so that these four receivers restore their respective protocol variables from the initiator number value 4 to the default state, that is, the empty state.

[0089] by Figure 2b For example, Figure 2b In the example, monitoring service 4 is also the initiator, and the remaining surviving monitoring services are the receivers, that is, monitoring service 0, monitoring service 1, monitoring service 2, and monitoring service 3 are all receivers. Monitoring service 4, as the initiator, will send a compression request to each receiver.

[0090] Monitoring service 4 sent 4 compression requests, so it also needs to receive 4 first response instructions. That is, if any one of the recipients does not respond, or responds with a second response instruction indicating disagreement with compression, monitoring service 4 will regard it as a monitoring service that is currently compressing metadata information.

[0091] And in Figure 2bIn the case where monitoring service 4 only receives the first response instructions fed back by monitoring service 0, monitoring service 1, and monitoring service 2, for example, due to communication anomalies, monitoring service 3 failure, etc., monitoring service 3 times out and does not respond. That is, monitoring service 4 does not receive the first response instruction fed back by monitoring service 3 within the set time. Therefore, monitoring service 4 regards that there is currently a monitoring service that is compressing metadata information. That is, monitoring service 4 determines that there is currently a monitoring service that is compressing metadata information, and thus sends a compression cancellation request to each recipient. Figure 2b In the example, monitoring service 4 sends a compression cancellation request to monitoring service 0, monitoring service 1, monitoring service 2, and monitoring service 3, so that each receiver restores its own protocol variable from the initiator number value 4 to the default state.

[0092] by Figure 2c For example, Figure 2c In the example, monitoring service 4 is the initiator and the remaining surviving monitoring services are the receivers. Figure 2c In this example, since Monitoring Service 3 has failed, the only surviving Monitoring Services besides Monitoring Service 4 are Monitoring Service 0, Monitoring Service 1, and Monitoring Service 2. Monitoring Service 3 is not a receiver. Monitoring Service 4, as the initiator, sends compression requests to Monitoring Service 0, Monitoring Service 1, and Monitoring Service 2.

[0093] Monitoring service 4 sends three compression requests, and therefore also needs to receive three first response instructions. Figure 2c In the example, all three recipients respond with a first response indicating their agreement to compression. Therefore, Monitoring Service 4 can compress the metadata it manages. Simultaneously, Monitoring Service 0, Monitoring Service 1, and Monitoring Service 2 set their respective protocol variables to the initiator ID value 4.

[0094] At this time, the previously failed monitoring service 3 has returned to normal, rejoined the distributed cluster, and sent compression requests to each surviving monitoring service except monitoring service 3 itself, that is, Figure 2c In the example, Monitoring Service 3 sends compression requests to Monitoring Service 0, Monitoring Service 1, Monitoring Service 2, and Monitoring Service 4. At this point, because Monitoring Service 4 is compressing metadata, the protocol variables of Monitoring Service 0, Monitoring Service 1, and Monitoring Service 2 are all set to the initiator ID value 4. In response to the compression request sent by Monitoring Service 3, Monitoring Service 0, Monitoring Service 1, and Monitoring Service 2 all send back a second response instruction to Monitoring Service 3, indicating that they do not agree to the compression.

[0095] Furthermore, in a specific embodiment of the present invention, after sending the compression request to each receiving party as described in step 1 above, the following steps may also be included:

[0096] Set the initiator's own protocol variable to the initiator number value carried in the compression request;

[0097] Accordingly, after sending the compression cancellation request to each receiving party as described in the first step above, the following steps may also be included:

[0098] Restore the initiator's own protocol variables to the default state;

[0099] After sending the compression cancellation request to each recipient that has fed back the first response instruction as described in the second step above, the following steps may also be included:

[0100] Restore the initiator's own contract variables to their default state.

[0101] In the aforementioned implementation, the recipients primarily adjust their respective protocol variables from the default state to the initiator's numbered value in response to a compression request. In this implementation, the initiator also adjusts its own protocol variables from the default state to its numbered value. Furthermore, after the initiator sends a compression cancellation request, it restores its own protocol variables to the default state. This approach ensures consistency across all monitoring services' protocol variables and reduces the risk of errors during program design.

[0102] Figure 2c This implementation method is also adopted in the implementation method. Therefore, for the compression request sent by monitoring service 3, not only monitoring service 0, monitoring service 1, and monitoring service 2 will feedback a second response instruction to monitoring service 3 indicating that they do not agree to the compression, but monitoring service 4 will also feedback a second response instruction to monitoring service 3 indicating that they do not agree to the compression.

[0103] Since monitoring service 3 did not receive the four first response instructions, it determines that there is a monitoring service that is currently compressing metadata information. Therefore, it will send a compression cancellation request to each recipient, that is, monitoring service 3 will send a compression cancellation request to monitoring service 0, monitoring service 1, monitoring service 2 and monitoring service 4.

[0104] After a period of time, when monitoring service 4 completes the compression process, monitoring service 3 can continue to request compression. After the consent of each recipient, monitoring service 3 can execute the compression, and after the compression is completed, a compression cancellation request is sent to restore the agreement variables of each recipient to the default state.

[0105] See Figure 2dIn this implementation, since monitoring service 2 and monitoring service 3 fail, the number of surviving monitoring services is 3, that is, A=3, and the total number of monitoring services is 5, that is, B=5, 3<5 / 2+1. Therefore, after monitoring service 4 determines that the disk space occupied by the metadata information it manages exceeds the preset space occupation threshold, it will not send a compression request because A>(B / 2+1) does not hold.

[0106] In a specific embodiment of the present invention, it also includes:

[0107] When the protocol variable itself is not in the default state for a first timeout period and no compression cancellation request is received from any monitoring service within the first timeout period, the protocol variable itself is restored to the default state.

[0108] In this embodiment, a first timeout period is set. For any monitoring service, if the protocol variables of that monitoring service are changed, and no compression cancellation request is received from any monitoring service within the first timeout period to restore the protocol variables to the default state, the monitoring service can automatically restore its own protocol variables to the default state. This embodiment can avoid situations where the protocol variables of various monitoring services cannot be restored to the default state due to abnormal circumstances. For example, if a monitoring service fails after compressing metadata information and does not send a compression cancellation request, the remaining monitoring services will be unable to compress metadata information.

[0109] The specific value of the first timeout period can be set as needed. For example, considering that a typical compression process does not exceed 5 minutes, the first timeout period can be set to 5 minutes or slightly longer than 5 minutes.

[0110] In a specific embodiment of the present invention, it may further include:

[0111] After exiting the distributed cluster due to an exception, rejoin the distributed cluster to restore to a surviving state when the exception is resolved.

[0112] In this implementation, it is allowed to exit the monitoring service of the distributed cluster due to an exception, and rejoin the distributed cluster after the exception is eliminated, which is beneficial to ensuring the performance of the distributed cluster.

[0113] In a specific embodiment of the present invention, when executing step S103, after determining that a monitoring service currently compressing metadata information exists, the following steps may also be performed:

[0114] Set its own compression flag to its own number value;

[0115] Among them, for any monitoring service, the compression flag of the monitoring service is automatically reduced according to the set rules until it is reduced to the set target value; the number values of each monitoring service are different, and the number value of any monitoring service is not less than the target value; for any monitoring service, when the compression flag of the monitoring service is not the target value, the monitoring service is prohibited from triggering the metadata compression process.

[0116] In this embodiment, to prevent a monitoring service from constantly determining whether there is a monitoring service currently compressing metadata information, after the monitoring service determines that there is a monitoring service currently compressing metadata information, the monitoring service will set its own compression flag, that is, the monitoring service will set its own no_compact_tick to its own number value;

[0117] The compression flag of a monitoring service will automatically decrease according to a set rule until it reaches a set target value. For example, the set rule is to decrease by 1 every minute, with a target value of 0. In addition, in this embodiment, the number value of each monitoring service is different, so this embodiment can reduce the probability of different monitoring services initiating compression at the same time.

[0118] For example, monitoring service 2 is currently performing compression, while monitoring service 3 and monitoring service 4 simultaneously initiate compression. As can be seen from the above description, since the solution of this application only allows one monitoring service to perform compression at the same time, both monitoring service 3 and monitoring service 4 will determine that there is a monitoring service currently performing compression of metadata information, and monitoring service 3 sets its own compression flag to its own number value 3, and monitoring service 4 sets its own compression flag to its own number value 4. After monitoring service 2 subsequently completes compression, since the compression flag of monitoring service 3 is 3, which is lower than the compression flag 4 of monitoring service 4, the compression flag of monitoring service 3 is first reduced to 0, and monitoring service 3 initiates the compression request first, and the situation where monitoring service 3 and monitoring service 4 initiate compression at the same time will not occur again.

[0119] Furthermore, in a specific embodiment of the present invention, after step S104, the following steps may be further included:

[0120] Determine whether the disk space occupied by the metadata information managed by itself exceeds the space usage threshold;

[0121] If so, the compression flag thereof is set to a first value; wherein the first value is higher than the number value of any monitoring service.

[0122] In this implementation, it is considered that after some monitoring services compress the metadata information they manage, the disk space occupied by the metadata information may still exceed the space occupation threshold. This may be normal, or it may be caused by an abnormal situation such as a compression failure. Therefore, in this implementation, the monitoring service will set its own compression flag to a first value to prevent the monitoring service from performing compression within a certain period of time. The specific value of the first value can be set as needed, but should be set to a higher value. Therefore, in this implementation, the first value is higher than the number value of any one monitoring service.

[0123] Applying the technical solution provided by the embodiment of the present invention, the applicant has taken into consideration that in the traditional solution, whether the master monitor service issues a compression instruction to the slave monitor service to instruct it to compress, or each monitor service performs compression when it detects that its kv-store is too large, the service of the distributed cluster will become unavailable. This is because the monitor service cannot respond to any service request when compressing the kv-store, and the compression time may exceed the heartbeat time of the monitor service, causing the monitor service to be kicked out of the cluster. If more than half of the monitor services are not in the distributed cluster, the master monitor service cannot be elected, which in turn causes the distributed cluster to be unable to provide services to the outside world, that is, the service of the distributed cluster becomes unavailable.

[0124] In this regard, in the solution of this application, for any monitoring service, the monitoring service determines that the disk space occupied by the metadata information it manages exceeds the preset space occupation threshold, indicating that the monitoring service needs to compress the metadata information. However, in order to avoid the situation where the distributed cluster service is unavailable, this application will further determine whether A>(B / 2+1) is established, where A represents the number of currently surviving monitoring services in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster. When A>(B / 2+1) is established, that is, A-1>B / 2, that is, even if the monitoring service compresses the metadata information, the number of remaining surviving nodes is greater than half of the total number of monitoring services, so the distributed cluster service will not be unavailable. In addition, considering that in some situations there may be two or more monitoring services initiating requests at the same time, therefore, in the scheme of the present application, only one monitoring service is allowed to compress the metadata information at the same time, that is, in the scheme of the present application, not only is A>(B / 2+1) required to be established, but it is also necessary to determine whether there is a monitoring service in the distributed cluster that is currently compressing the metadata information. Only when there is no other monitoring service that is compressing the metadata information will the metadata information managed by itself be compressed, thereby further avoiding the unavailability of the distributed cluster service.

[0125] In summary, the solution of the present application can effectively compress the metadata of the distributed cluster, avoid the situation where the disk space of the node is exhausted, and at the same time avoid the situation where the service of the distributed cluster is unavailable.

[0126] Corresponding to the above method embodiment, an embodiment of the present invention further provides a metadata compression system for a distributed cluster, which can be referred to in correspondence with the above.

[0127] See also Figure 3 FIG. 1 is a schematic diagram of the structure of a metadata compression system for a distributed cluster in the present invention, which is applied to any monitoring service in a distributed cluster and includes:

[0128] The disk space occupancy determination module 301 is used to determine whether the disk space occupied by the metadata information managed by the monitoring service itself exceeds a preset space occupancy threshold. If the space occupancy threshold is exceeded, the first condition determination module 302 is triggered;

[0129] The first condition judgment module 302 is used to: judge whether A>(B / 2+1) is established, and if so, trigger the second condition judgment module;

[0130] The second condition judgment module 303 is used to: judge whether there is a monitoring service that is currently compressing metadata information in the distributed cluster, and if not, trigger the compression execution module;

[0131] The compression execution module 304 is used to compress the metadata information managed by itself;

[0132] A represents the number of currently active monitoring services in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster.

[0133] In a specific embodiment of the present invention, the second condition judgment module 303 is specifically configured to:

[0134] Use itself as the initiator and each surviving monitoring service in the distributed cluster as the receiver, and send a compression request to each receiver.

[0135] For any receiver that receives a compression request, when the receiver's protocol variable is in the default state, the receiver feeds back a first response instruction indicating that it agrees to the compression and sets the receiver's protocol variable to the initiator number value carried in the compression request; when the receiver's protocol variable is not in the default state, the receiver feeds back a second response instruction indicating that it disagrees with the compression;

[0136] When a first response instruction fed back by each receiving party is received, determining that no monitoring service currently compressing the metadata information exists; otherwise, determining that a monitoring service currently compressing the metadata information exists;

[0137] Accordingly, the metadata compression system of the distributed cluster further includes: a compression cancellation request sending module, which is used to:

[0138] After the second condition judgment module 303 determines that there is no monitoring service currently compressing metadata information, and the compression execution module 304 compresses the metadata information managed by itself, a compression cancellation request is sent to each receiving party;

[0139] After the second condition judgment module 303 determines that there is a monitoring service that is currently compressing metadata information, a compression cancellation request is sent to each receiving party;

[0140] Wherein, for any receiver that receives the compression cancellation request, the receiver restores its own protocol variables to the default state.

[0141] In a specific embodiment of the present invention, after the second condition determination module 303 sends a compression request to each recipient, the second condition determination module 303 is further configured to:

[0142] Set the initiator's own protocol variable to the initiator number value carried in the compression request;

[0143] Accordingly, after the compression cancellation request sending module sends the compression cancellation request to each recipient, the second condition judgment module 303 is further configured to:

[0144] Restore the initiator's own contract variables to their default state.

[0145] In a specific embodiment of the present invention, the second condition judgment module 303 is further configured to:

[0146] When the protocol variable itself is not in the default state for a first timeout period and no compression cancellation request is received from any monitoring service within the first timeout period, the protocol variable itself is restored to the default state.

[0147] In a specific embodiment of the present invention, a survival status update module is further included, which is used to:

[0148] After exiting the distributed cluster due to an exception, rejoin the distributed cluster to restore to a surviving state when the exception is resolved.

[0149] In a specific embodiment of the present invention, after the second condition determination module 303 determines that there is a monitoring service currently compressing metadata information, a compression flag adjustment module is further included, which is configured to:

[0150] Set its own compression flag to its own number value;

[0151] Among them, for any monitoring service, the compression flag of the monitoring service is automatically reduced according to the set rules until it is reduced to the set target value; the number values of each monitoring service are different, and the number value of any monitoring service is not less than the target value; for any monitoring service, when the compression flag of the monitoring service is not the target value, the monitoring service is prohibited from triggering the metadata compression process.

[0152] In a specific embodiment of the present invention, after the compression execution module 304 compresses the metadata information managed by itself, the compression flag adjustment module is further configured to:

[0153] Determine whether the disk space occupied by the metadata information managed by itself exceeds the space usage threshold;

[0154] If so, the compression flag thereof is set to a first value; wherein the first value is higher than the number value of any monitoring service.

[0155] Corresponding to the above method and system embodiments, embodiments of the present invention further provide a metadata compression device for a distributed cluster and a computer-readable storage medium, which can be referred to in correspondence with the above.

[0156] See Figure 4 , the metadata compression device of the distributed cluster may include:

[0157] Memory 401, used for storing computer programs;

[0158] The processor 402 is configured to execute a computer program to implement the steps of the metadata compression method for a distributed cluster in any of the above embodiments.

[0159] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the distributed cluster metadata compression method described in any of the above embodiments. The computer-readable storage medium herein includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0160] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0161] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0162] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the technical solutions and core concepts of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A metadata compression method for a distributed cluster, characterized in that: Applicable to any monitoring service in a distributed cluster, including: Determine whether the disk space occupied by the metadata information managed by the monitoring service itself exceeds a preset space occupation threshold; If the space occupancy threshold is exceeded, determine whether A>(B / 2+1); If so, determining whether there is a monitoring service currently compressing metadata information in the distributed cluster; If it does not exist, the metadata information managed by itself is compressed; Wherein, A represents the number of currently surviving monitoring services in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster; The determining whether there is a monitoring service currently compressing metadata information in the distributed cluster includes: Taking itself as the initiator and each currently surviving monitoring service in the distributed cluster other than itself as each receiver, a compression request is sent to each receiver; Wherein, for any one receiving party that receives the compression request, when the protocol variable of the receiving party is in the default state, the receiving party feeds back a first response instruction indicating that it agrees to the compression, and sets the protocol variable of the receiving party to the initiator number value carried in the compression request; when the protocol variable of the receiving party is not in the default state, the receiving party feeds back a second response instruction indicating that it does not agree to the compression; When the first response instruction fed back by each recipient is received, determining that no monitoring service currently compressing the metadata information exists; otherwise, determining that a monitoring service currently compressing the metadata information exists; Accordingly, the metadata compression method of the distributed cluster further includes: After determining that there is no monitoring service currently compressing metadata information and compressing the metadata information managed by itself, a compression cancellation request is sent to each receiving party; After determining that there is a monitoring service that is currently compressing the metadata information, a compression cancellation request is sent to each receiving party; Wherein, for any one receiving party that receives the compression cancellation request, the receiving party restores its own protocol variables to a default state.

2. The metadata compression method for a distributed cluster according to claim 1, characterized in that: After sending the compression request to each receiving party, the method further includes: Set the initiator's own protocol variable to the initiator number value carried in the compression request; Accordingly, after sending the compression cancellation request to each recipient, the following steps are also included: Restore the initiator's own contract variables to their default state.

3. The metadata compression method for a distributed cluster according to claim 1, characterized in that: Also includes: When the protocol variable itself is not in the default state for a first timeout period and no compression cancellation request is received from any monitoring service within the first timeout period, the protocol variable itself is restored to the default state.

4. The metadata compression method for a distributed cluster according to claim 1, characterized in that: Also includes: After exiting the distributed cluster due to an exception, rejoin the distributed cluster to restore to a surviving state when the exception is resolved.

5. The metadata compression method for a distributed cluster according to any one of claims 1 to 4, characterized in that: After determining that a monitoring service is currently compressing metadata information, the following steps are also included: Set its own compression flag to its own number value; Among them, for any monitoring service, the compression flag of the monitoring service is automatically reduced according to the set rules until it is reduced to the set target value; the number values of each monitoring service are different, and the number value of any monitoring service is not less than the target value; for any monitoring service, when the compression flag of the monitoring service is not the target value, the monitoring service is prohibited from triggering the metadata compression process.

6. The metadata compression method for a distributed cluster according to claim 5, characterized in that: After compressing the metadata information managed by itself, the method further includes: Determine whether the disk space occupied by the metadata information managed by itself exceeds the space occupation threshold; If so, the compression flag thereof is set to a first value; wherein the first value is higher than the number value of any monitoring service.

7. A metadata compression system for a distributed cluster, characterized in that: Applicable to any monitoring service in a distributed cluster, including: a disk space occupancy determination module, configured to determine whether the disk space occupied by the metadata information managed by the monitoring service itself exceeds a preset space occupancy threshold, and trigger a first condition determination module if the space occupancy threshold is exceeded; The first condition judgment module is used to: judge whether A>(B / 2+1) is established, and if so, trigger the second condition judgment module; The second condition judgment module is used to: judge whether there is a monitoring service that is currently compressing metadata information in the distributed cluster, and if not, trigger the compression execution module; The compression execution module is used to compress the metadata information managed by itself; Wherein, A represents the number of currently surviving monitoring services in the distributed cluster, and B represents the total number of monitoring services set in the distributed cluster; The second condition judgment module is specifically used for: Use itself as the initiator and each surviving monitoring service in the distributed cluster as the receiver, and send a compression request to each receiver. For any receiver that receives a compression request, when the receiver's protocol variable is in the default state, the receiver feeds back a first response instruction indicating that it agrees to the compression and sets the receiver's protocol variable to the initiator number value carried in the compression request; when the receiver's protocol variable is not in the default state, the receiver feeds back a second response instruction indicating that it disagrees with the compression; When a first response instruction fed back by each receiving party is received, determining that no monitoring service currently compressing the metadata information exists; otherwise, determining that a monitoring service currently compressing the metadata information exists; Accordingly, the metadata compression system of the distributed cluster further includes: a compression cancellation request sending module, which is used to: After the second condition judgment module determines that there is no monitoring service currently compressing the metadata information, and the compression execution module compresses the metadata information managed by itself, a compression cancellation request is sent to each receiving party; After the second condition judgment module determines that there is a monitoring service that is currently compressing the metadata information, a compression cancellation request is sent to each receiving party; Wherein, for any receiver that receives the compression cancellation request, the receiver restores its own protocol variables to the default state.

8. A metadata compression device for a distributed cluster, characterized in that: include: memory for storing computer programs; A processor, configured to execute a computer program to implement the steps of the metadata compression method for a distributed cluster as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the metadata compression method for a distributed cluster as claimed in any one of claims 1 to 6 are implemented.

Images