Disk management method, device, computer equipment and readable storage medium

Abstract

The present application relates to a disk management method, device, computer equipment and readable storage medium. Involving the field of distributed and cloud technology, the method includes: at the current detection time point, determining the reference water level of the disk corresponding to the node, the reference water level is less than or equal to the current physical water level of the disk at the current detection time point; when the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold, determining the target data to be deleted from the disk, and performing a data deletion operation to delete the target data from the disk; when the disk management policy is a write rejection policy and the reference water level is greater than a preset water level threshold, stopping the distributed service from writing messages to the disk. The use of this method can improve the stability of the distributed cluster.

Description

Technical Field

[0001] The present application relates to the field of distributed and cloud technologies, and in particular to a disk management method, apparatus, computer device, and readable storage medium. Background Art

[0002] With the continuous development of cloud technology, distributed clusters are increasingly being used. By deploying distributed services in each node in the distributed cluster and storing data in the disk corresponding to the node through the distributed services, data storage and reading can be achieved. Distributed services can be, but are not limited to, Kafka.

[0003] In the related art, since the capacity of the disk is limited and a full disk will cause the distributed service to be unavailable, the distributed service can delete expired data regularly. For example, Kafka can delete expired messages from the disk through a message aging mechanism.

[0004] However, when data traffic is high, the distributed service writes data to the disk at a high rate. If there is no time to clean up the data, the disk will be full, affecting the normal use of the distributed service and thus the stability of the distributed cluster. Summary of the invention

[0005] Based on this, it is necessary to provide a disk management method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the stability of a distributed cluster in response to the above technical problems.

[0006] On the one hand, the present application provides a disk management method, which is applied to a node in a distributed cluster, wherein a distributed service is deployed in the node, and the distributed service is used to write messages to the disk corresponding to the node. Each of the nodes is configured with policy configuration information, and the policy configuration information is used to determine the disk management policy adopted by the node, including: at a current detection time point, determining a reference water level of the disk corresponding to the node, wherein the reference water level is less than or equal to the current physical water level of the disk at the current detection time point; when the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold, determining target data to be deleted from the disk, and performing a data deletion operation to delete the target data from the disk; when the disk management policy is a reject write policy and the reference water level is greater than a preset water level threshold, stopping the distributed service from writing messages to the disk.

[0007] On the other hand, the present application also provides a disk management device, which belongs to a node in a distributed cluster, and a distributed service is deployed in the node. The distributed service is used to write messages to the disk corresponding to the node, and each of the nodes is configured with policy configuration information. The policy configuration information is used to determine the disk management policy adopted by the node, including: a water level determination module, which is used to determine the reference water level of the disk corresponding to the node at the current detection time point, and the reference water level is less than or equal to the current physical water level of the disk at the current detection time point; a data deletion module, which is used to determine the target data to be deleted from the disk when the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold, and execute a data deletion operation to delete the target data from the disk; a write rejection module, which is used to stop the distributed service from writing messages to the disk when the disk management policy is a write rejection policy and the reference water level is greater than the preset water level threshold.

[0008] On the other hand, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above disk management method when executing the computer program.

[0009] On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned disk management method are implemented.

[0010] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps in the above disk management method when executed by a processor.

[0011] The above-mentioned disk management method, apparatus, computer equipment, computer-readable storage medium and computer program product, when the disk management policy is a data deletion policy and the reference water level is greater than the preset water level threshold, determine the target data to be deleted from the disk, and perform a data deletion operation to delete the target data from the disk; when the disk management policy is a write-rejection policy and the reference water level is greater than the preset water level threshold, stop the distributed service from writing messages to the disk. Thus, through the policy configuration information, the policies required by the scenario can be flexibly configured to adapt to different business scenarios. Users can flexibly configure specific policies such as write-rejection policies or data deletion policies based on their own business characteristics to alleviate the problem of the disk being full and affecting the normal use of distributed services, and improve the stability of the distributed cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0013] FIG. 1A is a diagram of an application environment of a disk management method in some embodiments;

[0014] FIG1B is an architecture diagram of a non-intrusive system in some embodiments;

[0015] FIG2 is a schematic diagram of a flow chart of a disk management method in some embodiments;

[0016] FIG3 is a schematic diagram of deleting data in some embodiments;

[0017] FIG4 is a diagram showing the relationship between detection time points and deletion time points on a timeline in some embodiments;

[0018] FIG5 is a schematic diagram of various water levels of a disk in some embodiments;

[0019] FIG6 is a schematic diagram of a disk management method in some embodiments;

[0020] FIG7 is a block diagram of a disk management device in some embodiments;

[0021] FIG8 is a diagram of the internal structure of a computer device in some embodiments;

[0022] FIG. 9 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The disk management method provided in the embodiment of the present application can be applied to the application environment shown in FIG. 1A. The application environment includes a distributed cluster, and the distributed cluster includes multiple nodes (node). FIG. 1A only takes three nodes as an example but does not limit the number of nodes in the distributed cluster. The three nodes are nodes 1 to 3. Distributed services such as kafka and service management modules such as kafkaManager are deployed in each node. The service management module includes a configuration management module (ConfManager). The configuration management module is used to maintain policy configuration information. The policy configuration information is information used to determine the disk management policy adopted by the node. Each service management module is in the same machine, i.e., node, as the distributed service itself. The machine can be a virtual machine, a container, or an independent physical machine, etc. The distributed service and the service management module in the same node, i.e., the machine, interact through the local RPC (Remote Procedure Call, remote procedure call) communication protocol. In FIG. 1A, local synchronization refers to synchronizing the policy configuration information stored in the memory with the policy configuration information stored in the disk, and inter-node synchronization refers to synchronizing the policy configuration information between different nodes so that the policy configuration information in different nodes is consistent. The disk may be any disk, including but not limited to an OS (operating system) disk.

[0025] In FIG1A , by deploying a service management module in each node, the management of the disk corresponding to the node can be realized based on the service management module without intruding the distributed service. As shown in FIG1B , an architecture diagram of a non-intrusive system is shown. By deploying the same service management module in each node, the distributed cluster with distributed services deployed can be optimized without intruding the distributed cluster, and the control of the distributed cluster can be completed through a service management module.

[0026] As shown in Figure 1B, the service management module may include an anomaly detection module (AnomalyDetector), a quota control module (QuotaController), a partition selection module (PartitionSeletor), a log deletion module (LogCleaner), and an administrative proxy module (AdminProxy). The anomaly detection module is used to obtain disk index data to detect abnormal disk status, provide quasi-real-time disk usage index collection, and is responsible for timely discovering abnormal disk occupancy. When the disk occupancy is found to be abnormal (for example, exceeding the threshold set by the user), it will immediately send a notification to the controllers of various policies to trigger the calling process of the corresponding policy. The quota control module is used to execute the flow limiting strategy according to the quota, that is, the flow limit parameter Quota mentioned below. The partition selection module is used for intelligent partition selection, and dynamically selects some partition data that is most suitable for elimination for deletion. The log deletion module is responsible for executing specific data deletion tasks. The administrative proxy module is responsible for executing specific management commands such as configuration modification instructions (i.e., configuration modification requests) or deletion instructions (instructions for deleting log segment files). The administrative proxy module directly interacts with distributed clusters such as Kafka clusters.

[0027] Each node in the distributed cluster can respectively execute the disk management method of the present application. Taking any node as an example, the node is configured with policy configuration information, which is used to determine the disk management policy adopted by the node. The disk management policy is divided into a write rejection policy and a data deletion policy. At the current detection time point, the node obtains the current physical water level of the disk corresponding to the node at the current detection time point, and obtains the historical physical water level and historical logical water level of the disk corresponding to the node at the previous historical detection time point, determines the incremental water level of the current physical water level relative to the historical physical water level, and when the incremental water level represents an increase in the physical water level of the disk, determines the candidate water level according to the incremental water level and the historical logical water level, and determines the smaller one of the current physical water level and the candidate water level as the reference water level of the disk. When the reference water level is greater than the preset water level threshold, if the disk management policy is a data deletion policy, the node determines the target data to be deleted from the disk, and executes a data deletion operation to delete the target data from the disk; if the disk management policy is a write rejection policy and the reference water level is greater than the preset water level threshold, then if the write rejection switch of the distributed service is in the off state, the write rejection switch is set to the on state, so that the distributed service stops writing messages to the disk.

[0028] Among them, each node can be a terminal or a server. The node can also be a virtual machine or a container. The nodes can communicate with each other through the network. The terminal can be, but is not limited to, a desktop computer, a laptop computer, a smart phone, a tablet computer, an IoT device, and a portable wearable device. The IoT device can be a smart speaker, a smart TV, a smart air conditioner, a smart car device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, or a server cluster or distributed cluster composed of multiple physical servers, or a cloud server that provides cloud computing services. The cloud server is used to provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.

[0029] In some embodiments, as shown in FIG. 2 , a disk management method is provided, which is applied to a node in a distributed cluster, wherein a distributed service is deployed in the node, and the distributed service is used to write a message to a disk corresponding to the node, and each node is configured with policy configuration information, and the policy configuration information is used to determine the disk management policy adopted by the node. The method is applied to any node in the distributed cluster of FIG. 1A as an example for description, and includes the following steps 202 to 206. Among them:

[0030] Step 202: at the current detection time point, determine a reference water level of the disk corresponding to the node, the reference water level being less than or equal to the current physical water level of the disk at the current detection time point.

[0031] Among them, the distributed service can be but not limited to kafka, Apache Pulsar, Redpanda or Rochefort. Kafka is a high-performance and high-availability distributed message queue system. High availability is an indicator for system design. In the field of distributed clusters, high availability means that when some abnormal situations occur, the distributed cluster can still guarantee the service, that is, guarantee the normal provision of services to users. In Kafka, each partition contains one or more log segment files. Kafka log segment files are the basic units for Kafka to store and retrieve messages.

[0032] The physical water level is the real water level of the disk. The physical water level reflects the percentage of disk usage. For example, if the disk capacity is 100GB (Gigabyte) and the disk stores 80GB of data, the physical water level is 80%. The current physical water level refers to the real water level of the disk at the current detection time point.

[0033] The disk corresponding to a node can belong to the same machine as the node, or the disk corresponding to the node can be on another machine.

[0034] In some embodiments, the reference water level is less than or equal to the physical water level of the disk at the current detection time point. The reference water level can be the physical water level of the disk at the current detection time point. The physical water level refers to the real water level of the disk. The water level is the occupancy ratio of the disk. The physical water level represents the ratio of the disk capacity occupied by data in the disk. For example, if the physical water level is 80%, it means that 80% of the disk capacity is occupied by data. The reference water level can also be determined based on the historical water level of the disk.

[0035] Step 204 , when the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold, determine the target data to be deleted from the disk, and perform a data deletion operation to delete the target data from the disk.

[0036] Specifically, the disk management policy is divided into a data deletion policy and a write rejection policy. In the case where the disk management policy is a data deletion policy, if the reference water level is greater than a preset water level threshold, the target data to be deleted is determined from the disk, and a data deletion operation is performed to delete the target data from the disk.

[0037] Step 206: When the disk management policy is a write-rejection policy and the reference water level is greater than a preset water level threshold, stop the distributed service from writing messages to the disk.

[0038] Specifically, when the disk management policy is a data deletion policy, if the reference water level is greater than a preset water level threshold, the distributed service is controlled to refuse to write messages to the disk.

[0039] In some embodiments, the policy configuration information can be modified flexibly. When the policy configuration information needs to be modified, the control plane service can send a policy modification request to each node in the distributed cluster (or the distributed management module in the node), and the node (or the distributed management module in the node) responds to the policy modification request, modifies the recorded policy configuration information, that is, synchronously updates the policy configuration information in the memory and the policy configuration information in the disk, and notifies the control plane service that the modification is successful when the modification is confirmed to be successful. The control plane service determines that the modification is completed when it determines that each node has been successfully modified. Among them, the control plane service is a service on the control device, and the control device can be a terminal or a server. For example, the control device can be a terminal device of a user who has the right to use the distributed cluster.

[0040] In some embodiments, for each node, in the case of a modification failure, the node or the distributed management module in the node can notify the control plane service of the modification failure. When the control plane service determines that the modification of any node fails, it sends a rollback request to the node that has been successfully modified, so that the node performs a compensating rollback operation and restores the management policy to the policy configuration information before the modification, thereby ensuring that the policy configuration information in each node is consistent and avoiding the problem of inconsistent policy configuration information in abnormal situations.

[0041] In some embodiments, since the modification failure may be caused by a network connection failure, in order to avoid multiple rollbacks or rollback failures when the network is unstable, therefore, when it is determined that the node modification failure is caused by a network connection failure, the control plane service can repeatedly send a policy modification request to the node where the modification failed, that is, perform a retry operation, so that the node performs the operation of modifying the policy configuration information again. If the number of repeated policy modification requests reaches a preset number threshold, but all modifications fail, the control plane service can send a rollback request to the node where the modification is successful. The number threshold can be set as needed, for example, 5 times or 8 times. Among them, the retry operation can be a backOff retry, and the backOff retry is a strategy: when the request fails, it is not retried immediately, but waits for a period of time before retrying, and the waiting time will gradually increase until the request succeeds or reaches the maximum number of retries. Of course, the retry operation can be an immediate retry after the request fails.

[0042] In some embodiments, the control plane service can determine each node in the distributed cluster that is in a normal state, and only modify the policy for each node in a normal state, while for nodes that cannot work normally, such as downtime nodes, they do not participate in the policy modification. Thus, if a node in the distributed cluster is down, and the node in a normal state during the downtime may have modified the policy configuration information at least once, when the downtime node is started or restored to normal, the node can synchronize the policy configuration information with other nodes. Other nodes refer to normal nodes other than the node in the distributed cluster. Specifically, the node can traverse other nodes and request the effective policy configuration information in the other nodes from other nodes. If the effective policy configuration information in the other nodes is not successfully requested, the next other node is requested for the effective policy configuration information in the next other node, until the effective policy configuration information is successfully requested, the node can update its own policy configuration information according to the requested policy configuration information.

[0043] In some embodiments, if the node that has been started fails to request all other nodes, for example, the distributed cluster has just completed restarting, and the first node that was started failed to successfully request other nodes, then the node or the configuration management module in the node can read the policy configuration information from the persistent file on the local disk. The node or the configuration management module in the node will synchronously update the policy configuration information in the persistent file after each update of the policy configuration information in the memory, so that even if the distributed cluster as a whole is suddenly restarted, the behavior consistency between nodes can be guaranteed.

[0044] In some embodiments, if there is no persistent file in the local disk, for example, the distributed cluster has just been deployed and the nodes have not yet generated runtime data, the node or the configuration management module in the node can initialize a default policy configuration information.

[0045] In the above disk management method, when the disk management policy is a data deletion policy and the reference water level is greater than the preset water level threshold, the target data to be deleted is determined from the disk, and a data deletion operation is performed to delete the target data from the disk. When the disk management policy is a write rejection policy and the reference water level is greater than the preset water level threshold, the distributed service is stopped from writing messages to the disk. Thus, through the policy configuration information, the policies required by the scenario can be flexibly configured to adapt to different business scenarios. Users can flexibly configure specific policies such as write rejection policies or data deletion policies based on their own business characteristics to alleviate the problem of the disk being full and affecting the normal use of distributed services, thereby improving the stability of the distributed cluster.

[0046] The disk management method provided in this application can be applied to a Kafka cluster. Each node in the Kafka cluster is deployed with a Kafka service. This can reduce the situation where the Kafka service is unavailable due to disk data being full, and improve the stability of the distributed cluster.

[0047] In some embodiments, at the current detection time point, a reference water level of the disk corresponding to the node is determined, including: at the current detection time point, obtaining the current physical water level of the disk corresponding to the node at the current detection time point, and obtaining the historical physical water level and historical logical water level of the disk corresponding to the node at the previous historical detection time point; determining an incremental water level of the current physical water level relative to the historical physical water level, and determining a candidate water level based on the incremental water level and the historical logical water level when the incremental water level represents an increase in the physical water level of the disk; determining the smaller one of the current physical water level and the candidate water level as the reference water level of the disk; the method also includes: determining the current logical water level of the disk at the current detection time point based on the target data volume of the target data and the reference water level.

[0048] The node can periodically detect the disk. The previous historical detection time point refers to a detection time point before the current detection time point and closest to the current detection time point.

[0049] The physical water level is the real water level of the disk. The physical water level reflects the percentage of disk usage. For example, if the disk capacity is 100GB (Gigabyte) and the disk stores 80GB of data, the physical water level is 80%. The current physical water level refers to the real water level of the disk at the current detection time point. The historical physical water level is the real water level of the disk at the last historical detection time point. The historical logical water level is determined in the same way as the current detection time point, except that it is calculated at different detection time points.

[0050] Incremental water level = current physical water level - historical physical water level. Since data may be deleted or written to the disk during the period from the historical detection time point to the current detection time point, the incremental water level may be positive, zero, or negative. If the incremental water level is positive, it indicates that the physical water level of the disk has risen.

[0051] Specifically, when the incremental water level is a positive number, the node adds the incremental water level to the historical logical water level to obtain the candidate water level. When the incremental water level is not an integer, the historical logical water level is used as the candidate water level. If the current physical water level is less than or equal to the candidate water level, the current physical water level is used as the reference water level; if the current physical water level is greater than the candidate water level, the candidate water level is used as the reference water level.

[0052] In some embodiments, when the reference water level is greater than a preset water level threshold, the target data to be deleted is determined from the disk, the current logical water level of the disk at the current detection time point is determined based on the target data volume of the target data and the reference water level, and a data deletion operation is performed to delete the target data from the disk.

[0053] In some embodiments, the service management module in the node can delete the target data from the disk corresponding to the node through the distributed service in the node, such as Kafka's own data deletion interface (an interface for deleting data, i.e., messages), so deleting data from the disk is asynchronous. Executing a data deletion operation can be an operation of calling a data deletion interface. Calling the data deletion interface does not mean that the data is deleted immediately. Therefore, at the next future detection time point, the data that needs to be deleted at the current detection point may not be completely deleted. The preset water level threshold of the disk can be expressed as a percentage of the occupied disk capacity. For example, the preset water level threshold is 90%, and 90% refers to 90% of the disk capacity.

[0054] In some embodiments, the node can calculate the ratio of the target data volume to the capacity of the disk to obtain the water level drop, and the result obtained by subtracting the water level drop from the reference water level is determined as the current logical water level of the disk at the current detection time point. For example, if the target data volume is 10GB and the capacity of the disk is 100GB, the water level drop is 10%. If the reference water level is 92%, the current logical water level is 82%.

[0055] Taking the first detection time point, the second detection time point and the third detection time point as examples, since the first detection time point does not have the previous historical detection time point, at the first detection time point, the node can obtain the physical water level 1 of the disk at the first detection time point, and set the reference water level 1 of the disk at the first detection time point to be consistent with the physical water level 1, that is, reference water level 1 = physical water level 1. If the preset water level threshold is 90%, the physical water level 1 = reference water level 1 is 92%. Since the reference water level 1 is greater than the preset water level threshold, the data needs to be deleted. If it is determined that 10% of the data needs to be deleted, that is, the water level drop is 10%, then the logical water level of the disk at the first detection time point is 82%; at the second detection time point, if the physical water level 2 of the disk at the second detection time point is 95%, since the logical water level of the disk at the first detection time point is The reference water level is 82% and the physical water level is 92%. We can get the incremental water level = 95%-92% = 3%, and the candidate water level = 3%+82%=85%. Since 85% is less than 95%, the reference water level is 85%. Since 85% is less than the preset water level threshold of 90%, it is determined that there is no need to delete data at the second detection time point. The reasons why the physical water level 2 of the disk is 95% at the second detection time point include: (1) 10% of the data was deleted and 13% of the data was written during the period from the first detection time point to the second detection time point. (2) 1% of the data was deleted and 4% of the data was written during the period from the first detection time point to the second detection time point, which means that 9% of the data has not been deleted. For reason (2), since it is determined that there is no need to delete data at the second detection time point, too much data on the disk will not be deleted. After the 9% of undeleted data is deleted, the physical water level of the disk may be less than 90%. If the data deletion is determined based on the relationship between the physical water level and the preset water level threshold, since the physical water level of 95% is greater than 90%, it will be determined that data needs to be deleted further, which may easily lead to excessive data deletion.

[0056] As shown in Figure 3, the principle diagram of deleting data is shown. Specifically, when the detection time point arrives, the incremental water level is calculated. When the incremental water level is greater than 0, the historical logical water level is added to the incremental water level to obtain the candidate water level. When the incremental water level is ≤0, the historical logical water level is used as the candidate water level. Then, the smaller of the current physical water level and the candidate water level is used as the reference water level. Min is a function of taking the smaller one. When the reference water level is greater than the preset water level threshold, the target data and the target data volume are determined, and the water level represented by the target data volume is subtracted from the reference water level to obtain the current logical water level. The deletion instruction is executed to enable the distributed service to delete the target data from the disk. When the reference water level is ≤ the preset water level threshold, the detection is stopped and waits for the next detection time point to arrive, and the detection process is repeated at the next detection time point.

[0057] In this embodiment, the smaller one between the current physical water level and the candidate water level is determined as the reference water level of the disk, thereby alleviating the problem of excessive data deletion.

[0058] Since the service management module's intelligent message deletion mechanism relies on Kafka's own data deletion interface, the deletion of real disk data is asynchronous. After each cleaning strategy is triggered, the disk water level does not change immediately. When the next abnormal detection module checks, the disk water level may still be the same as the previous time, but at this time, enough data has been deleted at the last detection point, and data deletion is not required at this detection point. However, due to the delay of real disk data deletion, data deletion still occurs this time, which leads to the problem of excessive deletion of user data, which will have a bad impact on user use.

[0059] Since a large amount of data may be written between two adjacent detection time points, and the data will not be deleted immediately, the disk may be filled up in the gap between two adjacent detection time points, resulting in anomalies. As shown in Figure 4, the relationship between the detection time point and the deletion time point on the timeline is shown. The deletion time point refers to the time point when the data is actually deleted. It can be seen that the deletion time point may be between the current detection time point (the current detection time point) and the next detection time point (the next future detection time point). Of course, the deletion time point may also be after the next detection time point.

[0060] Based on this, in some embodiments, the disk management policy is a data deletion policy, and the method also includes: obtaining a preset water level upper limit of the disk, determining the difference between the preset water level upper limit and the current physical water level, and obtaining the water level difference; determining the time interval between the current detection time point and the next future detection time point; determining the flow limitation parameters of the disk based on the water level difference and the time interval; and using the flow limitation parameters to limit the rate at which distributed services write data to the disk.

[0061] Among them, the preset water level upper limit of the disk can be set as needed, and the preset water level upper limit is greater than the preset water level threshold. The preset water level upper limit and the preset water level threshold can be expressed as a percentage of the occupied disk capacity. For example, the preset water level upper limit is 97%, and the preset water level threshold is 90%. 97% refers to 97% of the disk capacity, and 90% refers to 90% of the disk capacity.

[0062] The next future detection time point refers to the first detection time point after the current detection time point. When the node periodically detects the disk, the time interval between the current detection time point and the next future detection time point is the duration of the detection cycle. The duration of the detection cycle can be set according to actual needs. For example, the duration of the detection cycle can be 15 seconds, 20 seconds, 1 minute or 5 minutes. The traffic limit parameter is used to limit the amount of data written to the disk by the distributed service per unit time, that is, to limit the rate at which the distributed service writes data to the disk.

[0063] Specifically, the node may determine the result obtained by subtracting the current physical water level from the preset water level upper limit as the water level difference. For example, curr represents the current physical water level, and LIMIT represents the preset water level upper limit, then the water level difference = LIMIT-curr.

[0064] In some embodiments, the node can use the ratio of the water level difference value to the time interval as the flow limit parameter of the disk. The node can limit the speed at which the distributed service writes data to the disk according to the flow limit parameter, that is, the rate at which the distributed service writes data to the disk is less than or equal to the flow limit parameter. For example, Quota represents the flow limit parameter, and nextCheckpointIntv represents the time interval between the current detection time point and the next future detection time point, and Quota = (LIMIT-curr) / nextCheckpointIntv. Then the rate at which the distributed service writes data to the disk is less than or equal to Quota.

[0065] In some embodiments, the anomaly detection module in the service management module of the node can calculate the flow limit parameter, i.e., the quota, and provide the flow limit parameter to the quota control module, which executes a specific flow limit strategy, i.e., limits the flow of distributed services written to the disk.

[0066] In this embodiment, the rate at which the distributed service writes data to the disk is limited by the flow limit parameter, so that during the period from the current detection time point to the next future detection time point, even if the distributed service writes data to the disk, the real water level of the disk will be less than the preset water level upper limit. At the next future detection time point, the real water level of the disk is less than the preset water level upper limit, so that the water level of the disk is always below the preset water level upper limit and is always at a safe water level. For example, as shown in Figure 5, a schematic diagram of each water level of the disk is shown. The future water level refers to the real water level of the disk at any time between the current detection time point and the next future detection time point. Since there is a rate limit, that is, the rate at which the distributed service writes data to the disk is limited, the future water level is consistently less than the preset water level upper limit, thereby ensuring the safety of the water level.

[0067] In some embodiments, determining target data to be deleted from the disk includes: obtaining a preset deletion water level, the deletion water level being greater than or equal to the difference between a preset water level upper limit of the disk and a preset water level threshold; determining the amount of data represented by the deletion water level based on the deletion water level and the capacity of the disk; determining the target data to be deleted from the disk based on the amount of data represented by the deletion water level, the difference between the target data amount of the target data and the data amount represented by the deletion water level being less than the data amount difference threshold.

[0068] The disk stores data according to partitions, and each partition of the disk can store one or more log segment files, and multiple means at least two. The messages written to the disk by the distributed service are stored in the log segment file. When deleting data, the log segment file can be deleted as the smallest unit. The data volume difference threshold is the size of a single log segment file. The size of each log segment file in the disk, that is, the data volume, can be the same or different. When the sizes of the log segment files are different, the data volume difference threshold can be the data volume of the log segment file with the largest data volume. LIMIT represents the preset water level upper limit, and threshold represents the preset water level threshold, then the deletion water level ≥ (LIMIT-threshold). For example, if the preset water level upper limit is 97% and the preset water level threshold is 90%, then the deletion water level is greater than or equal to 7%, for example, the deletion water level is 10%.

[0069] In some embodiments, the node may calculate the product of the deletion water level and the capacity of the disk to obtain the amount of data represented by the deletion water level. For example, if the deletion water level is 10% and the capacity of the disk is 100 GB (Gigabyte), the amount of data represented by the deletion water level is 10 GB.

[0070] In some embodiments, the target data volume of the target number is greater than or equal to the data volume represented by the deletion water level. Thus, the real water level of the disk after deleting the target data can be made as small as possible to be smaller than the preset water level threshold, so that when the next inspection time point arrives, even if there is data written between the current inspection time point and the next inspection time point, the real water level of the disk at the next inspection time point can be made as small as possible to have a certain gap with the preset water level upper limit, so as to avoid the real water level of the disk at each inspection time point being closer to the preset water level upper limit, and to avoid the restricted flow becoming smaller and smaller, thereby increasing the impact on the business. For example, in Figures 4 and 5, if 10% of the data is deleted at the deletion time point, and the preset water level upper limit-the preset water level threshold = 7%, and no new message is written between this inspection time point and the next inspection time point, then at the next inspection time point, the real water level of the disk is after the preset water level threshold, so that the flow limit parameter calculated each time will not become smaller and smaller, thereby reducing the adverse impact of flow restriction on the business.

[0071] In this embodiment, since the deletion water level is greater than or equal to the difference between the preset water level upper limit of the disk and the preset water level threshold, the ratio of the deleted target data to the capacity of the disk can be made close to the deletion water level, so that the actual water level of the disk after deleting the target data can be made as small as possible below the preset water level threshold, so that when the next inspection time point arrives, even if there is data written between the current inspection time point and the next inspection time point, the actual water level of the disk at the next inspection time point can be made to have a certain gap with the preset water level upper limit, so as to avoid the actual water level of the disk at each inspection time point being closer to the preset water level upper limit, and to avoid the restricted traffic becoming smaller and smaller, thereby preventing the impact on the business from becoming greater and greater.

[0072] In some embodiments, the target data includes at least one log segment file in the disk; based on the amount of data represented by the deletion watermark, the target data to be deleted is determined from the disk, including: selecting at least one log segment file from each log segment file in the disk in order of file creation time from earliest to latest.

[0073] The target data includes at least one target log file. The at least one target log file may be one target log file or multiple target log files, where multiple refers to at least two. The target data volume is the total data volume, which refers to the total amount of data of each target log file in the at least one target log file. Each log segment file has a file creation time, and the file creation time of the log segment file refers to the time when the log segment file is created.

[0074] In some embodiments, the size of each log segment file is the same. The node can determine the data volume of the log segment file, obtain the log segment data volume, calculate the ratio of the data volume represented by the deletion water level to the log segment data volume, and when the ratio is an integer, use the ratio as the number of log segment files to be deleted. For example, if the deletion water level is 10%, if the disk capacity is 100GB, the data volume represented by the deletion water level is 10%×100GB=10GB, and if the log segment data volume is 1GB, the number of log segment files to be deleted is 10GB / 1GB=10. If the ratio is not an integer, the ratio can be rounded up or down, and the integer obtained by the rounding operation is determined as the number of log segment files to be deleted. The number of log segment files to be deleted is recorded as the number of deletions. The node can select the number of log segment files to be deleted from each log segment file on the disk in the order of file creation time from earliest to latest. Taking the number of deletions as 5 as an example, 5 log segment files created relatively early are selected from the disk.

[0075] In some embodiments, if the number of log segment files is inconsistent, the node can traverse the log segment files in the disk in the order of file creation time from earliest to latest, and for the traversed log segment file, sum the data volume of the log segment file and the log segment files whose file creation time is earlier than the log segment file, and continue traversing when the sum result is less than the data volume represented by the deletion watermark. For example, when traversing to the second log segment file, sum the data volume of the first log segment file and the data volume of the second log segment file. If the sum result is less than the data volume represented by the deletion watermark, continue traversing to the third log segment file; when the sum result is equal to the data volume represented by the deletion watermark, the traversed log segment file and the log segment files whose file creation time is earlier than the traversed log segment file are deleted. Each log segment file of the log segment file is determined as the log segment file to be deleted; when the sum result is greater than the amount of data represented by the deletion water level, the traversed log segment file and the log segment files whose file creation time is earlier than the traversed log segment file can be determined as the log segment files to be deleted, or the log segment file traversed last time and the log segment files whose file creation time is earlier than the log segment file traversed last time can be determined as the log segment files to be deleted. For example, the third log segment file is currently traversed. If the total amount of data of the first to second log segment files is less than the amount of data represented by the deletion water level, and the total amount of data of the first to third log segment files is greater than the amount of data represented by the deletion water level, then the first log segment file and the second log segment file are used as the log segment files to be deleted.

[0076] In some embodiments, for each selected log segment file, the node can delete the log segment file from the partition to which the log segment file belongs in the disk. Specifically, the service management module in the node can delete the log segment file through a distributed service such as Kafka's own data deletion interface (an interface for deleting data, i.e., messages).

[0077] In this embodiment, at least one log segment file is selected from the log segment files in the disk in the order of the file creation time from earliest to latest, so that the log segment files created earlier can be deleted preferentially.

[0078] In some embodiments, the current logical water level of the disk at the current detection time point is determined based on the target data volume of the target data and the reference water level, including: taking the ratio of the target data volume of the target data to the capacity of the disk as the water level drop; and taking the difference between the reference water level and the water level drop as the current logical water level of the disk at the current detection time point.

[0079] Specifically, the node may use the ratio of the total data volume to the capacity of the disk as the water level drop. The water level drop may be expressed as a percentage, such as 5%.

[0080] In some embodiments, the node can use the difference between the reference water level and the water level drop as the current logical water level of the disk at the current detection time point. That is, reference water level - water level drop = current logical water level. For example, if the reference water level is 92% and the water level drop is 5%, the current logical water level is 87%.

[0081] In this embodiment, the ratio of the target data volume of the target data to the capacity of the disk is used as the water level drop amount, and the difference between the reference water level and the water level drop amount is used to determine the current logical water level of the disk at the current detection time point. The current logical water level represents the water level after deleting the target data. At the next future detection time point, the current logical water level can be used as a reference to determine whether the data needs to be deleted, so as to avoid excessive deletion of data on the disk.

[0082] In some embodiments, when the disk management policy adopted by the node is a write rejection policy, a write rejection switch is configured in the distributed service of the node; stopping the distributed service from writing messages to the disk includes: determining the state of the write rejection switch of the distributed service; when the state is closed and the reference water level is greater than a preset water level threshold, changing the state of the write rejection switch to an open state, so that the distributed service stops writing messages to the disk.

[0083] The write rejection policy is that the distributed service rejects writing data from the client to the disk. The distributed service can determine whether to reject writing data from the client to the disk based on the state of the write rejection switch. When the write rejection switch is on, the distributed service determines that it is necessary to reject writing data from the client to the disk; when the write rejection switch is off, the distributed service determines that it is allowed to write data from the client to the disk.

[0084] Each node stores policy configuration information in memory, and also stores policy configuration information in the disk corresponding to the node. When updating policy configuration information, the node or the configuration management module in the node can synchronously update the policy configuration information in memory and the policy configuration information on disk. The policy configuration information in memory may be lost due to restart, while the policy configuration information on disk still exists after restart.

[0085] In this embodiment, in some business scenarios, there may be a need to retain data from distributed services such as Kafka for a long time, such as some OLAP (Online Analytical Processing) scenarios, which often require the analysis of data from multiple days back. At this time, it is not suitable to frequently delete older data under high traffic, and it is more appropriate to use a write rejection policy. Through the policy configuration information, the policy required by the scenario can be flexibly configured to adapt to different business scenarios. Users can flexibly configure specific policies such as write rejection policies and data deletion policies based on their own business characteristics.

[0086] In some embodiments, the method also includes: when the state is on and the reference water level is less than or equal to the preset water level threshold, obtaining water level indication information of each other node in the distributed cluster except the node, the water level indication information of other nodes represents the size relationship between the current reference water level of the disk corresponding to the other node and the preset water level threshold of the disk; when each size relationship is that the current reference water level of the disk is less than or equal to the preset water level threshold of the disk, setting the write rejection switch of the distributed service to the off state.

[0087] The current reference water levels of the disks corresponding to other nodes refer to the most recently calculated reference water levels of other nodes.

[0088] Specifically, the distributed service has a message aging and deletion mechanism, that is, the distributed service can delete some old messages on the disk at set intervals. Through the message aging and deletion mechanism, the reference water level of the disk can be made lower than the preset water level threshold of the disk.

[0089] The disk management method provided in this application can be applied to the Kafka cluster. The core idea of ​​the write rejection strategy is to set a dynamic write rejection switch for Kafka. When the anomaly detection module detects that the disk capacity threshold exceeds the limit, it can send an RPC request to Kafka to turn on the write rejection switch. When Kafka receives the RPC request, it will reject the traffic written by the client. At the next checkpoint, if the current disk no longer exceeds the threshold, and the anomaly detection module requesting other nodes finds that other nodes no longer exceed the threshold, it will send an RPC request to Kafka to turn off the write rejection switch. It must wait until other nodes also meet the disk water level requirements before it can be revoked. Otherwise, if only a single node meets the water level requirements and other nodes still exceed the threshold, if the write rejection switch is turned off early, the disks corresponding to other nodes are prone to be filled with traffic and fail.

[0090] In this embodiment, the write rejection switch of the distributed service is set to the off state only when the current reference water level of the disk in each size relationship is less than or equal to the preset water level threshold of the disk, thereby ensuring that the disks corresponding to each node in the distributed cluster are not fully written.

[0091] In some embodiments, as shown in FIG6 , a schematic diagram of a disk management method is shown, including: editing a policy: the control plane service configures policy configuration information in a node; policy synchronization: the configuration management module (ConfManager) is responsible for receiving configuration modification requests sent by the control plane service, the configuration modification requests are used to modify policy configuration information, and is responsible for persisting the policy configuration information and ensuring that each node configuration behavior is consistent; timing policy: regular quasi-real-time monitoring of the reference water level of the disk, when the reference water level exceeds the preset water level threshold of the disk, the policy execution module corresponding to the configured disk management policy is called to restrict, such as deleting data or refusing to write data; write restriction: turn on the dynamic write rejection switch of a distributed service such as the Kafka kernel, and cancel the switch after ensuring that the water levels of all nodes in the distributed cluster are in a safe state; partition selection: taking the log segment file as the dimension, a batch of the most suitable log segment files for deletion are selected in a backpack-like algorithm; delayed deletion: maintaining the logical water level of the disk, and calculating the planned deletion data size according to the set ratio of the disk capacity, generating the corresponding physical deletion calculation, and finally submitting it to a distributed service such as the Kafka kernel for asynchronous deletion.

[0092] The disk management method provided in this application can provide distributed services for Kafka services, with flexible hot switching in multiple modes (i.e., the policy configuration information can be modified without restarting the distributed cluster), and a secure Kafka disk restriction protection strategy. Two different restriction protection strategies are provided for business scenarios with different data importance levels, solving the existing disk capacity problem of Kafka and ensuring the availability of Kafka disks.

[0093] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0094] Based on the same inventive concept, the embodiment of the present application also provides a disk management device for implementing the disk management method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more disk management device embodiments provided below can refer to the limitations of the disk management method above, and will not be repeated here.

[0095] In some embodiments, as shown in FIG. 7 , a disk management device is provided. The disk management device belongs to a node in a distributed cluster. A distributed service is deployed in the node. The distributed service is used to write a message to a disk corresponding to the node. Each node is configured with policy configuration information. The policy configuration information is used to determine the disk management policy adopted by the node, including: a water level determination module 702, a data deletion module 704, and a write rejection module 706, wherein:

[0096] The water level determination module 702 is used to determine the reference water level of the disk corresponding to the node at the current detection time point, and the reference water level is less than or equal to the current physical water level of the disk at the current detection time point.

[0097] The data deletion module 704 is used to determine the target data to be deleted from the disk and perform a data deletion operation to delete the target data from the disk when the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold.

[0098] The write rejection module 706 is used to stop the distributed service from writing messages to the disk when the disk management policy is the write rejection policy and the reference water level is greater than a preset water level threshold.

[0099] In some embodiments, the water level determination module 702 is also used to obtain, at the current detection time point, the current physical water level of the disk corresponding to the node at the current detection time point, and obtain the historical physical water level and historical logical water level of the disk corresponding to the node at the previous historical detection time point; determine the incremental water level of the current physical water level relative to the historical physical water level, and when the incremental water level represents an increase in the physical water level of the disk, determine the candidate water level based on the incremental water level and the historical logical water level; determine the smaller one of the current physical water level and the candidate water level as the reference water level of the disk; and determine the current logical water level of the disk at the current detection time point based on the target data volume of the target data and the reference water level.

[0100] In some embodiments, the device also includes a current limiting module, which is used to obtain a preset water level upper limit of the disk, determine the difference between the preset water level upper limit and the current physical water level, and obtain the water level difference; determine the time interval between the current detection time point and the next future detection time point; determine the flow limitation parameters of the disk based on the water level difference and the time interval; and use the flow limitation parameters to limit the rate at which the distributed service writes data to the disk.

[0101] In some embodiments, the data deletion module 704 is also used to obtain a preset deletion water level, where the deletion water level is greater than or equal to the difference between a preset water level upper limit of the disk and a preset water level threshold; based on the deletion water level and the capacity of the disk, the amount of data represented by the deletion water level is determined; based on the amount of data represented by the deletion water level, the target data to be deleted is determined from the disk, and the difference between the target data amount of the target data and the data amount represented by the deletion water level is less than the data amount difference threshold.

[0102] In some embodiments, the water level determination module 702 is also used to use the ratio of the target data volume of the target data to the capacity of the disk as the water level drop; and use the difference between the reference water level and the water level drop to determine the current logical water level of the disk at the current detection time point.

[0103] In some embodiments, when the disk management policy adopted by the node is a write rejection policy, a write rejection switch is configured in the distributed service of the node; the write rejection module 706 is also used to determine the state of the write rejection switch of the distributed service; when the state is closed and the reference water level is greater than the preset water level threshold, the state of the write rejection switch is changed to an open state so that the distributed service stops writing messages to the disk.

[0104] In some embodiments, the write rejection module 706 is also used to obtain water level indication information of each other node in the distributed cluster except the node when the state is on and the reference water level is less than or equal to the preset water level threshold. The water level indication information of other nodes represents the size relationship between the current reference water level of the disk corresponding to the other node and the preset water level threshold of the disk; when each size relationship is that the current reference water level of the disk is less than or equal to the preset water level threshold of the disk, the write rejection switch of the distributed service is set to the off state.

[0105] Each module in the above disk management device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0106] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the disk management method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a disk management method is implemented.

[0107] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a disk management method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0108] Those skilled in the art will understand that the structures shown in FIGS. 8 and 9 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

[0109] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above disk management method when executing the computer program.

[0110] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned disk management method are implemented.

[0111] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above disk management method when executed by a processor.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0113] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0115] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A disk management method, characterized in that: Applied to a node in a distributed cluster, a distributed service is deployed in the node, the distributed service is used to write a message to a disk corresponding to the node, each of the nodes is configured with policy configuration information, the policy configuration information is used to determine the disk management policy adopted by the node, the method includes: At the current detection time point, obtain the current physical water level of the disk corresponding to the node at the current detection time point, and obtain the historical physical water level and historical logical water level of the disk corresponding to the node at the previous historical detection time point, determine the incremental water level of the current physical water level relative to the historical physical water level, and in the case where the incremental water level represents a rise in the physical water level of the disk, determine the candidate water level according to the incremental water level and the historical logical water level, and determine the smaller one of the current physical water level and the candidate water level as the reference water level of the disk corresponding to the node, the reference water level is less than or equal to the current physical water level of the disk at the current detection time point, the current physical water level refers to the real water level of the disk at the current detection time point, and the historical physical water level is the real water level of the disk at the previous historical detection time point; In the case where the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold, target data to be deleted is determined from the disk, a data deletion operation of deleting the target data from the disk is performed, a ratio of the target data volume of the target data to the capacity of the disk is used as the water level drop amount, and a difference between the reference water level and the water level drop amount is determined as the current logical water level of the disk at the current detection time point; When the disk management policy is a write-rejection policy and the reference water level is greater than a preset water level threshold, the distributed service is stopped from writing messages to the disk.

2. The method according to claim 1, characterized in that: The method further comprises: Obtaining a preset water level upper limit of the disk, determining a difference between the preset water level upper limit and the current physical water level, and obtaining a water level difference; Determine the time interval between the current detection time point and the next future detection time point; determining a flow limit parameter of the magnetic disk according to the water level difference and the time interval; The flow limit parameter is used to limit the rate at which the distributed service writes data to the disk.

3. The method according to any one of claims 1 to 2, characterized in that: The step of determining target data to be deleted from the disk includes: Obtaining a preset deletion water level, where the preset deletion water level is greater than or equal to a difference between a preset water level upper limit of the disk and the preset water level threshold; Determining the amount of data represented by the preset deletion water level according to the preset deletion water level and the capacity of the disk; According to the data volume represented by the preset deletion water level, target data to be deleted is determined from the disk, and the difference between the target data volume of the target data and the data volume represented by the preset deletion water level is less than a data volume difference threshold.

4. The method according to claim 1, characterized in that: In the case where the disk management policy adopted by the node is a write-rejection policy, a write-rejection switch is configured in the distributed service of the node; The step of stopping the distributed service from writing a message to the disk includes: Determining a state of a write deny switch of the distributed service; When the state is the closed state and the reference water level is greater than a preset water level threshold, the state of the write rejection switch is changed to the open state, so that the distributed service stops writing messages to the disk.

5. The method according to claim 4, characterized in that The method further comprises: When the state is on and the reference water level is less than or equal to the preset water level threshold, obtain water level indication information of each other node in the distributed cluster except the node, the water level indication information of the other nodes represents the size relationship between the current reference water level of the disk corresponding to the other node and the preset water level threshold of the disk; In the case that the current reference water level of the disk is less than or equal to the preset water level threshold of the disk in each of the size relationships, the write rejection switch of the distributed service is set to an off state.

6. A disk management device, characterized in that: The disk management device belongs to a node in a distributed cluster, a distributed service is deployed in the node, the distributed service is used to write a message to a disk corresponding to the node, each of the nodes is configured with policy configuration information, the policy configuration information is used to determine the disk management policy adopted by the node, and the device includes: A water level determination module is used to obtain, at a current detection time point, a current physical water level of the disk corresponding to the node at the current detection time point, and obtain a historical physical water level and a historical logical water level of the disk corresponding to the node at a previous historical detection time point, determine an incremental water level of the current physical water level relative to the historical physical water level, and in the case where the incremental water level represents a rise in the physical water level of the disk, determine a candidate water level according to the incremental water level and the historical logical water level, determine the smaller of the current physical water level and the candidate water level as a reference water level of the disk corresponding to the node, the reference water level is less than or equal to the current physical water level of the disk at the current detection time point, the current physical water level refers to the actual water level of the disk at the current detection time point, and the historical physical water level is the actual water level of the disk at the previous historical detection time point; A data deletion module, used to determine the target data to be deleted from the disk and perform a data deletion operation to delete the target data from the disk when the disk management policy is a data deletion policy and the reference water level is greater than a preset water level threshold; the water level determination module is also used to use the ratio of the target data volume of the target data to the capacity of the disk as the water level drop amount, and to determine the difference between the reference water level and the water level drop amount as the current logical water level of the disk at the current detection time point; The write rejection module is used to stop the distributed service from writing messages to the disk when the disk management policy is the write rejection policy and the reference water level is greater than a preset water level threshold.

7. The device according to claim 6, characterized in that The device is also used for: Obtaining a preset water level upper limit of the disk, determining a difference between the preset water level upper limit and the current physical water level, and obtaining a water level difference; Determine the time interval between the current detection time point and the next future detection time point; determining a flow limit parameter of the magnetic disk according to the water level difference and the time interval; The flow limit parameter is used to limit the rate at which the distributed service writes data to the disk.

8. The device according to claim 6 or 7, characterized in that The data deletion module is further used for: Obtaining a preset deletion water level, where the preset deletion water level is greater than or equal to a difference between a preset water level upper limit of the disk and the preset water level threshold; Determining the amount of data represented by the preset deletion water level according to the preset deletion water level and the capacity of the disk; According to the data volume represented by the preset deletion water level, target data to be deleted is determined from the disk, and the difference between the target data volume of the target data and the data volume represented by the preset deletion water level is less than a data volume difference threshold.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images