Data management method of distributed storage system, electronic equipment and storage medium

Abstract

The invention provides a data management method of a distributed storage system, electronic equipment and a storage medium, and belongs to the technical field of data storage. According to the embodiment of the invention, under the condition that the storage cluster has the abnormal node, the multi-dimensional speed limit threshold value is determined, and whether the to-be-processed I / O data meets the data migration condition or not can be effectively judged according to the multi-dimensional speed limit threshold value and the to-be-processed data volume of the to-be-processed I / O data of the abnormal node; and sending the to-be-processed I / O data to a target hard disk of a target server under the condition that the to-be-processed I / O data is determined to meet the data migration condition. According to the embodiment of the invention, the to-be-processed I / O data is subjected to speed-limiting processing from multiple dimensions such as the cluster dimension, the server dimension and the hard disk dimension, so that the influence on foreground I / O processing caused by resource snatching generated by various I / O processing in a storage cluster can be effectively prevented, and the service experience of a user is effectively ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data management method, electronic equipment and storage medium of a distributed storage system. Background Art

[0002] The total read and write capabilities of a distributed storage system are limited and will be affected by factors such as network bandwidth and the number of read and write operations per second. Therefore, the limited I / O capabilities need to be reasonably allocated to ensure that important businesses can enjoy sufficient I / O capabilities, that is, to give priority to the transmission of urgent and important information or to provide higher quality services for specific users.

[0003] Currently, I / O requests are mainly divided into two categories: foreground I / O requests and background I / O requests. The processing effect of foreground I / O requests affects the service quality provided by the cluster to the client, while the processing effect of background I / O requests affects the reliability of the cluster.

[0004] When a node in a distributed storage cluster is abnormal, the data stored in each node will be inconsistent, so data repair is required. When a large amount of data in the cluster needs to be restored, a large number of background I / O requests need to be processed, which may cause the cluster to have no resources to process foreground I / O requests, thus affecting the customer's service experience.

[0005] In related technologies, each node is usually assigned a corresponding data recovery speed limit value based on the current maximum amount of restored data in the distributed storage cluster. However, this method only limits the speed of background I / O requests from the cluster dimension, resulting in an inability to effectively coordinate front-end and back-end I / O requests, which in turn affects the user experience. Summary of the invention

[0006] The present invention provides a data management method, electronic device and storage medium of a distributed storage system, so as to solve the problem that it is difficult to effectively manage the background I / O requests of the distributed storage system at present.

[0007] In order to solve the above problems, the present invention adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a data management method for a distributed storage system, the method comprising:

[0009] In the case of an abnormal node in the storage cluster, a multi-dimensional speed limit threshold is determined; the multi-dimensional speed limit threshold includes a cluster dimension speed limit threshold, a server dimension speed limit threshold and a hard disk dimension speed limit threshold;

[0010] Determining whether the I / O data to be processed meets a data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold;

[0011] When it is determined that the I / O data to be processed meets the data migration condition, the I / O data to be processed is sent to the target hard disk of the target server.

[0012] In one embodiment of the present invention, the server dimension speed limit threshold includes a source server speed limit threshold of the source server and a target server speed limit threshold of the target server;

[0013] The hard disk dimension speed limit threshold includes a source hard disk speed limit threshold of a source hard disk and a target hard disk speed limit threshold of the target hard disk.

[0014] In one embodiment of the present invention, determining a multi-dimensional speed limit threshold includes:

[0015] Determining the cluster dimension speed limit threshold based on the first I / O processing capability and resource allocation ratio of the storage cluster;

[0016] Determining a speed limit threshold of the source server based on the second I / O processing capacity of the source server and the resource allocation ratio;

[0017] Determining a speed limit threshold of the target server based on the third I / O processing capability of the target server and the resource allocation ratio;

[0018] Determining a speed limit threshold of the source hard disk based on the fourth I / O processing capability of the source hard disk and the resource allocation ratio;

[0019] The target hard disk speed limit threshold is determined based on the fifth I / O processing capability of the target hard disk and the resource allocation ratio.

[0020] In one embodiment of the present invention, the method further includes:

[0021] The resource allocation ratio is determined based on the data access frequency of the I / O data to be processed.

[0022] In one embodiment of the present invention, the data migration condition includes a cluster migration condition, a server migration condition and a hard disk migration condition;

[0023] Determining whether the I / O data to be processed meets a data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold includes:

[0024] Determining whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of I / O data in transit of the storage cluster, and the cluster dimension speed limit threshold;

[0025] Determining whether the I / O data to be processed meets the server migration condition based on the amount of data to be processed, a second amount of I / O data in transit of the source server, a third amount of I / O data in transit of the target server, and the server dimension speed limit threshold;

[0026] Based on the amount of data to be processed, the fourth amount of I / O data in transit of the source hard disk, the fifth amount of I / O data in transit of the target hard disk, and the hard disk dimension speed limit threshold, determine whether the I / O data to be processed meets the hard disk migration condition.

[0027] In one embodiment of the present invention, determining whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of I / O data in transit of the storage cluster, and the cluster dimension rate limit threshold includes:

[0028] Determine a first total amount of data between the to-be-processed data amount and the first data amount;

[0029] When the first total amount of data is less than or equal to the cluster dimension speed limit threshold, determining that the to-be-processed I / O data meets the cluster migration condition;

[0030] Determining whether the I / O data to be processed meets the server migration condition based on the amount of data to be processed, a second amount of I / O data in transit of the source server, a third amount of I / O data in transit of the target server, and the server dimension speed limit threshold includes:

[0031] Determine a second total amount of data between the to-be-processed data amount and the second data amount, and a third total amount of data between the to-be-processed data amount and the third data amount;

[0032] When the second total amount of data is less than or equal to the source server speed limit threshold and the third total amount of data is less than or equal to the target server speed limit threshold, determining that the to-be-processed I / O data meets the server migration condition;

[0033] Determining whether the I / O data to be processed meets the hard disk migration condition based on the amount of data to be processed, a fourth amount of I / O data in transit of the source hard disk, a fifth amount of I / O data in transit of the target hard disk, and the hard disk dimension speed limit threshold includes:

[0034] Determine a fourth total amount of data between the to-be-processed data amount and the fourth data amount, and a fifth total amount of data between the to-be-processed data amount and the fifth data amount;

[0035] When the fourth total amount of data is less than or equal to the source hard disk speed limit threshold and the fifth total amount of data is less than or equal to the target hard disk speed limit threshold, it is determined that the I / O data to be processed meets the hard disk migration condition.

[0036] In one embodiment of the present invention, the method further includes:

[0037] When it is determined that the I / O data to be processed meets the cluster migration condition, the server migration condition and the hard disk migration condition, the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount are updated based on the amount of data to be processed.

[0038] In one embodiment of the present invention, the method further includes:

[0039] When it is determined that the to-be-processed I / O data does not satisfy the cluster migration condition, the server migration condition or the hard disk migration condition, triggering timing of a delayed sending time;

[0040] When the delayed sending duration reaches the duration threshold, the step of determining whether the I / O data to be processed meets the data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional speed limit threshold is performed.

[0041] In a second aspect, based on the same inventive concept, an embodiment of the present invention provides a data management device for a distributed storage system, the device comprising:

[0042] A threshold determination module is used to determine a multi-dimensional speed limit threshold when an abnormal node appears in the storage cluster; the multi-dimensional speed limit threshold includes a cluster dimension speed limit threshold, a server dimension speed limit threshold and a hard disk dimension speed limit threshold;

[0043] A condition determination module, configured to determine whether the I / O data to be processed meets a data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold;

[0044] The data sending module is used to send the I / O data to be processed to the target hard disk of the target server when it is determined that the I / O data to be processed meets the data migration condition.

[0045] In one embodiment of the present invention, the server dimension speed limit threshold includes a source server speed limit threshold of the source server and a target server speed limit threshold of the target server;

[0046] The hard disk dimension speed limit threshold includes a source hard disk speed limit threshold of a source hard disk and a target hard disk speed limit threshold of the target hard disk.

[0047] In one embodiment of the present invention, the threshold determination module includes:

[0048] A first threshold submodule, configured to determine the cluster dimension speed limit threshold based on a first I / O processing capability and a resource allocation ratio of the storage cluster;

[0049] A second threshold submodule, configured to determine a speed limit threshold of the source server based on a second I / O processing capability of the source server and the resource allocation ratio;

[0050] A third threshold submodule, configured to determine a speed limit threshold of the target server based on a third I / O processing capability of the target server and the resource allocation ratio;

[0051] A fourth threshold submodule, configured to determine a speed limit threshold of the source hard disk based on a fourth I / O processing capability of the source hard disk and the resource allocation ratio;

[0052] The fifth threshold submodule is used to determine the speed limit threshold of the target hard disk based on the fifth I / O processing capability of the target hard disk and the resource allocation ratio.

[0053] In one embodiment of the present invention, the data management device of the distributed storage system further includes:

[0054] The ratio determination module is used to determine the resource allocation ratio based on the data access frequency of the I / O data to be processed.

[0055] In an embodiment of the present invention, the data migration condition includes a cluster migration condition, a server migration condition and a hard disk migration condition; the condition determination module includes:

[0056] A first condition determination submodule, configured to determine whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, a first amount of I / O data in transit of the storage cluster, and the cluster dimension speed limit threshold;

[0057] A second condition determination submodule, configured to determine whether the I / O data to be processed meets the server migration condition based on the amount of data to be processed, a second amount of I / O data in transit of the source server, a third amount of I / O data in transit of the target server, and the server dimension speed limit threshold;

[0058] The third condition determination submodule is used to determine whether the I / O data to be processed meets the hard disk migration condition based on the amount of data to be processed, the fourth data amount of the in-transit I / O data of the source hard disk, the fifth data amount of the in-transit I / O data of the target hard disk, and the hard disk dimension speed limit threshold.

[0059] In one embodiment of the present invention, the first condition determination submodule includes:

[0060] A first total amount determining unit, used to determine a first total amount of data between the to-be-processed data amount and the first data amount;

[0061] A first condition determination unit, configured to determine that the to-be-processed I / O data satisfies the cluster migration condition when the first total amount of data is less than or equal to the cluster dimension speed limit threshold;

[0062] The second condition determination submodule includes:

[0063] A second total amount determining unit, configured to determine a second total amount of data between the to-be-processed data amount and the second data amount, and a third total amount of data between the to-be-processed data amount and the third data amount;

[0064] A second condition determination unit, configured to determine that the to-be-processed I / O data satisfies the server migration condition when the second total amount of data is less than or equal to the source server speed limit threshold and the third total amount of data is less than or equal to the target server speed limit threshold;

[0065] The third condition determination submodule includes:

[0066] A third total amount determining unit, configured to determine a fourth total amount of data between the to-be-processed data amount and the fourth data amount, and a fifth total amount of data between the to-be-processed data amount and the fifth data amount;

[0067] The third condition determination unit is used to determine that the to-be-processed I / O data meets the hard disk migration condition when the fourth total amount of data is less than or equal to the source hard disk speed limit threshold and the fifth total amount of data is less than or equal to the target hard disk speed limit threshold.

[0068] In one embodiment of the present invention, the data management device of the distributed storage system further includes:

[0069] A data update module is used to update the first data volume, the second data volume, the third data volume, the fourth data volume and the fifth data volume based on the amount of data to be processed when it is determined that the I / O data to be processed meets the cluster migration condition, the server migration condition and the hard disk migration condition.

[0070] In one embodiment of the present invention, the data management device of the distributed storage system further includes:

[0071] A timing module, configured to trigger timing for a delayed sending time when it is determined that the to-be-processed I / O data does not satisfy the cluster migration condition, the server migration condition or the hard disk migration condition;

[0072] A cyclic judgment module is used to execute the step of determining whether the I / O data to be processed meets the data migration condition based on the amount of to-be-processed data of the to-be-processed I / O data of the abnormal node and the multi-dimensional speed limit threshold when the delayed sending duration reaches the duration threshold.

[0073] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the data management method of the distributed storage system as described in the first aspect of the present invention.

[0074] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data management method for a distributed storage system as described in the first aspect of the present invention.

[0075] Compared with the prior art, the present invention has the following advantages:

[0076] A data management method for a distributed storage system provided by an embodiment of the present invention first determines a multi-dimensional speed limit threshold when an abnormal node appears in a storage cluster, and the multi-dimensional speed limit threshold includes a cluster dimension speed limit threshold, a server dimension speed limit threshold, and a hard disk dimension speed limit threshold; then, based on the amount of pending I / O data to be processed of the abnormal node and the multi-dimensional speed limit threshold, determines whether the pending I / O data meets the data migration condition; finally, when it is determined that the pending I / O data meets the data migration condition, the pending I / O data is sent to the target hard disk of the target server. The embodiment of the present invention comprehensively considers the I / O processing capabilities of the cluster, server, and hard disk, and performs speed limit processing on the pending I / O data from multiple dimensions such as the cluster dimension, the server dimension, and the hard disk dimension, which can effectively and fully improve the overall resource utilization of the system and prevent various I / O processing in the storage cluster from generating resource snatch and affecting the processing of the foreground I / O. In this way, when an abnormal node appears in the storage cluster, it can ensure that the foreground business has sufficient I / O capabilities and improve the user's service experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 It is a flow chart of the steps of a data management method of a distributed storage system in one embodiment of the present invention.

[0079] Figure 2 It is a flow chart of a data management method of a distributed storage system in one embodiment of the present invention.

[0080] Figure 3 It is a flow chart of judging data migration conditions in one embodiment of the present invention.

[0081] Figure 4 It is a schematic diagram of functional modules of a data management device of a distributed storage system in one embodiment of the present invention.

[0082] Figure 5 It is a schematic structural diagram of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] It should be noted that in a distributed storage system, the hardware resources of the data plane service nodes, such as CPU, memory, network, storage media and other resources, are all included in the entire system for scheduling and allocation, and provide external storage products such as blocks, files, and objects. Only by maximizing the use of hardware resources can we avoid waste of resources, thereby achieving the goal of controlling costs and providing a better customer experience. Under the premise of ensuring reliability and system performance, a reasonable and coordinated allocation of IO resources and limited system storage space can cope with various complex resource demand situations in the system and provide better services to the outside world.

[0085] In a distributed storage system, there are two main ways of distributing data: one is a central control node, which stores the route of data distribution, and the client determines the data plane service node and disk where the data shard is located through the routing information. Typical examples are Google's GFS (Google File System, a distributed file system) and big data storage HDFS (Distributed File System, a distributed file system). The other is a weak central control node, where the client determines the data plane service node and disk where the data shard is located through calculation, such as Ceph, an open source distributed storage system that is widely used in the industry. Regardless of the data distribution method, there will be roughly the following types of I / O in the distributed storage system:

[0086] ClientOp: read and write I / O requests sent by cluster clients. In most cases, the cluster should give priority to ensuring that such requests are processed in a timely manner.

[0087] SubOp: I / O requests between data plane nodes in the cluster. Mainly includes: read and write I / O requests between replicas and read and write I / O requests between EC shards generated by read and write I / O sent by cluster clients. Load balancing, data scanning, data synchronization and other operations will also cause this type of I / O requests.

[0088] SnapTrimOp: Deletion of cluster snapshot data will result in the generation of this type of I / O request.

[0089] ScrubOp: I / O requests generated by the scrub that discovers silent data errors in an object, including the scrub that only scans the object metadata and the deepscrub that scans the entire object.

[0090] RecoveryOp: I / O request for recovery migration caused by data rebalancing. Manual data rebalancing, failure or rejoining of data plane service processes, cluster expansion, data load balancing and other operations may all generate I / O requests. Among them, Recovery means data rebalancing. Adding nodes or hard disk data service processes in a distributed storage cluster, node offline, data service offline, and data load balancing will all cause data rebalancing. Recovery mainly repairs inconsistent data blocks in PG (PartitionedGlobal, placement group).

[0091] BackfillOp: Data backfilling. Adding nodes or hard disk data service processes in a distributed storage cluster, node offline, data service offline, and data load balancing will all cause data backfilling, and the data blocks in the PG will be copied as a whole. Among them, Backfill means data backfilling. Adding nodes or hard disk data service processes in a distributed storage cluster, node offline, data service offline, and data load balancing will all cause data backfilling. Backfill mainly copies the data blocks in the PG as a whole, which is a special recovery method.

[0092] It should be noted that PG is an internal data structure used by the cross-disk data service process to store data in a storage pool. Relative to the storage pool, PG is a virtual component, which is a virtual layer used when the Chunk (data block) is mapped to the storage pool. Among them, Chunk is a logical concept for storing user data, which is written in an append-write manner. The user's data may not be continuous in the data block. The index layer above the persistence layer records the location information of the user data in the data block.

[0093] The read and write capabilities of the total distributed storage system cluster are limited and will be affected by factors such as network bandwidth and the number of read and write operations per second. Therefore, it is necessary to reasonably allocate the limited I / O capabilities to ensure that important businesses can enjoy sufficient I / O capabilities, that is, to give priority to the transmission of urgent and important information or to provide higher quality services for specific users. The aforementioned I / O requests can be divided into two categories: foreground I / O requests, including ClientOp and SubOp generated by ClientOp; background I / O, including RecoveryOp, BackfillOP, ScrubOp and SnapTrimOp, etc. The foreground and background I / O request processing respectively ensure the high-quality services provided to customers and the reliability of the cluster. It is a common phenomenon that nodes in the distributed storage cluster go offline and abnormal downtime, which will cause inconsistent data blocks on each copy of the PG. It is necessary to repair the data and adopt the corresponding data balancing strategy to perform Recovery and Backfill, which will inevitably affect the services provided by the cluster to the client and may cause the cluster to be temporarily unavailable. Only by setting the data balance at a reasonable priority can the balance between availability and reliability be achieved.

[0094] In the related technology, in response to the above problems, the following three types of optimizations are currently being made: The first type is to assign a one-to-one data recovery speed limit value to each target storage node based on the current maximum amount of recovered data in the distributed storage system within a preset time length. The second type is to preset the maximum amount of recovered data, dynamically obtain the amount of recovered data of the PG of each data plane node within a certain time period, calculate the average value and compare it with the maximum recovery amount, adjust the recovery speed based on the result, and the central control node sends a message to adjust the recovery speed. The third type is the most typical type. In the data service of the data plane node, different types of sub-queues are set, mainly including priority queues and weight-based priority queues, classify I / O requests, design QoS templates for each type of I / O request, and control the order of I / O requests by weight.

[0095] The above three types of optimization adjust data recovery according to the amount of data to be restored according to the preset time. This is only based on historical experience. The data recovery speed is fast. Before the preset time is reached, the data in the cluster may have been restored, which leads to low data recovery efficiency and inability to accurately coordinate the front-end and back-end I / O. The weight-based queue control method has a small effect when the data plane node load is not high. Various I / O requests evenly divide the system's I / O resources, and also cannot accurately coordinate the front-end and back-end I / O. The above three types of methods are designed from the perspective of the overall recovery volume of the cluster at most. None of them considers the bandwidth load of the source node and the target node in data recovery, nor the bandwidth load of the source hard disk and the target hard disk, and even less considers the three at the same time in the limitation of the cluster background I / O. The third type of optimization only considers IOPS (Input / Output Operations per Second, which means the number of I / Os that can be processed per second), without considering the limitation of I / O bandwidth.

[0096] The problem that it is currently difficult to effectively manage the background I / O requests of the distributed storage system. The present invention aims to provide a data management method for a distributed storage system, which can effectively and fully improve the overall resource utilization of the system by limiting the speed of the I / O data to be processed from multiple dimensions such as the cluster dimension, the server dimension and the hard disk dimension, and prevent various I / O processes in the distributed storage cluster from generating resource snatch and affecting the foreground I / O processing. In this way, when an abnormal node appears in the storage cluster, it can ensure that the foreground business has sufficient I / O capacity and improve the user's service experience.

[0097] Reference Figure 1 , shows a flow chart of the steps of a data management method of a distributed storage system of the present invention, the method may include the following steps:

[0098] S101: When an abnormal node appears in a storage cluster, a multi-dimensional speed limit threshold is determined.

[0099] It should be noted that a distributed storage cluster is a system architecture that stores data in a dispersed manner on multiple physical or virtual nodes. It stores data in a dispersed manner on multiple nodes in multiple physical locations, which are connected together through a network.

[0100] In this embodiment, an abnormal node represents a node that generates pending I / O data. Specifically, data recovery, data migration, data scanning, data repair, and data balancing are all distributed reads and writes performed across nodes and disks by PG. Silent data error checking, manual data rebalancing, data plane service process failure or rejoining, cluster expansion, data load balancing and other operations may all generate PG migration, deletion, and read and write, all of which may generate pending I / O data.

[0101] In this embodiment, the multi-dimensional speed limit threshold includes a cluster dimension speed limit threshold, a server dimension speed limit threshold, and a hard disk dimension speed limit threshold. The cluster dimension speed limit threshold is a speed limit threshold set based on the I / O processing capability of the storage cluster; the server dimension speed limit threshold is a speed limit threshold set based on the I / O processing capability of the server; and the hard disk dimension speed limit threshold is a speed limit threshold set based on the I / O processing capability of the hard disk.

[0102] In a specific implementation, considering that the I / O processing capabilities of the source server, target server, source hard disk, and target hard disk are all limited, the server dimension speed limit threshold may specifically include the source server speed limit threshold of the source server and the target server speed limit threshold of the target server; the hard disk dimension speed limit threshold includes the source hard disk speed limit threshold of the source hard disk and the target hard disk speed limit threshold of the target hard disk. Among them, the source server speed limit threshold is a speed limit threshold set based on the I / O processing capability of the source server; the target server speed limit threshold is a speed limit threshold set based on the I / O processing capability of the target server; the source hard disk speed limit threshold is a speed limit threshold set based on the I / O processing capability of the source hard disk; and the target hard disk speed limit threshold is a speed limit threshold set based on the I / O processing capability of the target hard disk.

[0103] It should be noted that the source hard disk refers to the hard disk storing the I / O data to be processed; the source server refers to the server where the I / O data to be processed is located. That is to say, after an abnormal node appears in the storage cluster, the I / O data to be processed stored in the source hard disk of the source server will be migrated to the target hard disk of the target server.

[0104] S102: Determine whether the I / O data to be processed meets the data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold.

[0105] In this embodiment, since the I / O processing capabilities of the cluster, server and hard disk are all limited, by comprehensively considering the data volume relationship between the amount of data to be processed and the cluster dimension speed limit threshold, the server dimension speed limit threshold and the hard disk dimension speed limit, it is possible to effectively determine whether the I / O data to be processed meets the data migration conditions.

[0106] S103: When it is determined that the I / O data to be processed meets the data migration condition, the I / O data to be processed is sent to the target hard disk of the target server.

[0107] In this embodiment, if it is detected that the I / O data to be processed meets the data migration conditions, it means that the current cluster, server and hard disk have sufficient I / O processing capabilities to process the I / O data to be processed. At this time, the I / O data to be processed is allowed to be sent to the target hard disk of the target server.

[0108] In the specific implementation, if there is no in-transit I / O data on the target hard disk, that is, no other node transmits I / O data to the target hard disk, the I / O data to be processed will be directly sent to the target hard disk of the target server; if there is in-transit I / O data on the target hard disk, that is, other nodes have sent I / O data to the target hard disk before the abnormal node, the I / O data to be processed will be placed in the data transmission queue until the in-transit I / O data before the I / O data to be processed is transmitted, and then the I / O data to be processed will be transmitted to the target hard disk of the target server.

[0109] In this implementation, considering that the configurations of various servers in the distributed storage cluster are different, resulting in different I / O processing capabilities, by fully considering the processing capabilities of the cluster, server, and disk, the speed limit processing of the I / O data to be processed is performed through the three dimensions of cluster, server, and disk, so that the I / O processing capabilities of each server node can be used evenly, the waste of some server resources can be prevented, the overall resource utilization of the system can be fully improved, and resource grabbing caused by various I / O processing in the distributed storage cluster can be prevented, thereby affecting the processing of the foreground I / O. It is ensured that the foreground business can enjoy sufficient I / O capabilities and provide high-quality services to the upper-level business.

[0110] In a feasible implementation manner, the step of determining the multi-dimensional speed limit threshold in S101 may specifically include the following sub-steps:

[0111] S101 - 1 : Determine a cluster dimension speed limit threshold based on a first I / O processing capability and a resource allocation ratio of the storage cluster.

[0112] S101 - 2 : Determine a speed limit threshold of the source server based on the second I / O processing capability and resource allocation ratio of the source server.

[0113] S101 - 3 : Determine a speed limit threshold of the target server based on the third I / O processing capability and resource allocation ratio of the target server.

[0114] S101 - 4 : Determine a source hard disk speed limit threshold based on the fourth I / O processing capability and resource allocation ratio of the source hard disk.

[0115] S101 - 5 : Determine a target hard disk speed limit threshold based on the fifth I / O processing capability and resource allocation ratio of the target hard disk.

[0116] In this implementation, considering that there may be certain differences in the configurations of the storage cluster, source server, target server, source hard disk, and target hard disk, the multi-dimensional speed limit threshold is accurately calculated based on the I / O processing capabilities of the storage cluster, source server, target server, source hard disk, and target hard disk. It should be noted that the I / O processing capability refers to the ability to process input and output (I / O) requests per unit time.

[0117] In a specific implementation, the I / O processing capacity can be determined based on factors such as network bandwidth and hardware resources. Specifically, the first I / O processing capacity of the storage cluster can be determined based on the network bandwidth of each server in the storage cluster; the second I / O processing capacity of the source server and the third I / O processing capacity of the target server can be obtained by comprehensive evaluation based on the network bandwidth, CPU and memory of the source server and the target server respectively; the fourth I / O processing capacity of the source hard disk and the fifth I / O processing capacity of the target hard disk can be determined based on the hard disk read and write bandwidth.

[0118] In this embodiment, in order to determine the appropriate resource allocation ratio, the resource allocation ratio can be determined based on the data access frequency of the I / O data to be processed. Specifically, the higher the data access frequency, the higher the resource allocation ratio. In this way, for data with a high user access frequency, more I / O resources can be deployed to process the I / O data to be processed, thereby effectively meeting the user's data usage needs; and for data with a low user access frequency, fewer I / O resources are deployed to process the I / O data to be processed, thereby reserving more I / O resources for foreground I / O processing, thereby improving the user's foreground service experience.

[0119] In a feasible implementation manner, the data migration conditions include cluster migration conditions, server migration conditions, and hard disk migration conditions; S102 may specifically include the following sub-steps:

[0120] S102-1: Determine whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of I / O data in transit of the storage cluster, and the cluster dimension speed limit threshold.

[0121] It should be noted that the in-transit I / O data of the storage cluster refers to the I / O data that is in a transmission state in the storage cluster.

[0122] In this embodiment, the cluster dimension is first used to determine whether the target hard disk that sends the I / O data to be processed can be migrated. If the I / O data to be processed meets the cluster migration conditions, the server migration conditions and hard disk migration conditions are determined; if not, the sending of the I / O data to be processed is delayed.

[0123] In a specific implementation, the first total amount of data between the amount of data to be processed and the first amount of data can be determined; when the first total amount of data is less than or equal to the cluster dimension speed limit threshold, it is determined that the I / O data to be processed meets the cluster migration conditions.

[0124] In this embodiment, if it is detected that the total amount of the first data is greater than the cluster dimension speed limit threshold, it means that the storage cluster has no excess I / O resources to process the I / O data to be processed. At this time, the sending of the I / O data to be processed will be delayed until the total amount of the first data is less than or equal to the cluster dimension speed limit threshold.

[0125] S102-2: Determine whether the I / O data to be processed meets the server migration conditions based on the amount of data to be processed, the second amount of I / O data in transit of the source server, the third amount of I / O data in transit of the target server, and the server dimension speed limit threshold.

[0126] It should be noted that the in-transit I / O data of the target server refers to the I / O data that is in a transmission state in the target server.

[0127] In this implementation, after the I / O data to be processed meets the cluster migration condition, the server migration condition is further determined.

[0128] In a specific implementation, the total amount of the second data between the amount of data to be processed and the second data amount, and the total amount of the third data between the amount of data to be processed and the third data amount can be determined; when the total amount of the second data is less than or equal to the speed limit threshold of the source server and the total amount of the third data is less than or equal to the speed limit threshold of the target server, it is determined that the I / O data to be processed meets the server migration conditions.

[0129] In this embodiment, if it is detected that the total amount of the second data is greater than the speed limit threshold of the source server, it means that the source server has no extra I / O resources to process the I / O data to be processed. Similarly, if it is detected that the total amount of the third data is greater than the speed limit threshold of the target server, it means that the target server has no extra I / O resources to process the I / O data to be processed. At this time, the sending of the I / O data to be processed will be delayed. When the total amount of the second data is less than or equal to the speed limit threshold of the source server and the total amount of the third data is less than or equal to the speed limit threshold of the target server, it means that the source server can send the I / O data to be processed to the target server, and then determine that the I / O data to be processed meets the server migration conditions.

[0130] S102-3: Based on the amount of data to be processed, the fourth amount of I / O data in transit on the source hard disk, the fifth amount of I / O data in transit on the target hard disk, and the hard disk dimension speed limit threshold, determine whether the I / O data to be processed meets the hard disk migration conditions.

[0131] In this implementation, after the I / O data to be processed meets the cluster migration condition and the hard disk migration condition, the hard disk migration condition is finally determined.

[0132] In a specific implementation, the total amount of fourth data between the amount of data to be processed and the fourth amount of data, and the total amount of fifth data between the amount of data to be processed and the fifth amount of data can be determined; when the total amount of fourth data is less than or equal to the source hard disk speed limit threshold and the total amount of fifth data is less than or equal to the target hard disk speed limit threshold, it is determined that the I / O data to be processed meets the hard disk migration conditions.

[0133] In this embodiment, if it is detected that the total amount of the fourth data is greater than the speed limit threshold of the source hard disk, it means that the source hard disk has no extra I / O resources to process the I / O data to be processed. Similarly, if it is detected that the total amount of the fifth data is greater than the speed limit threshold of the target hard disk, it means that the target hard disk has no extra I / O resources to process the I / O data to be processed. At this time, the sending of the I / O data to be processed will be delayed. When the total amount of the fourth data is less than or equal to the speed limit threshold of the source hard disk and the total amount of the fifth data is less than or equal to the speed limit threshold of the target hard disk, it means that the source hard disk can send the I / O data to be processed to the target hard disk for storage, and then determine that the I / O data to be processed meets the hard disk migration conditions.

[0134] That is, if it is detected that the I / O data to be processed meets the cluster migration condition, the server migration condition and the hard disk migration condition at the same time, it means that the I / O data to be processed meets the data migration condition, and then the I / O data to be processed is sent to the target hard disk of the target server.

[0135] In this embodiment, by judging whether the I / O data to be processed meets the cluster migration conditions, server migration conditions and hard disk migration conditions in order from large to small granularity, it is possible to achieve precise restriction on the I / O data to be processed, and then achieve precise regulation of the I / O data to be processed.

[0136] In this embodiment, the data management method of the distributed storage system may further include the following steps: when it is determined that the I / O data to be processed does not meet the cluster migration conditions, the server migration conditions or the hard disk migration conditions, triggering the timing of the delayed sending time; when the delayed sending time reaches the time threshold, executing the step of determining whether the I / O data to be processed meets the data migration conditions based on the amount of to-be-processed data and the multi-dimensional speed limit threshold of the to-be-processed I / O data of the abnormal node.

[0137] In this embodiment, by cyclically detecting whether the I / O data to be processed meets the data migration conditions at every time threshold, the I / O data to be processed can be sent to the target hard disk of the target server in a timely manner when there are excess I / O resources in the system.

[0138] In this embodiment, the data management method of the distributed storage system may also include the following steps: when it is determined that the I / O data to be processed meets the cluster migration conditions, server migration conditions and hard disk migration conditions, based on the amount of data to be processed, updating the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount.

[0139] It should be noted that since each node in the storage cluster may generate pending I / O data, once it is detected that the pending I / O data of any abnormal node meets its corresponding data migration conditions, the background in-transit I / O data volume of the storage cluster, source server, target server, source hard disk, and target hard disk will be updated, that is, the pending data volume of the pending I / O data will be added.

[0140] It should be noted that after the background I / O of this migration is completed, the background in-transit I / O data volume of the storage cluster, source server, target server, source hard disk and target hard disk will also be updated, that is, the data volume of the background I / O of this migration will be subtracted.

[0141] Reference Figure 2 , shows a schematic flow chart of a data management method for a distributed storage system according to an embodiment of the present invention. Specifically, the following steps may be included:

[0142] S201: The system starts and determines the limit thresholds of the storage cluster, the source server, the target server, the source hard disk, and the target hard disk.

[0143] S202: An abnormal node appears in the storage cluster, generating I / O data to be processed.

[0144] S203: Check whether the I / O data to be processed exceeds the limit thresholds of the storage cluster, the source server, the target server, the source hard disk, and the target hard disk.

[0145] S204: Determine that the I / O data to be processed does not exceed the limit thresholds of the storage cluster, the source server, the target server, the source hard disk, and the target hard disk, and process the I / O data to be processed.

[0146] S205: After the processing of the pending I / O data is completed, the amount of in-transit I / O data of the storage cluster, the source server, the target server, the source hard disk, and the target hard disk is updated.

[0147] In one example, taking the node abnormal downtime scenario as an example, after the node abnormal downtime, the PG on the node belongs to the replica pool. The PG on the node lacks a replica, so a new node must be selected to migrate data from other replicas of the PG to the new node. The migration of the PG will generate background I / O. The information of the PG migration includes the source server, source hard disk information, target server, target hard disk, migration data volume and other information. Each time data migration is performed, the sum of the pending data volume of the I / O data to be processed for this migration and the data volume of the in-transit background I / O of the storage cluster, source server, target server, source hard disk and target hard disk will be calculated to see whether it exceeds the cluster background I / O speed limit threshold, the source server background I / O speed limit threshold, the target server background I / O speed limit threshold, the source hard disk background I / O speed limit threshold, and the target hard disk background I / O speed limit threshold. If any speed limit threshold is exceeded, the migration will enter the waiting state and wait for the preset time to check again. If all speed limit thresholds are not exceeded, the in-transit background I / O data volume of the storage cluster, source server, target server, source hard disk, and target hard disk is updated, and the pending I / O data volume of this migration is increased to the original in-transit background I / O data volume. After the background I / O of this migration is completed, the data volume of this migration is subtracted to update the in-transit I / O data volume of the storage cluster, source server, target server, source hard disk, and target hard disk.

[0148] Reference Figure 3 , shows a flow chart of judging data migration conditions. Specifically, the following steps may be included:

[0149] S301: Determine whether the sum of the amount of data to be processed and the amount of I / O data in transit in the storage cluster exceeds the cluster dimension speed limit threshold; if so, proceed to S302; if not, proceed to S303.

[0150] S302: trigger timing for delayed sending duration to determine whether the delayed sending duration reaches a duration threshold; if the delayed sending duration reaches the duration threshold, proceed to S301.

[0151] S303: Determine whether the sum of the amount of data to be processed and the amount of I / O data in transit on the source server exceeds the speed limit threshold of the source server; if so, proceed to S302; if not, proceed to S304.

[0152] S304: Determine whether the sum of the amount of data to be processed and the amount of I / O data in transit on the target server exceeds the target server speed limit threshold; if so, proceed to S302; if not, proceed to S305.

[0153] S305: Determine whether the sum of the amount of data to be processed and the amount of I / O data in transit on the source hard disk exceeds the source hard disk speed limit threshold; if so, proceed to S302; if not, proceed to S306.

[0154] S306: Determine whether the sum of the amount of data to be processed and the amount of I / O data in transit on the target hard disk exceeds the target hard disk speed limit threshold; if so, proceed to S302; if not, proceed to S307.

[0155] S307: According to the amount of data to be processed, increase the amount of in-transit I / O data of the storage cluster, the source server, the target server, the source hard disk, and the target hard disk.

[0156] In the implementation of the present invention, by fully considering the I / O processing capabilities of the storage cluster, source server, target server, source hard disk and target hard disk, and limiting the speed of background I / O processing in three dimensions of cluster, server and disk, the impact of the distributed storage system on the foreground I / O processing can be reduced, and resource grabbing caused by various types of I / O processing in the distributed storage cluster can be prevented, thereby further improving the utilization rate of system resources, ensuring that the foreground business can enjoy sufficient I / O capabilities, providing high-quality services to the outside world, achieving a balance between availability and reliability, achieving a balance between service quality and system reliability, and ensuring the smooth operation of the distributed storage system.

[0157] In the second aspect, based on the same inventive concept, refer to Figure 4 The embodiment of the present invention provides a data management device 400 for a distributed storage system, and the data management device 400 for a distributed storage system includes:

[0158] The threshold determination module 401 is used to determine the multi-dimensional speed limit threshold when an abnormal node appears in the storage cluster; the multi-dimensional speed limit threshold includes a cluster dimension speed limit threshold, a server dimension speed limit threshold and a hard disk dimension speed limit threshold;

[0159] A condition determination module 402, configured to determine whether the I / O data to be processed meets the data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold;

[0160] The data sending module 403 is used to send the I / O data to be processed to the target hard disk of the target server when it is determined that the I / O data to be processed meets the data migration condition.

[0161] In one embodiment of the present invention, the server dimension speed limit threshold includes a source server speed limit threshold of a source server and a target server speed limit threshold of a target server;

[0162] The hard disk dimension speed limit threshold includes the source hard disk speed limit threshold of the source hard disk and the target hard disk speed limit threshold of the target hard disk.

[0163] In one embodiment of the present invention, the threshold determination module 401 includes:

[0164] A first threshold submodule, used to determine a cluster dimension speed limit threshold based on a first I / O processing capability and a resource allocation ratio of the storage cluster;

[0165] A second threshold submodule, used to determine a speed limit threshold of the source server based on a second I / O processing capability and a resource allocation ratio of the source server;

[0166] A third threshold submodule, used to determine a target server speed limit threshold based on a third I / O processing capability and a resource allocation ratio of the target server;

[0167] A fourth threshold submodule, used to determine a source hard disk speed limit threshold based on a fourth I / O processing capability and a resource allocation ratio of the source hard disk;

[0168] The fifth threshold submodule is used to determine the target hard disk speed limit threshold based on the fifth I / O processing capability and resource allocation ratio of the target hard disk.

[0169] In one embodiment of the present invention, the data management device 400 of the distributed storage system further includes:

[0170] The ratio determination module is used to determine the resource allocation ratio based on the data access frequency of the I / O data to be processed.

[0171] In one embodiment of the present invention, the data migration conditions include cluster migration conditions, server migration conditions and hard disk migration conditions; the condition determination module 402 includes:

[0172] A first condition determination submodule, configured to determine whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of I / O data in transit of the storage cluster, and the cluster dimension speed limit threshold;

[0173] A second condition determination submodule, configured to determine whether the I / O data to be processed meets the server migration condition based on the amount of data to be processed, the second amount of I / O data in transit of the source server, the third amount of I / O data in transit of the target server, and the server dimension speed limit threshold;

[0174] The third condition determination submodule is used to determine whether the I / O data to be processed meets the hard disk migration condition based on the amount of data to be processed, the fourth data amount of the in-transit I / O data of the source hard disk, the fifth data amount of the in-transit I / O data of the target hard disk, and the hard disk dimension speed limit threshold.

[0175] In one embodiment of the present invention, the first condition determination submodule includes:

[0176] A first total amount determining unit, used to determine a first total amount of data between the amount of data to be processed and the first amount of data;

[0177] A first condition determination unit, configured to determine that the I / O data to be processed meets the cluster migration condition when the total amount of the first data is less than or equal to the cluster dimension speed limit threshold;

[0178] The second condition determination submodule includes:

[0179] A second total amount determining unit, used to determine a second total amount of data between the amount of data to be processed and the second amount of data, and a third total amount of data between the amount of data to be processed and the third amount of data;

[0180] A second condition determination unit, configured to determine that the to-be-processed I / O data satisfies the server migration condition when the second data total amount is less than or equal to the source server speed limit threshold and the third data total amount is less than or equal to the target server speed limit threshold;

[0181] The third condition determination submodule includes:

[0182] A third total amount determining unit, used to determine a fourth total amount of data between the to-be-processed data amount and the fourth data amount, and a fifth total amount of data between the to-be-processed data amount and the fifth data amount;

[0183] The third condition determination unit is used to determine that the to-be-processed I / O data meets the hard disk migration condition when the fourth data total amount is less than or equal to the source hard disk speed limit threshold and the fifth data total amount is less than or equal to the target hard disk speed limit threshold.

[0184] In one embodiment of the present invention, the data management device 400 of the distributed storage system further includes:

[0185] The data update module is used to update the first data volume, the second data volume, the third data volume, the fourth data volume and the fifth data volume based on the amount of data to be processed when it is determined that the I / O data to be processed meets the cluster migration condition, the server migration condition and the hard disk migration condition.

[0186] In one embodiment of the present invention, the data management device 400 of the distributed storage system further includes:

[0187] A timing module, for triggering timing of a delayed sending time when it is determined that the I / O data to be processed does not meet the cluster migration condition, the server migration condition or the hard disk migration condition;

[0188] The cyclic judgment module is used to execute the step of determining whether the I / O data to be processed meets the data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional speed limit threshold when the delayed sending time reaches the time threshold.

[0189] It should be noted that the specific implementation of the data management device 400 of the distributed storage system in the embodiment of the present invention refers to the specific implementation of the data management method of the distributed storage system proposed in the first aspect of the embodiment of the present invention, which will not be repeated here.

[0190] Based on the same inventive concept, Figure 5 An embodiment of the present invention further provides an electronic device 500, comprising a memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor 502, which, when executed by the processor, implements the steps in the data management method of the distributed storage system of any of the above embodiments.

[0191] It should be noted that the specific implementation of the electronic device 500 in the embodiment of the present invention refers to the specific implementation of the data management method of the distributed storage system proposed in the first aspect of the embodiment of the present invention, and will not be repeated here.

[0192] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the data management method of the distributed storage system of any of the above embodiments are implemented.

[0193] It should be noted that the specific implementation of the computer-readable storage medium of the embodiment of the present invention refers to the specific implementation of the data management method of the distributed storage system proposed in the first aspect of the embodiment of the present invention, and will not be repeated here.

[0194] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the data management method of a distributed storage system of any of the above embodiments.

[0195] It should be noted that the specific implementation of the computer program product in the embodiment of the present invention refers to the specific implementation of the data management method of the distributed storage system proposed in the first aspect of the embodiment of the present invention, and will not be repeated here.

[0196] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0197] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0203] The data management method, electronic device and storage medium of a distributed storage system provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data management method for a distributed storage system, characterized in that: The method comprises: In the case of an abnormal node in the storage cluster, a multi-dimensional speed limit threshold is determined; the multi-dimensional speed limit threshold includes a cluster dimension speed limit threshold, a server dimension speed limit threshold and a hard disk dimension speed limit threshold; Determining whether the I / O data to be processed meets a data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold; When it is determined that the I / O data to be processed meets the data migration condition, the I / O data to be processed is sent to the target hard disk of the target server.

2. The data management method of a distributed storage system according to claim 1, characterized in that: The server dimension speed limit threshold includes a source server speed limit threshold of the source server and a target server speed limit threshold of the target server; The hard disk dimension speed limit threshold includes a source hard disk speed limit threshold of a source hard disk and a target hard disk speed limit threshold of the target hard disk.

3. The data management method of a distributed storage system according to claim 2, characterized in that: Determine multi-dimensional speed limit thresholds, including: Determining the cluster dimension speed limit threshold based on the first I / O processing capability and resource allocation ratio of the storage cluster; Determining a speed limit threshold of the source server based on the second I / O processing capacity of the source server and the resource allocation ratio; Determining a speed limit threshold of the target server based on the third I / O processing capability of the target server and the resource allocation ratio; Determining a speed limit threshold of the source hard disk based on the fourth I / O processing capability of the source hard disk and the resource allocation ratio; The target hard disk speed limit threshold is determined based on the fifth I / O processing capability of the target hard disk and the resource allocation ratio.

4. The data management method of the distributed storage system according to claim 3, characterized in that: The method further comprises: The resource allocation ratio is determined based on the data access frequency of the I / O data to be processed.

5. The data management method of a distributed storage system according to claim 2, characterized in that: The data migration conditions include cluster migration conditions, server migration conditions and hard disk migration conditions; Determining whether the I / O data to be processed meets a data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional rate limit threshold includes: Determining whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of I / O data in transit of the storage cluster, and the cluster dimension speed limit threshold; Determining whether the I / O data to be processed meets the server migration condition based on the amount of data to be processed, a second amount of I / O data in transit of the source server, a third amount of I / O data in transit of the target server, and the server dimension speed limit threshold; Based on the amount of data to be processed, the fourth amount of I / O data in transit of the source hard disk, the fifth amount of I / O data in transit of the target hard disk, and the hard disk dimension speed limit threshold, determine whether the I / O data to be processed meets the hard disk migration condition.

6. The data management method of the distributed storage system according to claim 5, characterized in that: Determining whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of I / O data in transit of the storage cluster, and the cluster dimension speed limit threshold includes: Determine a first total amount of data between the to-be-processed data amount and the first data amount; When the first total amount of data is less than or equal to the cluster dimension speed limit threshold, determining that the to-be-processed I / O data meets the cluster migration condition; Determining whether the I / O data to be processed meets the server migration condition based on the amount of data to be processed, a second amount of I / O data in transit of the source server, a third amount of I / O data in transit of the target server, and the server dimension speed limit threshold includes: Determine a second total amount of data between the to-be-processed data amount and the second data amount, and a third total amount of data between the to-be-processed data amount and the third data amount; When the second total amount of data is less than or equal to the source server speed limit threshold and the third total amount of data is less than or equal to the target server speed limit threshold, determining that the to-be-processed I / O data meets the server migration condition; Determining whether the I / O data to be processed meets the hard disk migration condition based on the amount of data to be processed, a fourth amount of I / O data in transit of the source hard disk, a fifth amount of I / O data in transit of the target hard disk, and the hard disk dimension speed limit threshold includes: Determine a fourth total amount of data between the to-be-processed data amount and the fourth data amount, and a fifth total amount of data between the to-be-processed data amount and the fifth data amount; When the fourth total amount of data is less than or equal to the source hard disk speed limit threshold and the fifth total amount of data is less than or equal to the target hard disk speed limit threshold, it is determined that the I / O data to be processed meets the hard disk migration condition.

7. The data management method of a distributed storage system according to claim 6, characterized in that: The method further comprises: When it is determined that the I / O data to be processed meets the cluster migration condition, the server migration condition and the hard disk migration condition, the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount are updated based on the amount of data to be processed.

8. The data management method of a distributed storage system according to claim 5, characterized in that: The method further comprises: When it is determined that the to-be-processed I / O data does not satisfy the cluster migration condition, the server migration condition or the hard disk migration condition, triggering timing of a delayed sending time; When the delayed sending duration reaches the duration threshold, the step of determining whether the I / O data to be processed meets the data migration condition based on the amount of the I / O data to be processed of the abnormal node and the multi-dimensional speed limit threshold is performed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data management method of the distributed storage system according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data management method of the distributed storage system according to any one of claims 1 to 8 is implemented.

Images