Data management method of distributed storage system, electronic device and storage medium

Abstract

The application provides a data management method of a distributed storage system, an electronic device and a storage medium, and belongs to the technical field of data storage. In the case that an abnormal node exists in a storage cluster, a multi-dimensional speed limiting threshold is determined, whether the to-be-processed I / O data meets a data migration condition is effectively judged according to the multi-dimensional speed limiting threshold and a to-be-processed data amount of the to-be-processed I / O data of the abnormal node, and in the case that it is determined that the to-be-processed I / O data meets the data migration condition, the to-be-processed I / O data is sent to a target hard disk of a target server. Through speed limiting processing of to-be-processed I / O data from multiple dimensions such as a cluster dimension, a server dimension and a hard disk dimension, the application can effectively prevent various I / O processing in the storage cluster from affecting the processing of foreground I / O due to resource grabbing, thereby effectively ensuring the service experience of users.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a data management method of a distributed storage system, an electronic device and a storage medium. BACKGROUND

[0002] The total read-write capability of a distributed storage system is limited and is affected by factors such as network bandwidth and the number of read-write operations per second, so it is necessary to reasonably allocate the limited I / O capability to ensure that important businesses can enjoy sufficient I / O capability, that is, to prioritize the transmission of urgent and important information or to provide higher quality services to specific users.

[0003] Current I / O requests are mainly divided into foreground I / O requests and background I / O requests, wherein the processing effect of foreground I / O requests affects the quality of service provided by the cluster to the client, and the processing effect of background I / O requests affects the reliability of the cluster.

[0004] When an exception occurs in a node of a distributed storage cluster, the data stored in each node will be inconsistent, so data repair is needed. When there is a large amount of data to be recovered in the cluster, a large number of background I / O requests need to be processed, which may cause the cluster to have no resources to process foreground I / O requests, thereby affecting the service experience of the customer.

[0005] In related technologies, a corresponding data recovery speed limit value is allocated to each node according to the current maximum amount of data to be recovered in the distributed storage cluster. However, this method only limits the speed of background I / O requests from the cluster dimension, which cannot effectively coordinate foreground and background I / O requests, thereby affecting the user experience. SUMMARY

[0006] The present application provides a data management method of a distributed storage system, an electronic device and a storage medium to solve the problem that it is difficult to effectively manage background I / O requests of a distributed storage system.

[0007] To solve the above problems, the present application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a data management method of a distributed storage system, which comprises:

[0009] In the case of an abnormal node in the storage cluster, a multi-dimensional speed limit threshold is determined; the multi-dimensional speed limit threshold comprises a cluster dimension speed limit threshold, a server dimension speed limit threshold and a hard disk dimension speed limit threshold;

[0010] determining whether the to-be-processed I / O data meets a data migration condition based on a to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional throttling threshold;

[0011] in a case where it is determined that the to-be-processed I / O data meets the data migration condition, sending the to-be-processed I / O data to a target hard disk of a target server.

[0012] In an embodiment of the present application, the server dimension throttling threshold comprises a source server throttling threshold of a source server and a target server throttling threshold of the target server.

[0013] The hard disk dimension throttling threshold comprises a source hard disk throttling threshold of a source hard disk and a target hard disk throttling threshold of the target hard disk.

[0014] In an embodiment of the present application, determining the multi-dimensional throttling threshold comprises:

[0015] determining the cluster dimension throttling threshold based on a first I / O processing capability of the storage cluster and a resource allocation ratio;

[0016] determining the source server throttling threshold based on a second I / O processing capability of the source server and the resource allocation ratio;

[0017] determining the target server throttling threshold based on a third I / O processing capability of the target server and the resource allocation ratio;

[0018] determining the source hard disk throttling threshold based on a fourth I / O processing capability of the source hard disk and the resource allocation ratio;

[0019] determining the target hard disk throttling threshold based on a fifth I / O processing capability of the target hard disk and the resource allocation ratio.

[0020] In an embodiment of the present application, the method further comprises:

[0021] determining the resource allocation ratio based on a data access frequency of the to-be-processed I / O data.

[0022] In an embodiment of the present application, the data migration condition comprises a cluster migration condition, a server migration condition and a hard disk migration condition.

[0023] determining whether the to-be-processed I / O data meets a data migration condition based on a to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional throttling threshold, comprises:

[0024] determining whether the to-be-processed I / O data meets the cluster migration condition based on the to-be-processed data amount, a first data amount of in-transit I / O data of the storage cluster, and a cluster dimension throttling threshold value;

[0025] determining whether the to-be-processed I / O data meets the server migration condition based on the to-be-processed data amount, a second data amount of in-transit I / O data of the source server, a third data amount of in-transit I / O data of the target server, and a server dimension throttling threshold value;

[0026] determining whether the to-be-processed I / O data meets the hard disk migration condition based on the to-be-processed data amount, a fourth data amount of in-transit I / O data of the source hard disk, a fifth data amount of in-transit I / O data of the target hard disk, and a hard disk dimension throttling threshold value.

[0027] In an embodiment of the present application, determining whether the to-be-processed I / O data meets the cluster migration condition based on the to-be-processed data amount, a first data amount of in-transit I / O data of the storage cluster, and a cluster dimension throttling threshold value comprises:

[0028] determining a first total data amount between the to-be-processed data amount and the first data amount;

[0029] in a case where the first total data amount is less than or equal to the cluster dimension throttling threshold value, determining that the to-be-processed I / O data meets the cluster migration condition;

[0030] determining whether the to-be-processed I / O data meets the server migration condition based on the to-be-processed data amount, a second data amount of in-transit I / O data of the source server, a third data amount of in-transit I / O data of the target server, and a server dimension throttling threshold value comprises:

[0031] determining a second total data amount between the to-be-processed data amount and the second data amount, and a third total data amount between the to-be-processed data amount and the third data amount;

[0032] in a case where the second total data amount is less than or equal to the source server throttling threshold value and the third total data amount is less than or equal to the target server throttling threshold value, determining that the to-be-processed I / O data meets the server migration condition;

[0033] determining whether the to-be-processed I / O data meets the hard disk migration condition based on the to-be-processed data amount, a fourth data amount of in-transit I / O data of the source hard disk, a fifth data amount of in-transit I / O data of the target hard disk, and a hard disk dimension throttling threshold value comprises:

[0034] determining a fourth total data amount between the to-be-processed data amount and the fourth data amount, and a fifth total data amount between the to-be-processed data amount and the fifth data amount;

[0035] in a case where the fourth total data amount is less than or equal to the source hard disk speed limit threshold and the fifth total data amount is less than or equal to the target hard disk speed limit threshold, determining that the to-be-processed I / O data satisfies the hard disk migration condition.

[0036] In an embodiment of the present application, the method further comprises:

[0037] in a case where it is determined that the to-be-processed I / O data satisfies the cluster migration condition, the server migration condition and the hard disk migration condition, updating the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount based on the to-be-processed data amount.

[0038] In an embodiment of the present application, the method further comprises:

[0039] in a case where it is determined that the to-be-processed I / O data does not satisfy the cluster migration condition, the server migration condition or the hard disk migration condition, triggering timing for a delayed sending time length;

[0040] in a case where the delayed sending time length reaches a time length threshold, performing the step of determining whether the to-be-processed I / O data satisfies a data migration condition based on a to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional speed limit threshold.

[0041] In a second aspect, based on the same inventive concept, an embodiment of the present application provides a data management device of a distributed storage system, the device comprising:

[0042] a threshold value determination module configured to determine a multi-dimensional speed limit threshold in a case where an abnormal node exists in a storage cluster; the multi-dimensional speed limit threshold comprises a cluster dimension speed limit threshold, a server dimension speed limit threshold and a hard disk dimension speed limit threshold;

[0043] a condition determination module configured to determine whether to-be-processed I / O data of the abnormal node satisfies a data migration condition based on a to-be-processed data amount of the to-be-processed I / O data and the multi-dimensional speed limit threshold;

[0044] a data sending module configured to send the to-be-processed I / O data to a target hard disk of a target server in a case where it is determined that the to-be-processed I / O data satisfies the data migration condition.

[0045] In an embodiment of the present application, the server dimension speed limit threshold comprises a source server speed limit threshold of a source server and a target server speed limit threshold of the target server;

[0046] The hard disk dimension speed limiting threshold comprises a source hard disk speed limiting threshold of a source hard disk and a target hard disk speed limiting threshold of the target hard disk.

[0047] In an embodiment of the present application, the threshold determining module comprises:

[0048] A first threshold sub-module, configured to determine the cluster dimension speed limiting threshold based on a first I / O processing capability of the storage cluster and a resource allocation ratio;

[0049] A second threshold sub-module, configured to determine the source server speed limiting threshold based on a second I / O processing capability of the source server and the resource allocation ratio;

[0050] A third threshold sub-module, configured to determine the target server speed limiting threshold based on a third I / O processing capability of the target server and the resource allocation ratio;

[0051] A fourth threshold sub-module, configured to determine the source hard disk speed limiting threshold based on a fourth I / O processing capability of the source hard disk and the resource allocation ratio;

[0052] A fifth threshold sub-module, configured to determine the target hard disk speed limiting threshold based on a fifth I / O processing capability of the target hard disk and the resource allocation ratio.

[0053] In an embodiment of the present application, the data management apparatus of the distributed storage system further comprises:

[0054] A ratio determining module, configured to determine the resource allocation ratio based on a data access frequency of the to-be-processed I / O data.

[0055] In an embodiment of the present application, the data migration condition comprises a cluster migration condition, a server migration condition and a hard disk migration condition; and the condition determining module comprises:

[0056] A first condition determining sub-module, configured to determine whether the to-be-processed I / O data satisfies the cluster migration condition based on the to-be-processed data amount, a first data amount of in-transit I / O data of the storage cluster and the cluster dimension speed limiting threshold;

[0057] A second condition determining sub-module, configured to determine whether the to-be-processed I / O data satisfies the server migration condition based on the to-be-processed data amount, a second data amount of in-transit I / O data of the source server, a third data amount of in-transit I / O data of the target server and the server dimension speed limiting threshold;

[0058] The third condition determining sub-module is configured to determine whether the to-be-processed I / O data satisfies the hard disk migration condition based on the to-be-processed data amount, a fourth data amount of in-transit I / O data of the source hard disk, a fifth data amount of in-transit I / O data of the target hard disk, and the hard disk dimension limit threshold.

[0059] In an embodiment of the present application, the first condition determining sub-module comprises:

[0060] A first total amount determining unit is configured to determine a first data total amount between the to-be-processed data amount and the first data amount.

[0061] A first condition determining unit is configured to determine that the to-be-processed I / O data satisfies the cluster migration condition when the first data total amount is less than or equal to the cluster dimension limit threshold.

[0062] The second condition determining sub-module comprises:

[0063] A second total amount determining unit is configured to determine a second data total amount between the to-be-processed data amount and the second data amount, and a third data total amount between the to-be-processed data amount and the third data amount.

[0064] A second condition determining unit is configured to determine that the to-be-processed I / O data satisfies the server migration condition when the second data total amount is less than or equal to the source server limit threshold and the third data total amount is less than or equal to the target server limit threshold.

[0065] The third condition determining sub-module comprises:

[0066] A third total amount determining unit is configured to determine a fourth data total amount between the to-be-processed data amount and the fourth data amount, and a fifth data total amount between the to-be-processed data amount and the fifth data amount.

[0067] A third condition determining unit is configured to determine that the to-be-processed I / O data satisfies the hard disk migration condition when the fourth data total amount is less than or equal to the source hard disk limit threshold and the fifth data total amount is less than or equal to the target hard disk limit threshold.

[0068] In an embodiment of the present application, the data management apparatus of the distributed storage system further comprises:

[0069] A data updating module is configured to update the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount based on the to-be-processed data amount when it is determined that the to-be-processed I / O data satisfies the cluster migration condition, the server migration condition and the hard disk migration condition.

[0070] In an embodiment of the present application, the data management device of the distributed storage system further comprises:

[0071] a timing module configured to trigger timing of a delay sending duration when it is determined that the to-be-processed I / O data does not satisfy the cluster migration condition, the server migration condition or the hard disk migration condition;

[0072] a cycle judgment module configured to execute the step of determining whether the to-be-processed I / O data satisfies the data migration condition based on the to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional limiting threshold when the delay sending duration reaches a duration threshold.

[0073] In a third aspect, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the data management method of the distributed storage system according to the first aspect of the present application when executed.

[0074] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the program is executable on a processor to implement the data management method of the distributed storage system according to the first aspect of the present application.

[0075] Compared with the prior art, the present application has the following advantages:

[0076] The data management method of the distributed storage system provided by the embodiment of the present application first determines a multi-dimensional limiting threshold when an abnormal node appears in a storage cluster, the multi-dimensional limiting threshold comprising a cluster dimension limiting threshold, a server dimension limiting threshold and a hard disk dimension limiting threshold; then determines whether the to-be-processed I / O data satisfies a data migration condition based on the to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional limiting threshold; and finally sends the to-be-processed I / O data to a target hard disk of a target server when it is determined that the to-be-processed I / O data satisfies the data migration condition. The embodiment of the present application comprehensively considers the I / O processing capacity of the cluster, the server and the hard disk, limits the to-be-processed I / O data from multiple dimensions such as the cluster dimension, the server dimension and the hard disk dimension, can effectively and sufficiently improve the overall resource utilization rate of the system, and prevents various I / O processing in the storage cluster from affecting the processing of foreground I / O due to resource grabbing. In this way, the foreground business can be ensured to have sufficient I / O capacity when an abnormal node appears in the storage cluster, and the service experience of the user is improved. BRIEF DESCRIPTION OF DRAWINGS

[0077] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only need to be some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0078] Figure 1 is a step flow chart of a data management method of a distributed storage system in an embodiment of the present application.

[0079] Figure 2 is a flow chart of a data management method of a distributed storage system in an embodiment of the present application.

[0080] Figure 3 is a flow chart of a data migration condition judgment in an embodiment of the present application.

[0081] Figure 4 is a functional module schematic diagram of a data management device of a distributed storage system in an embodiment of the present application.

[0082] Figure 5 is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the present application.

[0084] It should be noted that in the distributed storage system, the hardware resources of the data plane service node, such as CPU, memory, network, storage medium and other resources, are all included in the whole system for scheduling and allocation, and the block, file, object and other storage products are provided externally. The maximum utilization of hardware resources can avoid waste of resources, so as to achieve the goal of cost control and provide better customer experience. Under the premise of ensuring reliability and system performance, reasonable overall allocation of IO resources and limited system storage space can cope with various complex resource demand situations in the system, and provide better service externally.

[0085] In a distributed storage system, there are mainly two data distribution methods: one is a central control node that stores data distribution routes, and clients determine the data plane service nodes and disks where data shards are located through route information. A typical example is Google's GFS (Google File System, a distributed file system) and big data storage HDFS (Distributed File System, a distributed file system). The other is a weak central control node, and clients determine the data plane service nodes and disks where data shards are located through calculation, such as the open source distributed storage system Ceph that is widely used in the industry. Regardless of the data distribution method, there are generally the following types of I / O in a distributed storage system:

[0086] ClientOp: read and write I / O requests sent by cluster clients. In most cases, the cluster should prioritize timely processing of this type of request.

[0087] SubOp: I / O requests between data plane nodes in the cluster. This mainly includes read and write I / O requests between replicas generated by read and write I / O requests sent by cluster clients, read and write I / O requests between EC shards, load balancing, data scanning, data synchronization, and other operations that may cause this type of I / O request.

[0088] SnapTrimOp: deletion of cluster snapshot data may cause this type of I / O request.

[0089] ScrubOp: I / O requests generated by Scrub that discovers silent data errors of objects, including Scrub that only scans object metadata and deepScrub that scans objects as a whole.

[0090] RecoveryOp: I / O requests caused by data rebalancing recovery migration. Manual data rebalancing, data plane service process failure or rejoining, cluster expansion, and data load balancing operations may generate I / O requests. Among them, Recovery represents data rebalancing, and increasing nodes or hard disk data service processes in a distributed storage cluster, node downtime, data service downtime, and data load balancing may cause data rebalancing. Recovery mainly repairs inconsistent data blocks in PG (Partitioned Global, Placement Group).

[0091] BackfillOp: data backfill, adding nodes or hard disk data service processes in the distributed storage cluster, node offline, data service offline, data load balancing, will cause data backfill, copy the data block in PG as a whole. Among them, Backfill means data backfill, adding nodes or hard disk data service processes in the distributed storage cluster, node offline, data service offline, data load balancing, will cause data backfill, Backfill mainly copies the data block in PG as a whole, and is a special recovery means.

[0092] It should be noted that PG is an internal data structure used to store data in a certain storage pool across hard disk data service processes. With respect to the storage pool, PG is a virtual component, which is a virtual layer used when Chunk (data block) is mapped to the storage pool. Among them, Chunk is a logical concept for storing user data, written in an append mode. The user's data may not be continuous in the data block, and the index layer above the persistent layer records the location information of the user's data in the data block.

[0093] The read and write capabilities of the overall distributed storage system cluster are limited and will be affected by factors such as network bandwidth and the number of read and write operations per second. Therefore, it is necessary to reasonably allocate the limited I / O capability to ensure that important businesses can enjoy sufficient I / O capability, i.e., to prioritize the transmission of urgent and important information or to provide higher quality services to specific users. The foregoing I / O requests can be divided into two categories: foreground I / O requests, including ClientOp and SubOp generated by ClientOp; and background I / O, including RecoveryOp, BackfillOP, ScrubOp, and SnapTrimOp. The foreground and background I / O request processing ensures high-quality services provided to customers and the reliability of the cluster. In the distributed storage cluster, node offline and abnormal downtime are common phenomena, which can cause inconsistent data blocks on each replica of PG, and data repair must be performed using the corresponding data balancing strategy for Recovery and Backfill, which will inevitably affect the services provided by the cluster to the clients and may cause the cluster to be temporarily unavailable. Setting data balancing at a reasonable priority can achieve a balance between availability and reliability.

[0094] In the related art, in view of the above problems, the following three types of optimization are currently mainly done: the first type is that the current maximum recovery data amount in the preset time length is used to allocate a one-to-one data recovery speed limit value for each target storage node according to the current maximum recovery data amount of the distributed storage system. The second type is that the maximum recovery data amount is preset, the recovery data amount of the PG of each data plane node in a certain time period is dynamically obtained, the average value is calculated and compared with the maximum recovery amount, and the recovery speed is adjusted according to the result, and the center control node sends a message to adjust the recovery speed. The third type is the most typical one, in which different types of sub-queues are set in the data service of the data plane node, mainly including a priority queue and a priority queue based on weight, I / O requests are classified, a QoS template is designed for each type of I / O request, and the order of I / O requests is controlled through weight.

[0095] The above three types of optimization adjust the data recovery according to the preset time length of the set recovery data amount, which is only based on historical experience value, and the data recovery speed is relatively fast, so the data in the cluster may have been recovered before the preset time length is reached, thereby causing low data recovery efficiency and being unable to accurately coordinate foreground and background I / O. The queue control method based on weight has a small effect of weight when the load of the data plane node is not high, and the I / O resources of the system are equally divided among various types of I / O requests, so it is also unable to accurately coordinate foreground and background I / O. The above three types of methods are designed from the perspective of the total recovery amount of the cluster at most, none of which considers the bandwidth load of the source node and the target node in data recovery, none of which considers the bandwidth load of the source hard disk and the target hard disk, and none of which considers the three in the limitation of the cluster background I / O. The third type of optimization only considers IOPS (Input / Output Operations per Second, which represents the number of I / Os that can be processed per second), and does not consider the limitation of I / O bandwidth.

[0096] In view of the problem that the background I / O request of the distributed storage system is difficult to manage effectively at present, the present application aims to provide a data management method of a distributed storage system, which can effectively and fully improve the overall resource utilization rate of the system by limiting the speed of the I / O data to be processed from multiple dimensions such as cluster dimension, server dimension and hard disk dimension, and prevent various I / O processing in the distributed storage cluster from affecting the processing of foreground I / O due to resource grabbing. In this way, when an abnormal node appears in the storage cluster, the foreground business can be ensured to have sufficient I / O capability, and the service experience of the user can be improved.

[0097] Reference Figure 1 A step flowchart of a data management method of a distributed storage system of the present application is shown, which can include the following steps:

[0098] S101: In the case where an abnormal node appears in the storage cluster, a multi-dimensional speed limit threshold is determined.

[0099] It should be noted that the distributed storage cluster is a system architecture that stores data in multiple physical or virtual nodes. It stores data in multiple nodes at multiple physical locations connected through a network.

[0100] In this embodiment, the abnormal node refers to a node that generates I / O data to be processed. Specifically, data recovery, data migration, data scanning, data repair, and data balancing are all performed by PG to perform distributed read and write across nodes and disks. Silent data error checking, manual data rebalancing, and data service process failure or rejoining, cluster expansion, data load balancing, and other operations may generate PG migration, deletion, and read and write, which may generate I / O data to be processed.

[0101] In this embodiment, the multi-dimensional throttling threshold includes a cluster dimension throttling threshold, a server dimension throttling threshold, and a hard disk dimension throttling threshold. The cluster dimension throttling threshold is a throttling threshold set based on the I / O processing capacity of the storage cluster; the server dimension throttling threshold is a throttling threshold set based on the I / O processing capacity of the server; and the hard disk dimension throttling threshold is a throttling threshold set based on the I / O processing capacity of the hard disk.

[0102] In specific implementation, considering that the I / O processing capacity of the source server, the target server, the source hard disk, and the target hard disk is limited, the server dimension throttling threshold can specifically include a source server throttling threshold of the source server and a target server throttling threshold of the target server; and the hard disk dimension throttling threshold includes a source hard disk throttling threshold of the source hard disk and a target hard disk throttling threshold of the target hard disk. The source server throttling threshold is a throttling threshold set based on the I / O processing capacity of the source server; the target server throttling threshold is a throttling threshold set based on the I / O processing capacity of the target server; the source hard disk throttling threshold is a throttling threshold set based on the I / O processing capacity of the source hard disk; and the target hard disk throttling threshold is a throttling threshold set based on the I / O processing capacity of the target hard disk.

[0103] It should be noted that the source hard disk refers to a hard disk storing I / O data to be processed; and the source server refers to a server where the I / O data to be processed is located. That is, after the abnormal node of the storage cluster, the I / O data to be processed stored in the source hard disk of the source server will be migrated to the target hard disk of the target server.

[0104] S102: Based on the to-be-processed data amount of the I / O data to be processed of the abnormal node and the multi-dimensional throttling threshold, it is determined whether the I / O data to be processed meets the data migration condition.

[0105] In the embodiment, since the I / O processing capabilities of the cluster, the server and the hard disk are limited, by comprehensively considering the data volume relationship among the to-be-processed data volume, the cluster dimension limiting threshold, the server dimension limiting threshold and the hard disk dimension limiting threshold, it can be determined whether the to-be-processed I / O data meets the data migration condition.

[0106] S103: In the case where it is determined that the to-be-processed I / O data meets the data migration condition, the to-be-processed I / O data is sent to the target hard disk of the target server.

[0107] In the embodiment, if it is detected that the to-be-processed I / O data meets the data migration condition, it means that the current cluster, server and hard disk all have sufficient I / O processing capabilities to process the to-be-processed I / O data, and at this time, the to-be-processed I / O data is allowed to be sent to the target hard disk of the target server.

[0108] In a specific implementation, if the target hard disk does not have in-transit I / O data, i.e., no other node transmits I / O data to the target hard disk, the to-be-processed I / O data is directly sent to the target hard disk of the target server; if the target hard disk has in-transit I / O data, i.e., other nodes have transmitted I / O data to the target hard disk before the abnormal node, the to-be-processed I / O data is put into a data transmission queue, and after the in-transit I / O data before the to-be-processed I / O data is transmitted, the to-be-processed I / O data is transmitted to the target hard disk of the target server.

[0109] In the embodiment, considering that the configurations of the servers in the distributed storage cluster are different, resulting in different I / O processing capabilities, by fully considering the processing capabilities of the cluster, the server and the hard disk, the to-be-processed I / O data is limited in three dimensions of the cluster, the server and the hard disk, the I / O processing capabilities of each server node can be balanced, the waste of some server resources can be prevented, the overall resource utilization of the system can be improved, and the resource grabbing of various I / O processing in the distributed storage cluster can be prevented, so as to affect the processing of the foreground I / O. It is ensured that the foreground business can enjoy sufficient I / O capability, and high-quality services are provided for the upper-layer business.

[0110] In a feasible embodiment, the step of determining the multi-dimensional limiting threshold in S101 can specifically include the following sub-steps:

[0111] S101-1: determining a cluster dimension limiting threshold based on a first I / O processing capability of the storage cluster and a resource allocation ratio.

[0112] S101-2: determining a source server limiting threshold based on a second I / O processing capability of the source server and a resource allocation ratio.

[0113] S101-3: Determine the target server throttling threshold based on the third I / O processing capability of the target server and the resource allocation ratio.

[0114] S101-4: Determine the source hard disk throttling threshold based on the fourth I / O processing capability of the source hard disk and the resource allocation ratio.

[0115] S101-5: Determine the target hard disk throttling threshold based on the fifth I / O processing capability of the target hard disk and the resource allocation ratio.

[0116] In this embodiment, considering that there may be some differences between the configurations of the storage cluster, the source server, the target server, the source hard disk and the target hard disk, the multi-dimensional throttling threshold will be accurately calculated according to the I / O processing capability of each of the storage cluster, the source server, the target server, the source hard disk and the target hard disk. It should be noted that the I / O processing capability is the ability to process input / output (I / O) requests per unit time,

[0117] In a specific implementation, the I / O processing capability can be determined according to network bandwidth, hardware resources and other factors. Specifically, the first I / O processing capability of the storage cluster can be determined according to the network bandwidth of each server of the storage cluster; the second I / O processing capability of the source server and the third I / O processing capability of the target server can be respectively evaluated according to the network bandwidth, CPU and memory of the source server and the target server; and the fourth I / O processing capability of the source hard disk and the fifth I / O processing capability of the target hard disk can be determined based on the hard disk read / write bandwidth.

[0118] In this embodiment, to determine a suitable resource allocation ratio, the resource allocation ratio can be determined based on the data access frequency of the I / O data to be processed. Specifically, the higher the data access frequency, the higher the resource allocation ratio. In this way, for data with a high user access frequency, more I / O resources can be allocated to process the I / O data to be processed, thereby effectively meeting the user's data usage requirements; and for data with a low user access frequency, fewer I / O resources are allocated to process the I / O data to be processed, thereby reserving more I / O resources for processing foreground I / O and improving the user's foreground service experience.

[0119] In one possible implementation, the data migration condition includes a cluster migration condition, a server migration condition and a hard disk migration condition; S102 can specifically include the following sub-steps:

[0120] S102-1: Determine whether the I / O data to be processed meets the cluster migration condition based on the amount of data to be processed, the first amount of in-transit I / O data of the storage cluster, and the cluster dimension throttling threshold.

[0121] It should be noted that the in-transit I / O data of the storage cluster indicates the I / O data in the storage cluster that is in a transmission state.

[0122] In the embodiment, firstly, it is determined whether the to-be-processed I / O data can be migrated to the target hard disk of the target server in the cluster dimension. If the to-be-processed I / O data meets the cluster migration condition, the judgment of the server migration condition and the hard disk migration condition is continued; if not, the to-be-processed I / O data is delayed to be sent.

[0123] In a specific implementation, a first total data amount between the to-be-processed data amount and the first data amount can be determined; and in a case where the first total data amount is less than or equal to the cluster dimension rate limiting threshold, it is determined that the to-be-processed I / O data meets the cluster migration condition.

[0124] In the embodiment, if it is detected that the first total data amount is greater than the cluster dimension rate limiting threshold, it indicates that the storage cluster has no extra I / O resource to process the to-be-processed I / O data, and at this time, the to-be-processed I / O data is delayed to be sent until the first total data amount is less than or equal to the cluster dimension rate limiting threshold.

[0125] S102-2: Based on the to-be-processed data amount, the second data amount of the in-transit I / O data of the source server, the third data amount of the in-transit I / O data of the target server, and the server dimension rate limiting threshold, it is determined whether the to-be-processed I / O data meets the server migration condition.

[0126] It should be noted that the in-transit I / O data of the target server indicates the I / O data in the target server that is in a transmission state.

[0127] In the embodiment, after the to-be-processed I / O data meets the cluster migration condition, the judgment of the server migration condition is further performed.

[0128] In a specific implementation, a second total data amount between the to-be-processed data amount and the second data amount, and a third total data amount between the to-be-processed data amount and the third data amount can be determined; and in a case where the second total data amount is less than or equal to the source server rate limiting threshold and the third total data amount is less than or equal to the target server rate limiting threshold, it is determined that the to-be-processed I / O data meets the server migration condition.

[0129] In the embodiment, if the second total data amount is detected to be greater than the source server throttling threshold, it is indicated that the source server has no spare I / O resource to process the pending I / O data. Similarly, if the third total data amount is detected to be greater than the target server throttling threshold, it is indicated that the target server has no spare I / O resource to process the pending I / O data. At this time, the sending of the pending I / O data is delayed. When the second total data amount is less than or equal to the source server throttling threshold and the third total data amount is less than or equal to the target server throttling threshold, it is indicated that the source server can send the pending I / O data to the target server, and it is determined that the pending I / O data satisfies the server migration condition.

[0130] S102-3: Based on the pending data amount, the fourth data amount of the in-transit I / O data of the source hard disk, the fifth data amount of the in-transit I / O data of the target hard disk, and the hard disk dimension throttling threshold, it is determined whether the pending I / O data satisfies the hard disk migration condition.

[0131] In the embodiment, after the pending I / O data satisfies the cluster migration condition and the hard disk migration condition, the judgment of the hard disk migration condition is finally performed.

[0132] In the specific implementation, a fourth total data amount between the pending data amount and the fourth data amount, and a fifth total data amount between the pending data amount and the fifth data amount can be determined. In the case that the fourth total data amount is less than or equal to the source hard disk throttling threshold and the fifth total data amount is less than or equal to the target hard disk throttling threshold, it is determined that the pending I / O data satisfies the hard disk migration condition.

[0133] In the embodiment, if the fourth total data amount is detected to be greater than the source hard disk throttling threshold, it is indicated that the source hard disk has no spare I / O resource to process the pending I / O data. Similarly, if the fifth total data amount is detected to be greater than the target hard disk throttling threshold, it is indicated that the target hard disk has no spare I / O resource to process the pending I / O data. At this time, the sending of the pending I / O data is delayed. When the fourth total data amount is less than or equal to the source hard disk throttling threshold and the fifth total data amount is less than or equal to the target hard disk throttling threshold, it is indicated that the source hard disk can send the pending I / O data to the target hard disk for storage, and it is determined that the pending I / O data satisfies the hard disk migration condition.

[0134] That is, if it is detected that the pending I / O data satisfies the cluster migration condition, the server migration condition, and the hard disk migration condition, it is indicated that the pending I / O data satisfies the data migration condition, and the pending I / O data is sent to the target hard disk of the target server.

[0135] In the embodiment, by judging whether the to-be-processed I / O data meets the cluster migration condition, the server migration condition and the hard disk migration condition in sequence from the order of the granularity from large to small, the accurate restriction on the to-be-processed I / O data can be realized, and then the accurate regulation on the to-be-processed I / O data is realized.

[0136] In the embodiment, the data management method of the distributed storage system can further include the following steps: in the case that it is determined that the to-be-processed I / O data does not meet the cluster migration condition, the server migration condition or the hard disk migration condition, triggering the timing of the delay sending time length; in the case that the delay sending time length reaches the time length threshold, performing the step of determining whether the to-be-processed I / O data meets the data migration condition based on the to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional speed limit threshold.

[0137] In the embodiment, by detecting whether the to-be-processed I / O data meets the data migration condition every time length threshold, the to-be-processed I / O data can be sent to the target hard disk of the target server in time when there is excess I / O resource in the system.

[0138] In the embodiment, the data management method of the distributed storage system can further include the following steps: in the case that it is determined that the to-be-processed I / O data meets the cluster migration condition, the server migration condition and the hard disk migration condition, updating the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount based on the to-be-processed data amount.

[0139] It should be noted that since the to-be-processed I / O data can be generated in each node of the storage cluster, once it is detected that the to-be-processed I / O data of any abnormal node meets the corresponding data migration condition, the background in-transit I / O data amount of the storage cluster, the source server, the target server, the source hard disk and the target hard disk will be updated, that is, the to-be-processed data amount of the to-be-processed I / O data is added.

[0140] It should be noted that after the background I / O of the present migration is executed, the background in-transit I / O data amount of the storage cluster, the source server, the target server, the source hard disk and the target hard disk will also be updated, that is, the data amount of the background I / O of the present migration is subtracted.

[0141] Referring to Figure 2 , a flowchart of a data management method of a distributed storage system according to an embodiment of the present application is shown. Specifically, the method can include the following steps:

[0142] S201: The system is started, and the limit threshold of the storage cluster, the source server, the target server, the source hard disk and the target hard disk is determined.

[0143] S202: An abnormal node occurs in the storage cluster, and the to-be-processed I / O data is generated.

[0144] S203: It is checked whether the to-be-processed I / O data exceeds the limit threshold of the storage cluster, the source server, the target server, the source hard disk and the target hard disk.

[0145] S204: It is determined that the to-be-processed I / O data does not exceed the limit threshold of the storage cluster, the source server, the target server, the source hard disk and the target hard disk, and the to-be-processed I / O data is processed.

[0146] S205: After the to-be-processed I / O data is processed, the in-transit I / O data amount of the storage cluster, the source server, the target server, the source hard disk and the target hard disk is updated.

[0147] In one example, taking the node abnormal downtime scenario as an example, after the node abnormal downtime, the PG on the node belongs to the replica pool, and the PG on the node lacks one replica, so a new node is selected to migrate data from other replicas of the PG to the new node. The PG migration generates background I / O, and the PG migration information includes the source server, the source hard disk information, the target server, the target hard disk, the migration data amount and the like. Each data migration will calculate whether the to-be-processed data amount of the to-be-processed I / O data of this migration and the sum of the in-transit background I / O data amounts of the storage cluster, the source server, the target server, the source hard disk and the target hard disk exceed the cluster background I / O limit threshold, the source server background I / O limit threshold, the target server background I / O limit threshold, the source hard disk background I / O limit threshold and the target hard disk background I / O limit threshold. If any limit threshold is exceeded, the migration enters a waiting state, and is checked again after a preset time length. If all limit thresholds are not exceeded, the in-transit background I / O data amounts of the storage cluster, the source server, the target server, the source hard disk and the target hard disk are updated, and the to-be-processed data amount of the to-be-processed I / O data of this migration is added to the original in-transit background I / O data amount. After the background I / O of this migration is executed, the data amount of this migration is subtracted to update the in-transit I / O data amounts of the storage cluster, the source server, the target server, the source hard disk and the target hard disk.

[0148] Referring to Figure 3 , a judgment flowchart of the data migration condition is shown. Specifically, the following steps can be included:

[0149] S301: It is judged whether the sum of the to-be-processed data amount and the data amount of the in-transit I / O data of the storage cluster exceeds the cluster dimension limit threshold; if yes, S302 is entered; if no, S303 is entered.

[0150] S302: Trigger the timing for the delay sending duration, judge whether the delay sending duration reaches the duration threshold value; if the delay sending duration reaches the duration threshold value, enter S301.

[0151] S303: Judge whether the sum of the to-be-processed data amount and the data amount of the in-transit I / O data of the source server exceeds the source server rate limiting threshold value; if yes, enter S302; if no, enter S304.

[0152] S304: Judge whether the sum of the to-be-processed data amount and the data amount of the in-transit I / O data of the target server exceeds the target server rate limiting threshold value; if yes, enter S302; if no, enter S305.

[0153] S305: Judge whether the sum of the to-be-processed data amount and the data amount of the in-transit I / O data of the source hard disk exceeds the source hard disk rate limiting threshold value; if yes, enter S302; if no, enter S306.

[0154] S306: Judge whether the sum of the to-be-processed data amount and the data amount of the in-transit I / O data of the target hard disk exceeds the target hard disk rate limiting threshold value; if yes, enter S302; if no, enter S307.

[0155] S307: According to the to-be-processed data amount, increase the data amount of the in-transit I / O data of the storage cluster, the source server, the target server, the source hard disk and the target hard disk.

[0156] In the implementation of the present application, by fully considering the I / O processing capacity of the storage cluster, the source server, the target server, the source hard disk and the target hard disk, and by limiting the background I / O processing rate in three dimensions of cluster, server and disk, the influence of the distributed storage system on the foreground I / O processing can be reduced, the resource grabbing of various I / O processing in the distributed storage cluster can be prevented, the system resource utilization can be further improved, the foreground business can be ensured to enjoy sufficient I / O capacity, high-quality services can be provided to the outside, the balance between availability and reliability can be achieved, the balance between service quality and system reliability can be achieved, and the stable operation of the distributed storage system can be ensured.

[0157] In a second aspect, based on the same inventive concept, referring to Figure 4 The data management device 400 of the distributed storage system provided in the embodiments of the present application comprises:

[0158] The threshold value determination module 401 is configured to determine a multi-dimensional rate limiting threshold value in the case that an abnormal node exists in the storage cluster; the multi-dimensional rate limiting threshold value comprises a cluster-dimensional rate limiting threshold value, a server-dimensional rate limiting threshold value and a hard disk-dimensional rate limiting threshold value.

[0159] The condition determining module 402 is configured to determine whether the to-be-processed I / O data meets a data migration condition based on the to-be-processed data amount of the abnormal node and the multi-dimensional throttling threshold.

[0160] The data sending module 403 is configured to send the to-be-processed I / O data to a target hard disk of a target server if it is determined that the to-be-processed I / O data meets the data migration condition.

[0161] In an embodiment of the present application, the server dimension throttling threshold includes a source server throttling threshold of a source server and a target server throttling threshold of a target server.

[0162] The hard disk dimension throttling threshold includes a source hard disk throttling threshold of a source hard disk and a target hard disk throttling threshold of a target hard disk.

[0163] In an embodiment of the present application, the threshold determining module 401 includes:

[0164] The first threshold sub-module is configured to determine a cluster dimension throttling threshold based on a first I / O processing capacity of the storage cluster and a resource allocation ratio.

[0165] The second threshold sub-module is configured to determine a source server throttling threshold based on a second I / O processing capacity of the source server and the resource allocation ratio.

[0166] The third threshold sub-module is configured to determine a target server throttling threshold based on a third I / O processing capacity of the target server and the resource allocation ratio.

[0167] The fourth threshold sub-module is configured to determine a source hard disk throttling threshold based on a fourth I / O processing capacity of the source hard disk and the resource allocation ratio.

[0168] The fifth threshold sub-module is configured to determine a target hard disk throttling threshold based on a fifth I / O processing capacity of the target hard disk and the resource allocation ratio.

[0169] In an embodiment of the present application, the data management apparatus 400 of the distributed storage system further includes:

[0170] The ratio determining module is configured to determine the resource allocation ratio based on a data access frequency of the to-be-processed I / O data.

[0171] In an embodiment of the present application, the data migration condition includes a cluster migration condition, a server migration condition and a hard disk migration condition; and the condition determining module 402 includes:

[0172] The first condition determining sub-module is configured to determine whether the to-be-processed I / O data meets the cluster migration condition based on the to-be-processed data amount, a first data amount of in-transit I / O data of the storage cluster and the cluster dimension throttling threshold.

[0173] a second condition determining sub-module, configured to determine whether the to-be-processed I / O data meets a server migration condition based on the to-be-processed data amount, a second data amount of in-transit I / O data of the source server, a third data amount of in-transit I / O data of the target server, and a server dimension speed limiting threshold value;

[0174] a third condition determining sub-module, configured to determine whether the to-be-processed I / O data meets a hard disk migration condition based on the to-be-processed data amount, a fourth data amount of in-transit I / O data of the source hard disk, a fifth data amount of in-transit I / O data of the target hard disk, and a hard disk dimension speed limiting threshold value.

[0175] In an embodiment of the present application, the first condition determining sub-module comprises:

[0176] a first total amount determining unit, configured to determine a first data total amount between the to-be-processed data amount and the first data amount;

[0177] a first condition determining unit, configured to determine that the to-be-processed I / O data meets the cluster migration condition in a case where the first data total amount is less than or equal to the cluster dimension speed limiting threshold value;

[0178] the second condition determining sub-module comprises:

[0179] a second total amount determining unit, configured to determine a second data total amount between the to-be-processed data amount and the second data amount, and a third data total amount between the to-be-processed data amount and the third data amount;

[0180] a second condition determining unit, configured to determine that the to-be-processed I / O data meets the server migration condition in a case where the second data total amount is less than or equal to the source server speed limiting threshold value and the third data total amount is less than or equal to the target server speed limiting threshold value;

[0181] the third condition determining sub-module comprises:

[0182] a third total amount determining unit, configured to determine a fourth data total amount between the to-be-processed data amount and the fourth data amount, and a fifth data total amount between the to-be-processed data amount and the fifth data amount;

[0183] a third condition determining unit, configured to determine that the to-be-processed I / O data meets the hard disk migration condition in a case where the fourth data total amount is less than or equal to the source hard disk speed limiting threshold value and the fifth data total amount is less than or equal to the target hard disk speed limiting threshold value.

[0184] In an embodiment of the present application, the data management apparatus 400 of the distributed storage system further comprises:

[0185] The data updating module is configured to update the first data amount, the second data amount, the third data amount, the fourth data amount and the fifth data amount based on the to-be-processed data amount when it is determined that the to-be-processed I / O data meets the cluster migration condition, the server migration condition and the hard disk migration condition.

[0186] In an embodiment of the present application, the data management apparatus 400 of the distributed storage system further comprises:

[0187] The timing module is configured to trigger timing for the delay sending time length when it is determined that the to-be-processed I / O data does not meet the cluster migration condition, the server migration condition or the hard disk migration condition.

[0188] The cycle judgment module is configured to execute the step of determining whether the to-be-processed I / O data meets the data migration condition based on the to-be-processed data amount of the to-be-processed I / O data of the abnormal node and the multi-dimensional speed limiting threshold when the delay sending time length reaches the time length threshold.

[0189] It should be noted that the specific implementation of the data management apparatus 400 of the distributed storage system in the embodiments of the present application refers to the specific implementation of the data management method of the distributed storage system in the first aspect of the embodiments of the present application described above, which will not be repeated here.

[0190] Based on the same inventive concept, referring to Figure 5 The embodiments of the present application also provide an electronic device 500, which comprises a memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor 502, and the processor implements the steps in the data management method of the distributed storage system in any of the above embodiments when executing.

[0191] It should be noted that the specific implementation of the electronic device 500 in the embodiments of the present application refers to the specific implementation of the data management method of the distributed storage system in the first aspect of the embodiments of the present application described above, which will not be repeated here.

[0192] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the steps in the data management method of the distributed storage system in any of the above embodiments.

[0193] It should be noted that the specific implementation of the computer readable storage medium in the embodiments of the present application refers to the specific implementation of the data management method of the distributed storage system in the first aspect of the embodiments of the present application described above, which will not be repeated here.

[0194] Based on the same inventive concept, the embodiment of the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the data management method of the distributed storage system of any of the above embodiments.

[0195] It should be noted that the specific implementation of the computer program product of the embodiment of the present application is described with reference to the specific implementation of the data management method of the distributed storage system of the first aspect of the embodiment of the present application, which will not be described here.

[0196] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0197] Those skilled in the art should understand that the embodiments of the embodiment of the present application can be provided as a method, device, or computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0198] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system), and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device that implements the functions specified in the flowchart and / or block diagram. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0199] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product comprising instruction devices that implement the functions specified in the flowchart and / or block diagram. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0200] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one block or multiple blocks.

[0201] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to cover all changes and modifications falling within the scope of the embodiments of the present application.

[0202] Finally, it should also be noted that, in this document, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or terminal device including a series of elements includes not only those elements but also other elements not explicitly listed or other elements inherent in such a process, method, article, or terminal device. Without more limitations, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or terminal device including the element.

[0203] The above describes in detail the data management method of a distributed storage system, electronic device and storage medium provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A data management method of a distributed storage system, characterized by, The method comprises: In the case of an abnormal node of a storage cluster, a multi-dimensional throttling threshold is determined, the multi-dimensional throttling threshold comprising a cluster dimension throttling threshold, a server dimension throttling threshold and a hard disk dimension throttling threshold, the server dimension throttling threshold comprising a source server throttling threshold of a source server and a target server throttling threshold of a target server, and the hard disk dimension throttling threshold comprising a source hard disk throttling threshold of a source hard disk and a target hard disk throttling threshold of a target hard disk; Based on the amount of to-be-processed data of the to-be-processed I / O data of the abnormal node and the multi-dimensional throttling threshold, it is determined whether the to-be-processed I / O data meets a data migration condition, the data migration condition comprising a cluster migration condition, a server migration condition and a hard disk migration condition; In the case where it is determined that the to-be-processed I / O data meets the data migration condition, the to-be-processed I / O data is sent to a target hard disk of a target server; Based on the amount of to-be-processed data, a first amount of in-transit I / O data of the storage cluster, and the cluster dimension throttling threshold, a first total amount of data between the amount of to-be-processed data and the first amount of data is determined; In the case where the first total amount of data is less than or equal to the cluster dimension throttling threshold, it is determined that the to-be-processed I / O data meets the cluster migration condition; Based on the amount of to-be-processed data, a second amount of in-transit I / O data of the source server, a third amount of in-transit I / O data of the target server, and the server dimension throttling threshold, a second total amount of data between the amount of to-be-processed data and the second amount of data, and a third total amount of data between the amount of to-be-processed data and the third amount of data are determined; In the case where the second total amount of data is less than or equal to the source server throttling threshold and the third total amount of data is less than or equal to the target server throttling threshold, it is determined that the to-be-processed I / O data meets the server migration condition; Based on the amount of to-be-processed data, a fourth amount of in-transit I / O data of the source hard disk, a fifth amount of in-transit I / O data of the target hard disk, and the hard disk dimension throttling threshold, a fourth total amount of data between the amount of to-be-processed data and the fourth amount of data, and a fifth total amount of data between the amount of to-be-processed data and the fifth amount of data are determined; In the case where the fourth total amount of data is less than or equal to the source hard disk throttling threshold and the fifth total amount of data is less than or equal to the target hard disk throttling threshold, it is determined that the to-be-processed I / O data meets the hard disk migration condition.

2. The data management method of a distributed storage system according to claim 1, wherein, The multi-dimensional throttling threshold is determined, comprising: Based on a first I / O processing capacity of the storage cluster and a resource allocation ratio, the cluster dimension throttling threshold is determined; Based on a second I / O processing capacity of the source server and the resource allocation ratio, the source server throttling threshold is determined; Based on a third I / O processing capacity of the target server and the resource allocation ratio, the target server throttling threshold is determined; Based on a fourth I / O processing capacity of the source hard disk and the resource allocation ratio, the source hard disk throttling threshold is determined; determining the target hard disk speed limiting threshold based on the fifth I / O processing capability of the target hard disk and the resource allocation ratio.

3. The data management method of a distributed storage system according to claim 2, wherein, The method further comprises: determining the resource allocation ratio based on a data access frequency of the I / O data to be processed.

4. The data management method of the distributed storage system according to claim 1, wherein determining whether the I / O data to be processed meets the data migration condition based on the to-be-processed data amount of the I / O data to be processed of the abnormal node and the multi-dimensional speed limiting threshold comprises: determining whether the I / O data to be processed meets the cluster migration condition based on the to-be-processed data amount, a first data amount of in-transit I / O data of the storage cluster, and the cluster dimension speed limiting threshold; determining whether the I / O data to be processed meets the server migration condition based on the to-be-processed data amount, a second data amount of in-transit I / O data of the source server, a third data amount of in-transit I / O data of the target server, and the server dimension speed limiting threshold; determining whether the I / O data to be processed meets the hard disk migration condition based on the to-be-processed data amount, a fourth data amount of in-transit I / O data of the source hard disk, a fifth data amount of in-transit I / O data of the target hard disk, and the hard disk dimension speed limiting threshold.

5. The data management method of a distributed storage system according to claim 1, wherein, The method further comprises: updating the first data amount, the second data amount, the third data amount, the fourth data amount, and the fifth data amount based on the to-be-processed data amount in a case where it is determined that the I / O data to be processed meets the cluster migration condition, the server migration condition, and the hard disk migration condition.

6. The data management method of a distributed storage system according to claim 4, wherein, The method further comprises: in a case where it is determined that the I / O data to be processed does not meet the cluster migration condition, the server migration condition, or the hard disk migration condition, triggering timing for a delay sending time length; in a case where the delay sending time length reaches a time length threshold, performing the step of determining whether the I / O data to be processed meets the data migration condition based on the to-be-processed data amount of the I / O data to be processed of the abnormal node and the multi-dimensional speed limiting threshold.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the data management method of the distributed storage system according to any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the data management method of the distributed storage system according to any one of claims 1-6.

Images