Data recovery method, system, device, and storage medium

Abstract

Embodiments of the present application provide a data recovery method, system, device and storage medium. A plurality of storage nodes in a storage system are used to store a plurality of data blocks of target data, the plurality of data blocks comprising: a first number of original data blocks of target data and a check data block generated by the first number of original data blocks. When a client triggers a degraded read operation, data blocks in at least the first number of storage nodes are preferentially read, and the speed of reading data blocks to redundant storage nodes is controlled based on a traffic control strategy, so that the pressure caused by the degraded read operation on the storage system is flexibly controllable. Thus, when the access pressure of the storage system is large, the risk of increasing a large additional access pressure on the storage system can be reduced based on the traffic control strategy, and when the access pressure of the storage system is small, a sufficient number of data blocks can be quickly received by at least the first number of storage nodes and the redundant storage nodes, thereby effectively reducing the latency glitch of the degraded read operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data recovery method, system, device and storage medium. BACKGROUND

[0002] Erasure Code is an encoding fault-tolerant technology, which is used to recover original data when part of the data is lost. Erasure Code can occupy less storage space while ensuring reliability, and is widely used in distributed storage systems. Based on the Erasure Code technology, M parity data blocks can be calculated and generated according to K original data blocks. When any data block is lost, data recovery of the lost data block can be performed by using at least K remaining data blocks.

[0003] In a distributed storage system, K+M data blocks are stored in data storage nodes. When any data block is lost, the client needs to send a degraded read request to at least K data storage nodes and wait for the at least K data storage nodes to return the data block before performing data recovery. In the above process, the probability of the time delay of part of the degraded read request being much greater than the average time delay (i.e. request spike) is high. Therefore, a new solution needs to be proposed. SUMMARY

[0004] Aspects of the present application provide a data recovery method, system, device and storage medium to optimize the probability of request spike in degraded read request.

[0005] The present application provides a data recovery method, wherein a plurality of data blocks of target data are stored in a plurality of storage nodes of a storage system; the plurality of data blocks include a first number of original data blocks of the target data and parity data blocks generated by the first number of original data blocks; the method comprises: detecting that there is any abnormal storage node in the storage system; determining a first node set and a second node set from non-abnormal storage nodes of the storage system; the number of storage nodes in the first node set is greater than or equal to the first number; sending a degraded read request for an abnormal data block in the abnormal storage node to the storage nodes in the first node set, and sending the degraded read request to target storage nodes in the second node set by using a preset flow control strategy; when data blocks returned by a second number of storage nodes in the first node set and / or the target storage nodes are received, performing data recovery on the abnormal data block according to the data blocks returned by the second number of storage nodes; the second number is greater than or equal to the first number.

[0006] Optionally, determining a first node set and a second node set from the non-abnormal storage nodes of the storage system includes: determining the access pressure of the storage system; determining, based on the access pressure of the storage system, a first node set by selecting a number of storage nodes greater than or equal to the first number from the non-abnormal storage nodes of the storage system; and selecting the storage nodes other than the first node set from the non-abnormal storage nodes as the second node set.

[0007] Optionally, a preset flow control strategy is adopted to send a degraded read request to the target storage node in the second node set, including: if the number of data blocks associated with the abnormal data block received within a set time range is less than the second number, then the preset flow control strategy is adopted to send a degraded read request to the target storage node in the second node set.

[0008] Optionally, a preset flow control strategy is adopted to send a degraded read request to the target storage node in the second node set, including: adding the degraded read request for the storage node in the second node set to the flow control queue; when the storage system meets the preset flow control conditions, selecting the degraded read request corresponding to the target storage node from the flow control queue; and sending the corresponding degraded read request to the target storage node.

[0009] Optionally, the flow control conditions include: the number of requests sent to the storage system per unit time is within a set number range; or, the bandwidth of the requests sent to the storage system per unit time is within a set bandwidth range; or, the sending speed of the requests sent to the storage system per unit time is within a set speed range.

[0010] Optionally, when the storage system meets preset flow control conditions, selecting a degraded read request corresponding to the target storage node from the flow control queue includes: when the storage system meets preset flow control conditions, determining the number of redundant degraded read requests to be sent based on the level of the flow control conditions met by the storage system; selecting degraded read requests from the flow control queue based on the number of redundant degraded read requests to be sent, thereby obtaining the degraded read request corresponding to the target storage node.

[0011] Optionally, when the storage system meets the preset traffic control conditions, selecting the degraded read request corresponding to the target storage node from the traffic control queue includes: when the storage system meets the preset traffic control conditions, determining from the second node set a storage node in a light-load state or a balanced state as the target storage node; and selecting the degraded read request corresponding to the target storage node from the traffic control queue.

[0012] This application embodiment also provides a storage system, including: a client and multiple storage nodes; wherein, the multiple storage nodes are used to store multiple data blocks of target data; the multiple data blocks include: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks; and, upon receiving a downgrade read request for an abnormal data block sent by the client, returning a data block associated with the abnormal data block to the client; the client is used to: when detecting any abnormal storage node among the multiple storage nodes, determine a first node set and a second node set from the unabnormal storage nodes; the number of storage nodes in the first node set is greater than or equal to the first number; send a downgrade read request for the abnormal data block in the abnormal storage node to the storage nodes in the first node set, and, using a preset flow control strategy, send the downgrade read request to the target storage node in the second node set; upon receiving a second number of data blocks returned by the first node set and / or the target storage node, perform data recovery on the abnormal data block according to the data blocks returned by the second number of storage nodes; the second number is greater than or equal to the first number.

[0013] Optionally, when the client sends a degraded read request to a target storage node in the second node set using a preset traffic control strategy, it specifically performs the following steps: adding the degraded read request for the storage node in the second node set to the traffic control queue; when the storage system meets the preset traffic control conditions, selecting the degraded read request corresponding to the target storage node from the traffic control queue; and sending the corresponding degraded read request to the target storage node.

[0014] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided in this application.

[0015] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.

[0016] In the data recovery method provided in this application embodiment, multiple storage nodes in the storage system are used to store multiple data blocks of the target data. These multiple data blocks include: a first number of original data blocks of the target data and verification data blocks generated from the first number of original data blocks. When a client triggers a degraded read operation, it prioritizes reading data blocks from at least a first number of storage nodes and controls the speed of reading data blocks from redundant storage nodes based on a flow control strategy. This makes the pressure on the storage system caused by the degraded read operation flexible and controllable. Therefore, when the access pressure on the storage system is high, the risk of significantly increasing additional access pressure on the storage system can be reduced based on the flow control strategy. When the access pressure on the storage system is low, a sufficient number of data blocks can be quickly received through at least a first number of storage nodes and redundant storage nodes, thereby effectively reducing latency spikes in the degraded read operation, especially long-tail latency spikes. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 A schematic diagram of the structure of a storage system provided in an exemplary embodiment of this application;

[0019] Figure 2 A schematic flowchart of a data recovery method provided for an exemplary embodiment of this application;

[0020] Figure 3 A flowchart illustrating a data recovery method provided as another exemplary embodiment of this application;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0024] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0025] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0026] Based on erasure coding technology, M parity data blocks can be generated from K original data blocks. When any data block is lost, the remaining at least K data blocks can be used to recover the lost data. In a distributed storage system, K+M data blocks are stored on different data storage nodes. When any data block is lost, the client needs to send degraded read requests to at least K data storage nodes and wait for all K data storage nodes to return data blocks before data recovery can proceed. In this process, there is a high probability that the latency of some degraded read requests will be much greater than the average latency (i.e., request glitches).

[0027] If the client sends degraded read requests to all K+M-1 data storage nodes, the lost data blocks can be recovered when any K requests return. This method can reduce the probability of data read glitches, but it will further amplify the data read pressure in the entire storage system, thereby affecting system stability and the average latency of the cluster.

[0028] To address the aforementioned technical problems, a solution is provided in some embodiments of this application. The technical solutions provided by each embodiment of this application are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the structure of a storage system provided in an exemplary embodiment of this application, as shown below. Figure 1As shown, the storage system 100 includes: a client 10 and a distributed storage cluster 20. The distributed storage cluster 20 includes multiple storage nodes, such as... Figure 1 The storage nodes shown are 201, 202, 203, 204, 205, and 206. Of course, Figure 1 The storage nodes shown above are only used as an example to illustrate the distributed storage cluster 20 and do not constitute a limitation on the number of storage nodes included in the distributed storage cluster 20.

[0030] The storage node can be a regular server, a cloud server, a cloud computing data center, or an elastic computing instance in the cloud, etc., and this embodiment does not impose any restrictions.

[0031] In this embodiment, the distributed storage cluster 20 includes multiple storage nodes for storing multiple data blocks of the target data. These multiple data blocks include: a first number of original data blocks of the target data and verification data blocks generated from the first number of original data blocks. The first number of original data blocks can be obtained by fragmenting the target data. Any data block among the first number of original data blocks and the verification data blocks can be restored from the other first number of data blocks. Different data blocks of the target data are stored on different storage nodes to improve data security.

[0032] The client 10 is used to receive data access requests from users or upper-layer systems, and to perform data read and write operations on the distributed storage cluster 20 according to the data access requests. In this embodiment, the client 10 can monitor the operating status of the distributed storage cluster 20, and when any abnormal storage node is detected, it can recover potentially damaged or lost data blocks in the abnormal storage node based on the coding fault tolerance technology corresponding to erasure coding. The following will provide an exemplary description.

[0033] When client 10 detects any abnormal storage node in the storage system, it can determine a first node set and a second node set from the non-abnormal storage nodes in the storage system. The first node set contains a number of storage nodes greater than or equal to a first quantity, allowing the client to preferentially request the reading of at least a first quantity of data blocks from the storage nodes in the first node set. The storage nodes in the second node set, referred to as redundant storage nodes, serve as backup storage nodes for the first node set, thereby improving the speed at which client 10 reads at least the first quantity of data blocks. Figure 1 As shown, assuming the target data is sharded into two original data blocks, the client 10 can add storage nodes 201, 202 and 203 to the first node set, and add storage nodes 204, 205 and 206 to the second node set.

[0034] For ease of description, data blocks that may be damaged or lost in an abnormal storage node can be described as abnormal data blocks.

[0035] Client 10 can send a degraded read request for the abnormal data block to the storage nodes in the first node set, and can also use a preset flow control strategy to send the degraded read request to the target storage node in the second node set.

[0036] Degradation read requests refer to requests triggered by the storage system to read related data from other nodes to recover abnormal data when abnormal data is accessed.

[0037] The flow control strategy controls the rate at which degraded read requests are sent to the target storage nodes in the second node set. Therefore, it can be used to limit the frequency and number of redundant degraded read requests, reducing the access pressure on the storage system.

[0038] When the storage nodes in the first node set and the target storage nodes in the second node set receive a downgrade read request for an abnormal data block sent by the client 10, they can return the data block associated with the abnormal data block to the client 10.

[0039] In this system, each data block stored in any storage node may carry an identification identifier for that data block. The storage node can parse the identification identifier of the data to be read from the received downgrade read request and identify the target data block to be returned from the locally stored data blocks based on that identification identifier. In some embodiments, the identification identifier of a data block may be the identifier of the target data to which the data block belongs. For example, data A is fragmented to obtain original data blocks D1, D2, D3, D4, and D5. These original data blocks can generate check blocks P1 and P2. Data blocks D1, D2, D3, D4, D5, P1, and P2 are stored on different storage nodes and may carry a unique identification identifier for data A. When any of the above data blocks is abnormal, the downgrade read request for that data block may carry the unique identifier of data A. Therefore, the storage node can return the data block associated with data A to the client based on the unique identifier of data A carried in the downgrade read request.

[0040] Among the storage nodes that have sent downgrade read requests, the time it takes for different storage nodes to receive the downgrade read request may vary, and the latency for processing the request will also differ. When client 10 receives the data blocks returned by the second number of storage nodes, it can recover the abnormal data blocks based on these second number of returned data blocks. Here, the second number is greater than or equal to the first number. For example, continuing... Figure 1In the example shown, when the target data is fragmented into two original data blocks, the client 10 receives data blocks returned by two or three storage nodes, enabling the recovery of any lost or corrupted data block in the target data. The second number of storage nodes can be storage nodes from the first node set and / or the target storage nodes, specifically determined based on the response speed of different storage nodes to requests; this embodiment does not impose any limitations.

[0041] In this embodiment, multiple storage nodes in the storage system are used to store multiple data blocks of the target data. These multiple data blocks include: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks. When a client triggers a degraded read operation, it prioritizes reading data blocks from at least a first number of storage nodes, and controls the speed at which data blocks are read from redundant storage nodes based on a flow control strategy. This makes the pressure on the storage system caused by the degraded read operation flexible and controllable. Therefore, when the access pressure on the storage system is high, as few storage nodes as possible can be accessed, and the risk of significantly increasing additional access pressure on the storage system is reduced based on the flow control strategy. When the access pressure on the storage system is low, a sufficient number of data blocks can be quickly received through at least a first number of storage nodes and redundant storage nodes, thereby effectively reducing latency spikes in the degraded read operation, especially long-tail latency spikes.

[0042] In some optional embodiments, when the client 10 determines the first node set and the second node set from the non-abnormal storage nodes of the storage system, it can dynamically adjust the number of storage nodes in the first node set and the second node set according to the current operating state of the storage system, so as to reduce the risk of causing significant additional pressure on the storage system. The current operating state of the storage system can include either the storage system's runtime phase or its operating pressure.

[0043] For example, in some embodiments, if a degraded read operation is triggered during off-peak hours (e.g., at night), client 10 can add more storage nodes to the first node set to improve the response speed of the degraded read request. Conversely, if a degraded read operation is triggered during peak hours (e.g., weekdays), client 10 can add as few as possible, but more than a first number of storage nodes to the first node set to reduce the access pressure on the storage system while ensuring that a sufficient number of data blocks are obtained.

[0044] In some alternative embodiments, the client 10 can perform real-time stress monitoring on the storage system. This stress monitoring can be achieved by monitoring the operating metrics of the storage system, including but not limited to at least one of the following: the number of disk I / O (Input / Output) per second of the storage system, the request wait time of the storage system, and the number of requests processed per second by the storage system. This embodiment does not impose any limitations on these metrics.

[0045] When a degraded read operation is triggered, client 10 can determine the current access pressure of the storage system. Based on this access pressure, it can select a first set of storage nodes from the non-abnormal storage nodes in the storage system, choosing a number greater than or equal to a first set. The remaining storage nodes from the non-abnormal storage nodes, excluding those in the first set, can be used as a second set of nodes. The lower the current access pressure of the storage system, the more storage nodes will be in the first set. Conversely, the higher the current access pressure, the fewer storage nodes will be in the first set, even if they are greater than the first number.

[0046] Based on this implementation method, the number of nodes in the first node set can be dynamically adjusted according to the pressure of the storage system, which can reduce the disk pressure on the storage nodes and reduce the latency spikes of degraded read requests.

[0047] In some optional embodiments, the client 10 may determine whether to initiate a degradation read request for the redundant storage node based on the return speed of the storage node that has sent the degradation read request.

[0048] Optionally, if the number of data blocks associated with the abnormal data block received by client 10 within a set time range is less than the second number, client 10 may use a preset flow control strategy to send a degraded read request to the target storage node in the second node set. The set time range can be configured based on latency tolerance. For example, a smaller time range can be set for degraded read requests that are more sensitive to latency, while a larger time range can be set for degraded read requests that are less sensitive to latency; this embodiment does not impose any limitations.

[0049] Based on this implementation, a dynamic switch can be set for sending degraded read requests to redundant storage nodes by configuring a time range and performing a judgment operation. This allows for selection of whether to send a redundant request based on the current latency. This helps save resources and reduce the pressure on the storage system.

[0050] Optionally, when client 10 sends a degraded read request to a target storage node in the second node set using a preset flow control strategy, it can add the degraded read request to the storage node in the second node set to a flow control queue. When the storage system meets the preset flow control conditions, client 10 can select the degraded read request corresponding to the target storage node from the flow control queue and send the corresponding degraded read request to the target storage node.

[0051] Optionally, the flow control conditions may include, but are not limited to: the number of requests sent to the storage system per unit time is within a set number range; or, the bandwidth of the requests sent to the storage system per unit time is within a set bandwidth range; or, the sending speed of the requests sent to the storage system per unit time is within a set speed range.

[0052] The number of storage nodes included in the target storage node can be all the nodes in the second node set, or it can be a subset of the nodes in the second node set.

[0053] In some optional embodiments, the number of storage nodes included in the target storage node can be dynamically varied according to the flow control conditions met by the storage system. For example, when the load pressure on the storage system is high, the number of storage nodes included in the target storage node is small, and when the load pressure on the storage system is low, the number of storage nodes included in the target storage node is large. Exemplary examples will be provided below.

[0054] Optionally, the flow control conditions may include multiple levels, with different levels corresponding to different ranges of indicator values.

[0055] In some exemplary embodiments, if the storage system meets preset traffic control conditions, when the client 10 selects the degraded read request corresponding to the target storage node from the traffic control queue, it can determine the number of redundant degraded read requests to be sent according to the level of traffic control conditions met by the storage system; and select the degraded read request from the traffic control queue according to the number of redundant degraded read requests to be sent, thereby obtaining the degraded read request corresponding to the target storage node.

[0056] For example, in some scenarios, the set quantity range in the flow control conditions may include a first quantity range, a second quantity range, and a third quantity range that increase sequentially. When the number of requests sent to the storage system per unit time is within the first quantity range, the storage system can be considered to be in a light-load state; when the number of requests sent to the storage system per unit time is within the second quantity range, the storage system can be considered to be in a balanced state; and when the number of requests sent to the storage system per unit time is within the third quantity range, the storage system can be considered to be in an overload state.

[0057] Similarly, the bandwidth range set in the flow control conditions can include a first bandwidth range, a second bandwidth range, and a third bandwidth range that increase sequentially. When the bandwidth of requests sent to the storage system per unit time is within the first bandwidth range, the storage system is considered to be in a light-load state; when the bandwidth of requests sent to the storage system per unit time is within the second bandwidth range, the storage system is considered to be in a balanced state; and when the bandwidth of requests sent to the storage system per unit time is within the third bandwidth range, the storage system is considered to be in an overload state. Optionally, when the storage system is in a light-load state, the number of storage nodes included in the target storage node can be all nodes in the second node set, thereby quickly easing downgraded read requests to redundant storage nodes when the pressure on the storage system is low, greatly reducing request latency. When the storage system is in a balanced state, the number of storage nodes included in the target storage node can be 60% of the nodes in the second node set. When the storage system is in an overload state, the number of storage nodes included in the target storage node can be 20% of the nodes in the second node set to reduce the pressure on the storage system. It should be understood that the above values ​​are only used to illustrate this embodiment and do not constitute a limitation on the method of determining the number of target storage nodes.

[0058] In some other exemplary embodiments, if the storage system meets the preset traffic control conditions, when the client 10 selects the degraded read request corresponding to the target storage node from the traffic control queue, it can determine the storage node in a light-load state or a balanced state from the second node set as the target storage node; and select the degraded read request corresponding to the target storage node from the traffic control queue.

[0059] In this implementation, client 10 can dynamically monitor the traffic metrics of storage nodes in the second set of storage nodes and determine the load status of the storage nodes based on the range of metric values ​​that the traffic metrics meet. As described in the previous embodiments, the storage node can be determined to be in a light-load state, a balanced state, or an overload state based on the range of at least one of the following metrics: current bandwidth, number of requests received per second, and request reception rate per second. Therefore, when the storage system as a whole meets the traffic control conditions, storage nodes in a light-load state or a balanced state can be selected from the second set of nodes as storage nodes for sending redundant requests.

[0060] In some cases, if all nodes in the second node set are in a light-load or balanced state, all nodes in the second node set can be used as target storage nodes. This allows for a quick relaxation of degraded read requests to redundant storage nodes when the pressure on the storage nodes in the second node set is relatively low, greatly reducing request latency.

[0061] In this implementation, selecting redundant storage nodes based on node-level stress monitoring results can further reduce the stress on redundant storage nodes and optimize the impact of degraded read requests on the normal services of the storage system.

[0062] The embodiments of this application will be further illustrated below with reference to the accompanying drawings and specific examples. Assume that the storage system, based on erasure coding technology, calculates M parity data blocks from K original data blocks, and the K+M data blocks are stored on different storage nodes, where K and M are positive integers.

[0063] like Figure 2 As shown, when any storage node is detected to be abnormal, the client can trigger a degradation read operation with erasure coding for the target data block on that storage node. The client can select K+n (0 <= n <= M-1, where n is configurable or determined according to the storage system pressure) storage nodes from the K+M-1 storage nodes in normal operation, and send a degradation read request to the data replicas (i.e., the data blocks associated with the target data block) in the K+n storage nodes. The client can record the remaining M-1-n redundant storage nodes that did not send requests, and add the degradation read requests for the M-1-n storage nodes to the flow control queue to control the traffic bandwidth, number of requests, or sending speed of all redundant degradation read requests in the entire process.

[0064] When the storage system meets the flow control conditions, the client can select m (0 <= n <= M-1-n) nodes from the M-1-n storage nodes that have not sent degraded read requests, and send degraded read requests to these m nodes. Afterwards, if there are still remaining storage nodes that have not sent degraded read requests, the degraded read requests for these remaining storage nodes are added to the flow control queue. If the number of successfully returned data blocks is less than K, degraded read requests from the flow control queue can continue to be sent when the storage system meets the flow control conditions. If the number of successfully returned data blocks is greater than or equal to K, the unsent requests are removed from the flow control queue, and the target data block is recovered using the data blocks that have already been successfully read.

[0065] For example, consider K=6 and m=6. In some cases, if a data block is abnormal, the client can send a degraded read request to 7 of the remaining 11 nodes. When the client receives the data block returned by the second-to-last request, the total number of data blocks acquired is 6, at which point the abnormal data block can be recovered. That is, the request latency depends on the return latency of the second-to-last request. In other cases, when the cluster pressure is low, if a data block is abnormal, the client can send a degraded read request to all 11 remaining nodes. When the client receives the data block returned by the sixth-to-last request, the total number of data blocks acquired is 6, at which point the abnormal data block can be recovered. That is, the request latency depends on the return latency of the sixth-to-last request. When the return times of degraded read requests are similar to a normal distribution, the later the request returns, the longer the latency; that is, the return latency of the second-to-last request is greater than the return latency of the sixth-to-last request. The more nodes that send degraded read requests, the lower the latency for the client to acquire a sufficient number of data blocks.

[0066] In this implementation, when cluster pressure is low, degraded read requests from redundant storage nodes added to the flow control queue are sent quickly, allowing the redundant storage nodes to reduce long-tail latency spikes in degraded reads. When cluster pressure is high, when a client triggers a degraded read operation, it can prioritize sending degraded read requests to K+n storage nodes, and the flow control queue controls the speed at which degraded read requests are sent to redundant storage nodes, thereby ensuring that disk pressure remains at a controllable level even under high cluster pressure.

[0067] In addition to the storage system described in the foregoing embodiments, this application also provides a data recovery method. Figure 3 This is a flowchart illustrating a data recovery method provided in an exemplary embodiment of this application. The method is applicable to clients of storage systems and may include, for example: Figure 3 The steps shown are as follows:

[0068] Step 301: When any storage node of the storage system is detected to be abnormal, a first node set and a second node set are determined from the non-abnormal storage nodes of the storage system; the number of storage nodes in the first node set is greater than or equal to a first number; wherein, multiple data blocks of the target data are stored on multiple storage nodes of the storage system; the multiple data blocks include: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks.

[0069] Step 302: Send a degraded read request for the abnormal data block in the abnormal storage node to the storage node in the first node set, and send the degraded read request to the target storage node in the second node set using a preset flow control strategy.

[0070] Step 303: When receiving data blocks returned by a second number of storage nodes in the first node set and / or the target storage node, perform data recovery on the abnormal data blocks based on the data blocks returned by the second number of storage nodes; the second number is greater than or equal to the first number.

[0071] In some optional embodiments, one way to determine a first node set and a second node set from the non-abnormal storage nodes of the storage system includes: determining the access pressure of the storage system; determining, based on the access pressure of the storage system, a first node set by selecting a number of storage nodes greater than or equal to a first number from the non-abnormal storage nodes of the storage system; and selecting the storage nodes other than the first node set from the non-abnormal storage nodes as the second node set.

[0072] In some optional embodiments, one method of sending a degraded read request to the target storage node in the second node set using a preset flow control strategy includes: if the number of data blocks associated with the abnormal data block received within a set time range is less than the second number, then a degraded read request is sent to the target storage node in the second node set using a preset flow control strategy.

[0073] In some optional embodiments, a method of sending a degraded read request to a target storage node in the second node set using a preset flow control strategy includes: adding the degraded read request for the storage node in the second node set to a flow control queue; when the storage system meets the preset flow control conditions, selecting the degraded read request corresponding to the target storage node from the flow control queue; and sending the corresponding degraded read request to the target storage node.

[0074] In some optional embodiments, the flow control conditions include: the number of requests sent to the storage system per unit time is within a set number range; or, the bandwidth of the requests sent to the storage system per unit time is within a set bandwidth range; or, the sending speed of the requests sent to the storage system per unit time is within a set speed range.

[0075] In some optional embodiments, the number of storage nodes included in the target storage node is dynamically configured according to the flow control conditions satisfied by the storage system.

[0076] Optionally, when the storage system meets preset flow control conditions, a method for selecting a degraded read request corresponding to the target storage node from the flow control queue includes: when the storage system meets preset flow control conditions, determining the number of redundant degraded read requests to be sent based on the level of the flow control conditions met by the storage system; and selecting degraded read requests from the flow control queue based on the number of redundant degraded read requests to be sent, thereby obtaining the degraded read request corresponding to the target storage node.

[0077] Optionally, when the storage system meets preset traffic control conditions, a method for selecting a degraded read request corresponding to the target storage node from the traffic control queue includes: when the storage system meets preset traffic control conditions, determining a storage node in a light-load state or a balanced state from the second node set as the target storage node; and selecting a degraded read request corresponding to the target storage node from the traffic control queue.

[0078] In this embodiment, multiple storage nodes in the storage system are used to store multiple data blocks of the target data. These multiple data blocks include: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks. When a client triggers a degraded read operation, it prioritizes reading data blocks from at least a first number of storage nodes, and controls the speed at which data blocks are read from redundant storage nodes based on a flow control strategy. This makes the pressure on the storage system caused by the degraded read operation flexible and controllable. Therefore, when the access pressure on the storage system is high, the risk of adding significant additional access pressure to the storage system can be reduced based on the flow control strategy. When the access pressure on the storage system is low, a sufficient number of data blocks can be quickly received through at least a first number of storage nodes and redundant storage nodes, thereby effectively reducing latency spikes in the degraded read operation, especially long-tail latency spikes.

[0079] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 301 to 304 can be device A; or the execution subject of steps 301 and 302 can be device A, and the execution subject of step 303 can be device B; and so on.

[0080] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 301, 302, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0081] Figure 4 This illustration shows a structural diagram of an electronic device provided in an exemplary embodiment of this application. This electronic device is applicable to the data recovery system provided in the foregoing embodiments. The electronic device runs a client within the data recovery system. In the data recovery system, multiple data blocks of target data are stored on multiple storage nodes of a storage system; the multiple data blocks include: a first number of original data blocks of the target data and verification data blocks generated from the first number of original data blocks.

[0082] like Figure 4 As shown, the electronic device includes: a memory 401, a processor 402, and a communication component 403.

[0083] Memory 401 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0084] The processor 402, coupled to the memory 401, is configured to execute a computer program in the memory 401 for: when any abnormal storage node is detected in the storage system, determining a first node set and a second node set from the non-abnormal storage nodes in the storage system; the number of storage nodes in the first node set is greater than or equal to the first number; sending a downgrade read request for abnormal data blocks in the abnormal storage node to the storage nodes in the first node set, and sending the downgrade read request to a target storage node in the second node set using a preset flow control strategy; and, upon receiving data blocks returned by a second number of storage nodes in the first node set and / or the target storage node, performing data recovery on the abnormal data blocks based on the data blocks returned by the second number of storage nodes; the second number is greater than or equal to the first number.

[0085] Optionally, when the processor 502 determines the first node set and the second node set from the non-abnormal storage nodes of the storage system, it is specifically configured to: determine the access pressure of the storage system; determine, based on the access pressure of the storage system, a number of storage nodes greater than or equal to the first number from the non-abnormal storage nodes of the storage system to obtain the first node set; and take the storage nodes other than the first node set from the non-abnormal storage nodes as the second node set.

[0086] Optionally, when the processor 502 sends a degraded read request to the target storage node in the second node set using a preset flow control strategy, it specifically uses the following method: if the number of data blocks associated with the abnormal data block received within a set time range is less than the second number, then the processor 502 sends a degraded read request to the target storage node in the second node set using a preset flow control strategy.

[0087] Optionally, when the processor 502 sends a degraded read request to a target storage node in the second node set using a preset flow control strategy, it specifically performs the following steps: adding the degraded read request for the storage node in the second node set to the flow control queue; when the storage system meets the preset flow control conditions, selecting the degraded read request corresponding to the target storage node from the flow control queue; and sending the corresponding degraded read request to the target storage node.

[0088] Optionally, the flow control conditions include: the number of requests sent to the storage system per unit time is within a set number range; or, the bandwidth of the requests sent to the storage system per unit time is within a set bandwidth range; or, the sending speed of the requests sent to the storage system per unit time is within a set speed range.

[0089] Optionally, the number of storage nodes included in the target storage node is dynamically configured according to the flow control conditions satisfied by the storage system.

[0090] Optionally, when the processor 502 selects a degraded read request corresponding to the target storage node from the flow control queue when the storage system meets the preset flow control conditions, it specifically performs the following steps: when the storage system meets the preset flow control conditions, it determines the number of redundant degraded read requests to be sent based on the level of the flow control conditions met by the storage system; and selects a degraded read request from the flow control queue based on the number of redundant degraded read requests to be sent, thereby obtaining the degraded read request corresponding to the target storage node.

[0091] Optionally, when the storage system meets the preset flow control conditions, the processor 502 selects the degraded read request corresponding to the target storage node from the flow control queue, specifically: when the storage system meets the preset flow control conditions, it determines from the second node set a storage node in a light-load state or a balanced state as the target storage node; and selects the degraded read request corresponding to the target storage node from the flow control queue.

[0092] Furthermore, such as Figure 4 As shown, the electronic device also includes other components such as a power supply component 404. Figure 4 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 4 The components shown.

[0093] The memory 401 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0094] The communication component 403 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), and other technologies.

[0095] The power supply component 404 is used to provide power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0096] In this embodiment, multiple storage nodes in the storage system are used to store multiple data blocks of the target data. These multiple data blocks include: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks. When a client triggers a degraded read operation, it prioritizes reading data blocks from at least a first number of storage nodes, and controls the speed at which data blocks are read from redundant storage nodes based on a flow control strategy. This makes the pressure on the storage system caused by the degraded read operation flexible and controllable. Therefore, when the access pressure on the storage system is high, the risk of adding significant additional access pressure to the storage system can be reduced based on the flow control strategy. When the access pressure on the storage system is low, a sufficient number of data blocks can be quickly received through at least a first number of storage nodes and redundant storage nodes, thereby effectively reducing latency spikes in the degraded read operation, especially long-tail latency spikes.

[0097] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by an electronic device in the above method embodiments.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxesFigure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0103] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0104] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0105] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data recovery method, wherein, Multiple data blocks of the target data are stored on multiple storage nodes of the storage system; The plurality of data blocks includes: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks; the method includes: When any abnormal storage node is detected in the storage system, a first node set and a second node set are determined from the non-abnormal storage nodes of the storage system, including: determining the access pressure of the storage system; determining, based on the access pressure of the storage system, a first node set by selecting a number of storage nodes greater than or equal to a first number from the non-abnormal storage nodes of the storage system; and selecting the storage nodes other than the first node set from the non-abnormal storage nodes as the second node set; wherein the number of storage nodes in the first node set is greater than or equal to the first number. Send a downgraded read request for the abnormal data block in the abnormal storage node to the storage node in the first node set, and send the downgraded read request to the target storage node in the second node set using a preset flow control strategy. Upon receiving data blocks returned by a second number of storage nodes from the first node set and / or the target storage node, data recovery is performed on the abnormal data blocks based on the data blocks returned by the second number of storage nodes; the second number is greater than or equal to the first number.

2. The method according to claim 1, wherein, Using a preset flow control strategy, a degraded read request is sent to the target storage node in the second node set, including: If the number of data blocks associated with the abnormal data block received within the set time range is less than the second number, a preset flow control strategy is adopted to send a degraded read request to the target storage node in the second node set.

3. The method according to claim 1, wherein, Using a preset flow control strategy, a degraded read request is sent to the target storage node in the second node set, including: Add the degraded read request for the storage node in the second node set to the flow control queue; When the storage system meets the preset flow control conditions, a degraded read request corresponding to the target storage node is selected from the flow control queue; Send the corresponding downgrade read request to the target storage node.

4. The method according to claim 3, wherein, The flow control conditions include: the number of requests sent to the storage system per unit time is within a set number range; or, the bandwidth of the requests sent to the storage system per unit time is within a set bandwidth range; or, the sending speed of the requests sent to the storage system per unit time is within a set speed range.

5. The method according to claim 3 or 4, wherein, When the storage system meets the preset flow control conditions, a degraded read request corresponding to the target storage node is selected from the flow control queue, including: When the storage system meets the preset flow control conditions, the number of redundant degradation read requests to be sent is determined according to the level of the flow control conditions met by the storage system. Based on the number of redundant degraded read requests to be sent, degraded read requests are selected from the flow control queue to obtain the degraded read request corresponding to the target storage node.

6. The method according to claim 3 or 4, wherein, When the storage system meets the preset flow control conditions, a degraded read request corresponding to the target storage node is selected from the flow control queue, including: When the storage system meets the preset flow control conditions, a storage node in a light-load state or a balanced state is determined from the second node set and used as the target storage node. Select the downgraded read request corresponding to the target storage node from the flow control queue.

7. A storage system, comprising: The client and multiple storage nodes; among them, The plurality of storage nodes are used to store a plurality of data blocks of target data; the plurality of data blocks include: a first number of original data blocks of the target data and a verification data block generated from the first number of original data blocks; and, upon receiving a downgrade read request for an abnormal data block sent by the client, returning the data block associated with the abnormal data block to the client; The client is used to: execute the data recovery method according to any one of claims 1-6.

8. An electronic device, wherein, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein, When a computer program is executed by a processor, it can implement the data recovery method according to any one of claims 1-6.

Images