Data reconstruction method, medium, device, program product, and distributed storage system

Abstract

A data reconstruction method, a medium, a device, a program product, and a distributed storage system. The method is applied to a distributed storage system comprising a plurality of data nodes, and comprises: determining a plurality of candidate data nodes where a plurality of target data blocks used for reconstructing original data are located, wherein the number of the candidate data nodes is greater than the minimum number of data nodes used for reconstructing the original data; on the basis of historical transmission performance of the plurality of candidate data nodes, respectively estimating current transmission performance of the plurality of candidate data nodes; requesting a target data block from at least one target data node among the plurality of candidate data nodes, so that after requesting the target data block from the at least one target data node, the total number of data blocks requested from the plurality of candidate data nodes is positively correlated with the current transmission performance of the plurality of candidate data nodes, wherein the number of the target data nodes is smaller than the number of the candidate data nodes; and performing data reconstruction on the original data on the basis of the target data block returned by the at least one target data node.

Description

[0001]Data Reconstruction Method, Medium, Device, Program Product, and Distributed Storage System. This disclosure claims priority to Chinese patent application No. 202410346179.X, filed with the China Patent Office on March 25, 2024, and entitled "Data Reconstruction Method, Medium, Device, Program Product, and Distributed Storage System," the entire contents of which are incorporated herein by reference. Technical Field: This disclosure relates to the field of distributed storage technology, and more particularly to a data reconstruction method, medium, device, program product, and distributed storage system. Background: In a distributed storage system, to improve data fault tolerance, multiple data blocks used to reconstruct original data can be stored on multiple data nodes of the distributed storage system. When original data is lost or damaged, data blocks can be read from multiple data nodes and reconstructed based on the read data blocks. The throughput of this process is limited by the data node with the slowest transmission speed among the multiple data nodes. When the transmission speeds of some data nodes in a distributed storage system remain low for an extended period, the cluster bandwidth of the distributed storage system cannot be effectively utilized, resulting in low throughput and poor throughput stability. In a first aspect, embodiments of the present disclosure provide a data reconstruction method for a distributed storage system comprising multiple data nodes. The method comprises: determining multiple candidate data nodes where multiple target data blocks for reconstructing original data are located; the number of candidate data nodes being greater than the minimum number of data nodes where target data blocks for reconstructing the original data are located; estimating the current transmission performance of each of the multiple candidate data nodes based on their historical transmission performance; requesting a target data block from at least one of the multiple candidate data nodes, such that, after requesting the target data block from the at least one target data node, the total number of data blocks requested from each of the multiple candidate data nodes is positively correlated with the current transmission performance of each of the multiple candidate data nodes; the number of target data nodes being less than the number of candidate data nodes; and reconstructing the original data based on the target data block returned by the at least one target data node. In some embodiments, a current transmission performance of any one of the multiple target data nodes is better than a current transmission performance of any one of the multiple candidate data nodes except the multiple target data nodes.In some embodiments, the historical transmission performance is represented based on historical transmission delay, and the current transmission performance is represented based on current transmission delay. Estimating the current transmission performance of each of the candidate data nodes based on their historical transmission performance includes estimating, for each candidate data node, the current transmission delay of the candidate data node based on the candidate data node's historical transmission delay and the total number of data blocks that the candidate data node has not yet completed transmitting. In some embodiments, the data blocks that the candidate data node has not yet completed transmitting include: data blocks requested by a data request currently being processed by the candidate data node; data blocks requested by data requests that the candidate data node has received and are in a queued state; and target data blocks stored on the candidate data node. In some embodiments, the multiple data nodes correspond one-to-one to multiple cache queues. The cache queue corresponding to each data node is used to cache data requests received by the data node, wherein the data requests are used to request data blocks stored by the data node for reconstructing original data. Data requests received by the candidate data node and in a queued state are cached in the cache queue corresponding to the candidate data node. After the candidate data node completes sending the data block requested by the data request currently being processed by the candidate data node, the candidate data node uses the next data request in the corresponding cache queue as the data request currently being processed by the candidate data node. In some embodiments, the method further includes: obtaining multiple data requests received by the candidate data node within a historical time window, wherein the data requests are used to request data blocks for reconstructing original data at the candidate data node; determining an average transmission delay of the candidate data node within the historical time window based on an average of transmission delays of the data blocks requested by the multiple data requests; and determining a historical transmission delay of the candidate data node based on the average transmission delay of the candidate data node within the historical time window. In some embodiments, determining the historical transmission delay of the candidate data node based on the average transmission delay of the candidate data node within the historical time window includes weighting the average transmission delays of the candidate data node within multiple historical time windows to obtain the historical transmission delay of the candidate data node. In some embodiments, the data blocks stored by the multiple data nodes include to-be-reconstructed data blocks or parity blocks obtained by erasure coding the original data.In a second aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method described in any embodiment of the present disclosure is implemented. In a third aspect, an embodiment of the present disclosure provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable by the processor. When the processor executes the program, the method described in any embodiment of the present disclosure is implemented. In a fourth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program. When the processor executes the computer program, the method described in any embodiment of the present disclosure is implemented. In a fifth aspect, an embodiment of the present disclosure provides a distributed storage system. The distributed storage system includes a metadata node and multiple data nodes. The metadata node is configured to distribute data reconstruction tasks to a reconstruction node among the multiple data nodes. The reconstruction node is any one of the multiple data nodes. The data node is configured to, when the data node is the reconstruction node, execute the method described in any embodiment of the present disclosure in response to receiving the data reconstruction task. In the disclosed embodiment, multiple candidate data nodes for reconstructing the target data blocks of the original data are first determined. Since the number of candidate nodes exceeds the minimum number of data nodes required for data reconstruction, there is redundancy among the candidate data nodes. Therefore, the target data blocks can be obtained from only some of the candidate data nodes to reconstruct the original data. Then, based on the current transmission performance of each candidate data node, a target data node is selected from each candidate data node, and the target data blocks are requested from the target data node. Since the total number of requested data blocks on each candidate data node is positively correlated with the current transmission performance of the corresponding candidate data node, candidate data nodes with better transmission performance will receive data requests for a larger number of data blocks, while candidate data nodes with worse transmission performance will receive data requests for a smaller number of data blocks. This reduces the situation where data nodes with better transmission performance are required to transmit a larger number of data blocks while data nodes with better transmission performance are required to transmit a smaller number of data blocks, resulting in the data nodes with better transmission performance being idle for long periods of time. This improves the cluster bandwidth utilization of the distributed storage system, increases the data reconstruction throughput, and improves throughput stability. It should be understood that the general description above and the detailed description below are merely exemplary and explanatory and do not limit the present disclosure.Figure 1 is a schematic diagram of a distributed storage system according to an embodiment of the present disclosure; Figure 2 is a schematic diagram of a data reconstruction process according to an embodiment of the present disclosure; Figure 3 is a flow chart of a data reconstruction method according to an embodiment of the present disclosure; Figure 4 is a schematic diagram of the relationship between cluster bandwidth and the amount of tasks on each data node according to an embodiment of the present disclosure; Figure 5 is a schematic diagram of a task scheduling process according to an embodiment of the present disclosure; and Figure 6 is a schematic diagram of a computer device according to an embodiment of the present disclosure. Specific Embodiments Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numerals in different drawings represent like or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims. The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more associated listed items. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality. It should be understood that while this disclosure may employ terms such as "first," "second," and "third" to describe various types of information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information could also be referred to as second information, and similarly, second information could also be referred to as first information. Depending on the context, the term "if" as used herein could be interpreted as meaning "at the time," "at that time," or "in response to a determination." To help those skilled in the art better understand the technical solutions in the embodiments of this disclosure and to further clarify the aforementioned objectives, features, and advantages of the embodiments of this disclosure, the technical solutions in the embodiments of this disclosure are described in further detail below with reference to the accompanying drawings. A distributed storage system is a system that stores data on multiple independent data nodes to achieve distributed data storage and management. A distributed storage system must ensure that the state presented to the outside world is consistent and that state rollbacks do not occur. As shown in FIG1 , the distributed storage system 10 includes a metadata node (MetaNode) 102 and multiple data nodes (ChunkServers) 104.Metadata nodes 102 are centralized metadata storage nodes, typically used to store data status information, storage location information, data length information, and so on. Data nodes 104 are nodes within the distributed storage system 10 that store data and are generally responsible for operations such as writing, storing, reading, and deleting data replicas. To improve data fault tolerance, multiple data blocks used to reconstruct the original data can be stored on multiple data nodes 104 within the distributed storage system 10. To improve data reliability within the distributed storage system 10, when some data is lost or damaged, the distributed storage system 10 can retrieve data blocks from multiple data nodes 104 to reconstruct the lost or damaged data and write the reconstructed data to other data nodes 104, thereby ensuring data security. One data reconstruction method is based on erasure codes. In this method, a segment of data is divided into multiple data blocks of equal length, each of which includes the original data block and at least one check block. The original data blocks contain the original data content to be stored or transmitted and can be obtained by dividing the original data into blocks according to a certain partitioning method. The parity blocks are redundant checksum information calculated based on the data blocks and are used to detect and recover errors or loss in the data blocks. The parity blocks do not contain the original user data but are redundant information calculated based on the content of the data blocks. If any one or more data blocks are lost, the lost data blocks can be reconstructed and recovered based on the remaining data blocks. Different data blocks can be stored on different data nodes 104. As shown in Figure 2, the number of data nodes 104 is four for illustration. For ease of distinction, different data nodes 104 are denoted as data node CS1, data node CS2, data node CS3, and data node CS4. Data node CS1 is used to store data block A1, data node CS2 is used to store data block A2, data node CS3 is used to store data block A3, and data node CS4 is used to store data block A4. Data blocks A1, A2, and A3 may be original data blocks, and data block A4 may be a parity block. If some of the data blocks A1, A2, and A3 are lost or damaged, the data blocks can be reconstructed using other data blocks. For example, if data block A1 is damaged, data block A1 can be reconstructed using data blocks A2, A3, and A4.Specifically, data nodes CS2, CS3, and CS4 can send data blocks A2, A3, and A4, respectively, to reconstruction node Y. Reconstruction node Y can reconstruct data based on the received data blocks A2, A3, and A4 to obtain reconstructed data block A1, and then send the reconstructed data block A1 to data node CS1 for storage. In some embodiments, when the minimum number of original data blocks is k and the number of check blocks is m, the total number of data blocks and check blocks required to reconstruct any original data block is at least k. Assuming k is 8 and m is 3, the total number of data blocks and check blocks required to reconstruct any original data block is at least 8. For example, data reconstruction can be performed using 7 original data blocks and 1 check block, or using 6 original data blocks and 2 check blocks. Related technologies generally randomly read data blocks from multiple data nodes to perform data reconstruction. The throughput of the above process is limited by the data node with the slowest transmission speed among the multiple data nodes. In situations such as device heterogeneity, partial device failures, high front-end device pressure, and large amounts of data to be copied from devices, the transmission speeds of some data nodes in a distributed storage system may remain low for extended periods. This prevents effective utilization of the distributed storage system's cluster bandwidth, resulting in low data reconstruction throughput and poor throughput stability. A related technique draws on a front-end service level agreement (SLA) approach, identifying significantly slow nodes and adding them to a read blacklist to minimize subsequent reads from these nodes. However, this approach completely blocks slow nodes, rendering them completely unable to provide any throughput. This method is unsuitable for back-end data reconstruction and fails to optimize throughput. Therefore, embodiments of the present disclosure provide a data reconstruction method that utilizes slower data nodes to provide a certain level of throughput while also reducing the impact of these slower nodes on the overall data reconstruction throughput. The method of the embodiment of the present disclosure is applied to a distributed storage system including multiple data nodes 104 . Each data node 104 stores data blocks for reconstructing original data. Each piece of original data is reconstructed based on data blocks on one or more data nodes 104 .Referring to FIG. 3 , the method includes the following steps: Step S2: For each piece of original data to be reconstructed, determining multiple candidate data nodes where multiple target data blocks for reconstructing the original data are located; the number of candidate data nodes is greater than the minimum number of data nodes 104 where target data blocks for reconstructing the original data are located; Step S4: Based on the historical transmission performance of the multiple candidate data nodes, estimating the current transmission performance of each of the multiple candidate data nodes; Step S6: Requesting target data blocks from at least one target data node among the multiple candidate data nodes, such that after requesting the target data blocks from the at least one target data node, the total number of data blocks requested from the multiple candidate data nodes is positively correlated with the current transmission performance of each of the multiple candidate data nodes; the number of target data nodes is less than the number of candidate data nodes; Step S8: Reconstructing the original data based on the target data blocks returned by the at least one target data node. In this embodiment of the present disclosure, there is redundancy in the number of candidate nodes; and target data blocks may be obtained from only some of the candidate data nodes to reconstruct the original data. Because the current transmission performance of each candidate data node is considered during the target data node selection process, after requesting the target data block from the target data node, the total number of data blocks requested on each candidate data node matches the current transmission performance of the corresponding candidate data node. Therefore, candidate data nodes with better current transmission performance will receive data requests for a larger number of data blocks, while candidate data nodes with poorer current transmission performance will receive data requests for a smaller number of data blocks. This fully utilizes the data transmission capacity of each data node, reduces queuing caused by a mismatch between the number of data blocks required to be transmitted and the node's capacity, improves cluster bandwidth utilization in the distributed storage system, and enhances data reconstruction throughput and throughput stability. The following examples illustrate specific implementations of the disclosed embodiments. For ease of explanation, it is assumed that the minimum number of data nodes containing target data blocks used to reconstruct the original data is k, and the total number of candidate data nodes is n. The methods of the disclosed embodiments can be executed by a reconstruction node. The reconstruction node can be selected from multiple data nodes 104. For example, a source data node (a data node where a data block used to reconstruct the original data is located) can be determined as the reconstruction node, or a destination data node (a data node used to store the reconstructed original data) can be determined as the reconstruction node. Alternatively, any other data node can be determined as the reconstruction node.Each piece of original data can be reconstructed based on data blocks on at least k data nodes 104. Each data node 104 can store one or more data blocks, which may include both original data blocks and check blocks. During data reconstruction, each data node 104 can send at least some of its data blocks to a reconstruction node for data reconstruction. For ease of description, the following example uses the storage of one data block on each data node 104 as an example. In step S2, when data is missing, the metadata node 102 can send a data reconstruction task to the reconstruction node. The reconstruction node can receive the data reconstruction task from the metadata node 102. The data reconstruction task can include information about the original data to be reconstructed and information about each data block used to reconstruct the original data. The original data information includes the storage location and length of the original data, and the data block information includes the storage location and length of the data block. The storage location can include the identification information of the data node 104 storing the original data or data block, as well as the starting storage location of the original data or data block on the corresponding data node 104. For each piece of original data requiring reconstruction, the data reconstruction task can determine the multiple target data blocks required to reconstruct the original data and the candidate data nodes where these target data blocks are located. The reconstruction node can send data requests to the target data nodes among the candidate data nodes, so that the target data nodes return data blocks in response to the data requests. The reconstruction node can then reconstruct the original data based on the data blocks returned by the target data nodes and return the reconstructed original data to the data node 104 storing the original data. The number n of candidate data nodes is greater than k. Therefore, there is redundancy among the candidate data nodes, and the original data can be reconstructed by obtaining target data blocks from only some of the target data nodes. For example, in an erasure coding scenario, assuming the number of original data blocks is k and the number of parity blocks is m, each original data block can be a piece of original data. Assuming each candidate data node is used to store one data block (original data block or parity block), the number n of candidate data nodes storing data blocks equals k + m, and the minimum number of data blocks required to reconstruct the original data is n. That is, the number of candidate data nodes is greater than the minimum number of data blocks required to reconstruct the original data. In step S4, the current transmission performance of the n candidate data nodes can be estimated based on the historical transmission performance of the n candidate data nodes.Both historical and current transmission performance can be represented by transmission bandwidth (referred to as historical transmission bandwidth and current transmission bandwidth, respectively), or by transmission delay (referred to as historical transmission delay and current transmission delay, respectively), or by other parameters that characterize the data transmission capabilities of a candidate data node. Taking the example of representing historical transmission performance by historical transmission delay and current transmission performance by current transmission delay, for any candidate data node, the current transmission delay of the candidate data node can be estimated based on the candidate data node's historical transmission delay and the total number of data blocks that the candidate data node has not yet completed transmitting. The candidate data node's historical transmission delay can be used to represent the average transmission delay of the candidate data node for data blocks on the node during a historical time period. A larger historical transmission delay indicates that the candidate data node took longer to transmit data blocks on the node during the historical time period, thus indicating that the candidate data node had poor transmission performance during the historical time period. Conversely, a smaller historical transmission delay indicates that the candidate data node required less time to transmit data blocks on this node during the historical time period, thus indicating that the candidate data node has better transmission performance over the past period. Multiplying the candidate data node's historical transmission delay by the total number of data blocks that the candidate data node has not yet completed transmitting can yield the candidate data node's current transmission delay. Assuming the candidate data node's historical transmission delay is D, the time required for the candidate data node to transmit a data block during the historical time period is D0. Assuming the total number of data blocks that the candidate data node has not yet completed transmitting is r, the candidate data node's current transmission delay D can be expressed as D = D0 * r. In some embodiments, the candidate data node's currently untransmitted data blocks include: data blocks requested by the data request currently being processed by the candidate data node, the number of which is denoted as r1; data blocks requested by data requests already received by the candidate data node and in a queued state, the number of which is denoted as r2; and target data blocks stored on the candidate data node, the number of which is denoted as r3. Assuming that the candidate data node sends each data block serially, the total number of data blocks that the candidate data node has not yet completed transmitting is r = r1 + r2 + r3. Therefore, the current transmission delay D of the candidate data node can be expressed as D = D0 * (r1 + r2 + r3). ​​It should be noted that in this step, whether the candidate data node needs to transmit the data blocks stored on the node has not yet been determined.Here, it is assumed that the candidate data node needs to transmit the data block stored on this node. Based on this, the current transmission delay of the data block on this node is estimated. In subsequent steps, a decision is made based on this current transmission delay to determine whether the candidate data node needs to transmit the data block on this node. In some embodiments, a cache queue can be set for each data node 104. Multiple data nodes 104 are associated with multiple cache queues. The cache queue corresponding to each data node 104 is used to cache data requests received by the data node 104. The data requests received by the data node 104 are used to request data blocks stored by the data node for reconstructing the original data. Taking the system shown in Figure 2 as an example, a cache queue corresponding to data node CS1, denoted as cache queue Q1, can be set up to store data requests for data blocks on data node CS1. A cache queue corresponding to data node CS2, denoted as cache queue Q2, can also be set up to store data requests for data blocks on data node CS2. A cache queue corresponding to data node CS3, denoted as cache queue Q3, can also be set up to store data requests for data blocks on data node CS3. A cache queue corresponding to data node CS4, denoted as cache queue Q4, can also be set up to store data requests for data blocks on data node CS4. After receiving a data request, each data node 104 can add the data request to its corresponding cache queue. Data node 104 can sequentially read each data request cached in the cache queue and, in response to each data request, transmit the data block stored on the node to the reconstruction node. After the data block is successfully transmitted, the corresponding data request is deleted from the cache queue. After a candidate data node completes the transmission of the data block requested by the data request currently being processed by the candidate data node, the candidate data node may proceed to use the next data request in its corresponding cache queue as the data request currently being processed. Data requests already received by the candidate data node and in a queued state may be cached in the cache queue corresponding to the candidate data node. Thus, the number of data blocks requested by data requests already received by the candidate data node and in a queued state can be determined based on the number of data requests in the cache queue corresponding to the candidate data node. Assuming that each data request requests a data block, and assuming that the number of data requests in the cache queue is V, the cache queue may include one currently being processed data request and v-1 received and queued data requests.It will be understood that the above description describes a case where the data node 104 transmits one data block at a time. If the data node 104 transmits multiple data blocks at a time, the number of data requests currently being processed in the cache queue may be greater than 10. In some embodiments, a time window of a preset length may be set, and the average transmission delay of the candidate data node for the data blocks on the node within the time window may be determined as the historical transmission delay of the candidate data node. Specifically, multiple data requests received by the candidate data node within the historical time window may be obtained, each data request being for a data block for reconstructing the original data from the candidate data node. The average transmission delay of the data blocks requested by the multiple data requests may be determined based on the average transmission delay of the candidate data node within the historical time window, and the historical transmission delay of the candidate data node may be determined based on the average transmission delay of the candidate data node within the historical time window. Assume that the number of data requests received within a historical time window is u, and each data request requests a data block. The time when the reconstruction node sent the data request (denoted as T1) can be determined based on the timestamp carried in the data request. The time when the candidate data node completed processing the data request (denoted as T2) can also be determined. This is the time when the candidate data node successfully sent the requested data block to the reconstruction node. The transmission delay of the requested data block can be expressed as T = T2 - T1. The data requests received by the candidate data node within the historical time window are those whose timestamps fall within the historical time window. The transmission delay T of the data blocks requested by each data request received by the candidate data node within the historical time window can be calculated using the above method. The average transmission delay T of the data blocks requested by each data request is calculated and used as the average transmission delay of the candidate data node within the historical time window, also known as the historical transmission delay T_x of the candidate data node, denoted as: Where Ti is the transmission delay of the data block requested by the i-th data request within the historical time window. Furthermore, the average transmission delay of the candidate data node to the data block on the current node within multiple time windows can be obtained, and the historical transmission delay of the candidate data node can be determined based on the average transmission delays within these multiple time windows. For example, the average transmission delays of the candidate data node within multiple historical time windows can be weighted using the weights corresponding to each time window to obtain the historical transmission delay of the candidate data node. Alternatively, the weights corresponding to each historical time window can be determined based on the time interval between the historical time window and the current time. The weights corresponding to the historical time windows are inversely correlated with the time interval between the historical time window and the current time. That is, the longer the time interval between the historical time window and the current time, the smaller the weight corresponding to the historical time window. In this way, the influence of historical time windows with a closer time interval to the current time on the calculation results can be increased. In other examples, the weights corresponding to the various time windows can also be equal, or can be calculated using other methods, which are not limited here. The above embodiment illustrates the use of transmission delay to characterize transmission performance. In other examples, transmission bandwidth can also be used to represent transmission performance. Transmission bandwidth and transmission delay can be inversely correlated. In some embodiments, the transmission delay can be calculated first, and then converted to transmission bandwidth based on the conversion relationship between transmission delay and transmission bandwidth. The specific conversion method is not limited in this disclosure. In step S6, a target data block can be requested from at least one target data node among the n candidate data nodes. The number of target data blocks is less than the number of candidate data nodes, and the number of target data nodes is greater than or equal to the minimum number k of data nodes where target data blocks for reconstructing the original data are located. For ease of description, the number of target data nodes is assumed to be k. The current transmission performance of any one of the multiple target data nodes is better than the current transmission performance of other data nodes among the multiple candidate data nodes, excluding the multiple target data nodes. In other words, this step selects k nodes with better current transmission performance from the multiple candidate data nodes as target nodes for transmitting the data block. When transmission delay is used to represent transmission performance, the k nodes with better current transmission performance are the k nodes with smaller current transmission delay. When transmission bandwidth is used to represent transmission performance, the k nodes with better current transmission performance are the k nodes with larger current transmission bandwidth.Since candidate data nodes with better transmission performance currently require less time to transmit data blocks, while candidate data nodes with poorer transmission performance currently require more time to transmit data blocks, assigning data block transmission tasks to candidate data nodes with better transmission performance can reduce the situation where some data nodes remain idle for extended periods after completing their own data block transmission tasks, waiting for other nodes to complete their data block transmissions, thereby improving data reconstruction throughput. Figure 4 shows a schematic diagram of the relationship between cluster bandwidth and the task load of each data node 104, according to an embodiment of the present disclosure. The metadata node 102 can issue data reconstruction tasks to the reconstruction node, which can establish a replica-level queue (i.e., a cache queue corresponding to each data node 104) and perform replica-level task scheduling. Assuming the system includes six data nodes 104, designated CS1, CS2, CS3, CS4, CS5, and CS6, the real-time bandwidth (i.e., current transmission bandwidth) of each data node 104 is shown in the figure. Without task scheduling by the reconfigured node, data blocks are randomly requested from each data node 104. The task load for each data node 104 is shown in the distribution of pre-scheduling tasks in the upper right corner of Figure 4 . The height of the rounded rectangle labeled "Task Load" represents the number of data block transmission tasks required by the corresponding data node 104. It can be seen that data node CS1 has a higher real-time bandwidth, but the task load allocated to data node CS1 is relatively small. Data node CS6 has a lower real-time bandwidth, but the task load allocated to data node CS6 is relatively large. This results in data node CS1 completing its data block transmission task quickly and waiting for data node CS6 to transmit its data block. After the reconstruction node performs task scheduling, it allocates tasks based on the current transmission bandwidth of each data node 104. It can be seen that the amount of tasks allocated to each data node 104 matches the bandwidth of data node 104. Data nodes with larger bandwidths (such as data nodes CS1 and CS4) are assigned a larger amount of tasks, while data nodes with smaller bandwidths (such as data nodes CS2, CS5, and CS6) are assigned a smaller amount of tasks. This fully utilizes the data transmission capacity of each data node, reduces idle waiting times for data nodes, and improves system throughput. Furthermore, in the embodiments of the present disclosure, the original data can be a data block to be reconstructed among the data to be reconstructed. This data block to be reconstructed can be a data block of a preset size (e.g., 1 MB).In scenarios where changes in foreground load cause transmission bandwidth to dynamically vary between data nodes 104, the reconstruction node can quickly detect changes in node capacity due to scheduling of small requests at the 1MB level, making timely adjustments during the scheduling of subsequent requests. For example, assuming the size of the data to be reconstructed is 10MB, consisting of ten 1MB data blocks. When reconstructing the first data block, data node CS1 has a higher current transmission bandwidth, so the data request for the first data block is sent to data node CS1. When reconstructing the second data block, data node CS1 has a lower current transmission bandwidth, while data node CS2 has a higher current transmission bandwidth. Therefore, the data request for the first data block is sent to data node CS2, and so on. This approach allows rapid response to changes in the transmission bandwidth of each data node 104 and adjustment of data block scheduling. In step S8, the reconstruction node can reconstruct the original data based on the target data blocks returned by each target data node. Specifically, the reconstruction node may reconstruct the original data after each target data node successfully returns the target data block. Alternatively, after any target node returns the target data block, the reconstruction node may reconstruct the original data based on the currently acquired target data block, obtain an intermediate reconstruction result, and store the intermediate reconstruction result in memory. Upon subsequent receipt of the target data block, the reconstruction node may reconstruct the original data again based on the subsequently received target data block and the intermediate reconstruction result, until a complete reconstruction of the original data is obtained. Figure 5 illustrates a schematic diagram of the task scheduling process in an embodiment of the present disclosure. The reconstruction node may include a node selection module for selecting a target data node from multiple candidate data nodes. The node selection module first performs node performance estimation, primarily based on the length of the cache queue corresponding to each candidate data node and the historical transmission delay of each candidate data node to estimate the current transmission performance of each candidate data node. This disclosure does not limit the specific method for calculating current transmission performance statistics. In some embodiments, the historical transmission delay of the candidate data node and the transmission delay of the candidate data node within the most recent time window can be obtained. The historical transmission delay of the candidate data node and the transmission delay of the candidate data node within the most recent time window are weighted to obtain the current transmission delay of the candidate data node. The historical transmission delay can also be updated as the current transmission delay for use in the next calculation of the current transmission delay. The target data node is then selected based on a pre-set node selection policy.Several target data nodes with the shortest current transmission delays and the ability to reconstruct the original data can be selected. As shown in Figure 5, before scheduling, data nodes CS1, CS2, and CS3± were each assigned two data requests. However, due to the poor transmission performance of data node CS1, data node CS1± takes a long time to transmit its data block. As shown in the figure, data node CS1±'s data block begins transmission at time t1 and completes at time t4. In contrast, data nodes CS2 and CS3 begin transmission at time t1 and complete at time t2. Therefore, data nodes CS2 and CS3 are in an idle waiting state between time t2 and time t4. After scheduling, data nodes CS1 and CS4 are each assigned one data request, and data nodes CS2 and CS3 are each assigned two data requests. This ensures that the workload of each data node matches its transmission capacity as closely as possible. Data blocks on data nodes CS1, CS2, and CS3 are all transmitted at time t2, and data blocks on data node CS4 are transmitted between time t2 and time t1. This reduces idle waiting times for data nodes and achieves load balancing during the data reconstruction process. Furthermore, the disclosed embodiments do not require shielding data nodes with poor transmission capabilities. Instead, they rely on the principle of negative feedback to fully utilize the transmission bandwidth of data nodes with poor transmission capabilities, thereby improving cluster bandwidth utilization. The disclosed embodiments also provide a computer device comprising at least a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method described in any of the aforementioned embodiments is implemented. Figure 6 shows a more specific hardware structural diagram of a computing device provided by the disclosed embodiments. The device may include a processor 20, a memory 22, an input / output interface 24, a communication interface 26, and a bus 28. The processor 20, memory 22, input / output interface 24, and communication interface 26 are interconnected within the device via a bus 28. The processor 20 can be implemented using a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure. The processor 20 may also include a graphics card, such as an Nvidia Titan X graphics card or an 1080Ti graphics card.The memory 22 can be implemented in the form of a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 22 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of the present disclosure via software or firmware, the relevant program code is stored in the memory 22 and is called and executed by the processor 20. The input / output interface 24 is used to connect to an input / output module to enable information input and output. The input / output module can be configured as a component within the device (not shown) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., while output devices may include a display, speaker, vibrator, indicator light, etc. The communication interface 26 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can achieve communication via wired means (e.g., USB, network cable, etc.) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth, etc.). The bus 28 comprises a pathway for transmitting information between various components of the device (e.g., the processor 20, the memory 22, the input / output interface 24, and the communication interface 26). It should be noted that although the device shown above only includes the processor 20, the memory 22, the input / output interface 24, the communication interface 26, and the bus 28, in a specific implementation, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will appreciate that the device described above may only include the components necessary to implement the embodiments of the present disclosure, rather than all of the components shown. The present disclosure also provides a distributed storage system 10, comprising a metadata node 102 and multiple data nodes 104. The metadata node 102 is configured to distribute data reconstruction tasks to a reconstruction node among the multiple data nodes 104, where the reconstruction node is any one of the multiple data nodes 104. The data node 104 is configured to, when serving as the reconstruction node, execute the method described in any of the embodiments of the present disclosure in response to receiving a data reconstruction task. An embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method described in any embodiment of the present disclosure when executed by a processor.The present disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the aforementioned embodiments. Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can implement information storage using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmitting medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves. From the above description of the embodiments, it can be seen that those skilled in the art will clearly understand that the embodiments of the present disclosure can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the embodiments of the present disclosure, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or portions thereof. The systems, devices, modules, or units described in the above embodiments can be implemented by a computer device or entity, or by a product having certain functions. A typical implementation device is a computer, which can be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices. The various embodiments in the present disclosure are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in detail. Each embodiment focuses on the differences from other embodiments.In particular, the device embodiments are described briefly because they are generally similar to the method embodiments. For relevant details, refer to the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware components. Alternatively, some or all of the modules can be selected based on actual needs to achieve the objectives of the embodiments. Persons of ordinary skill in the art can understand and implement the embodiments without inventive effort. The above description is merely a specific implementation of the embodiments of the present disclosure. It should be noted that persons of ordinary skill in the art can make various improvements and modifications without departing from the principles of the embodiments of the present disclosure. Such improvements and modifications should also be considered within the scope of protection of the embodiments of the present disclosure.

Claims

Claims 1. A data reconstruction method, applied to a distributed storage system including multiple data nodes, the method comprising: for each piece of original data to be reconstructed, determining multiple candidate data nodes where multiple target data blocks for reconstructing the original data are located; the number of candidate data nodes being greater than the minimum number of data nodes where target data blocks for reconstructing the original data are located; estimating current transmission performance of each of the multiple candidate data nodes based on historical transmission performance of the multiple candidate data nodes; requesting a target data block from at least one target data node among the multiple candidate data nodes, such that after requesting the target data block from the at least one target data node, a total number of requested data blocks on the multiple candidate data nodes is positively correlated with the current transmission performance of the multiple candidate data nodes; the number of target data nodes being less than the number of candidate data nodes; and reconstructing the original data based on the target data block returned by the at least one target data node.

2. The method according to claim 1, wherein the current transmission performance of any one of the multiple target data nodes is better than the current transmission performance of any one of the multiple candidate data nodes except the multiple target data nodes.

3. The method according to claim 1 or 2, wherein the historical transmission performance is represented based on historical transmission delay, and the current transmission performance is represented based on current transmission delay; and estimating the current transmission performance of the multiple candidate data nodes based on the historical transmission performance of the multiple candidate data nodes comprises: For any candidate data node, the current transmission delay of the candidate data node is estimated based on the historical transmission delay of the candidate data node and the total number of data blocks that have not yet been transmitted by the candidate data node.

4. The method according to claim 3, wherein the data blocks that the candidate data node has not yet completed transmitting include: data blocks requested by a data request that the candidate data node is currently processing; data blocks requested by a data request that the candidate data node has received and is in a queued state; and target data blocks stored on the candidate data node.

5. The method according to claim 4, wherein the multiple data nodes correspond one-to-one to multiple cache queues, and the cache queue corresponding to each data node is used to cache data requests received by the data node, wherein the data requests are used to request data blocks stored by the data node for reconstructing original data; data requests received by the candidate data node and in a queued state are cached in the cache queue corresponding to the candidate data node; after the candidate data node completes transmission of the data block requested by the data request currently being processed by the candidate data node, the candidate data node uses the next data request in the corresponding cache queue as the data request currently being processed by the candidate data node.

6. The method according to any one of claims 3 to 5, further comprising: Acquire multiple data requests received by the candidate data node within a historical time window, where the data requests are used to request data blocks on the candidate data node for reconstructing original data; Determine the candidate data based on the average value of the transmission delay of the data blocks requested by the multiple data requests The average transmission delay of the candidate data node in the historical time window is determined, and the historical transmission delay of the candidate data node is determined based on the average transmission delay of the candidate data node in the historical time window.

7. The method according to claim 6, wherein determining the historical transmission delay of the candidate data node based on the average transmission delay of the candidate data node within the historical time window comprises: Weighted processing is performed on average transmission delays of the candidate data node in multiple historical time windows to obtain a historical transmission delay of the candidate data node.

8. The method according to any one of claims 1 to 7, wherein the data blocks stored by the multiple data nodes include data blocks to be reconstructed or check blocks obtained by performing erasure coding on the original data.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 8.

12. A distributed storage system comprising a metadata node and a plurality of data nodes; wherein: The metadata node is used to distribute data reconstruction tasks to a reconstruction node among the multiple data nodes; the reconstruction node is any one of the multiple data nodes; the data node is used to, when the node is the reconstruction node, execute the method according to any one of claims 1 to 8 in response to receiving the data reconstruction task.

Images